in Code, HowTo

Converting Between cv::Mat and QImage or QPixmap

In a previous article, I outlined how to compile OpenCV for Mac OS X and to build and run a small example. In that post I used OpenCV‘s High-Level GUI (highgui) module to display a simple interface using the Qt framework. This is great if you’re writing a one-off program or are just experimenting with OpenCV, but how do you interface with your own Qt-based project if you don’t want to (or can’t) use the highgui module?

OpenCV Library

OpenCV Library

+

Qt Library

Qt Library

The first hurdle is converting between data formats so you can work with images in both OpenCV and Qt. So how do you move an image’s data between a cv::Mat and a QImage or QPixmap so you can display it in a widget?

In this article, I present some functions to handle these conversions and explain how to use them.

(The complete file, along with an example, is available on github.)

I chose to put all the functions in their own header file and in their own namespace. In the first section there are functions to convert from cv::Mat to QImage and from cv::Mat to QPixmap.

The first thing to note is that the cv::Mat to QPixmap conversion (cvMatToQPixmap) at the bottom of this section simply relies on cvMatToQImage() and uses Qt’s QPixmap::fromImage() to return a QPixmap, so that’s pretty straightforward.

The second thing to notice is the switch statement in the cv::Mat to QImage conversion (cvMatToQImage). Because both OpenCV and Qt provide multiple data formats for images, we need to do different conversions depending on their internal layout. In this case I am interested in three OpenCV image formats: CV_8UC4 (8-bit unsigned, 4 channels), CV_8UC3 (8-bit unsigned, 3 channels), and CV_8UC1 (8-bit unsigned, 1 channel – grayscale).

The CV_8UC4 conversion to QImage::Format_ARGB32 is straightforward – simply construct a QImage and return it. Since QImage uses implicit data sharing, we don’t need to worry about any memory management issues.

The CV_8UC3 conversion to QImage::Format_RGB888 is almost as straightforward. The only difference is that we need to swap the red and blue components of all pixels since OpenCV and Qt use different orders.

The final case – CV_8UC1 – adds a small twist if you are using Qt < 5.5 (otherwise we can simply use QImage::Format_Grayscale8) . The grayscale QImage needs a color lookup table, so the first time this code is executed, it will create and fill in the static table. No point in doing it every time! (As André points out in the comments there is a more efficient way to do this if your profiling suggests that this is taking too much time.) We then set the color table on the image and return it.

If we happen to pass a cv::Mat with a format that is not supported here, we give a warning and return a NULL QImage. If you find you need other formats, you can simply extend this switch statement and the one below in QImageToCvMat().

After a comment by Yuriy Kozlov in March 2018, I spent some time looking at this code again and realized that endianness is an issue that was not covered by this header. I added the comment at the top about it and an explicit check that we are compiling with a little endian system.

In the second section of code, we handle conversion the other way – from QImage to cv::Mat and from QPixmap to cv::Mat.

As with the conversion the other way, the conversion from QPixmap (QPixmapToCvMat()) simply calls our QImage version and uses the built-in QPixmap::toImage() to handle to conversion to QImage.

Aside from the issue of memory management and cloning the data discussed below, the function to convert from QImage to cv::Mat (QImageToCvMat) is pretty straightforward. It supports five QImage formats that are analogous (or close) to the OpenCV ones discussed above: QImage::Format_ARGB32 and its friend QImage::Format_ARGB32_Premultiplied (8-bit unsigned, 4 channels), QImage::Format_RGB32 (which requires a little bit of work to deal with the extraneous alpha channel), QImage::Format_RGB888 (8-bit unsigned, 3 channels), and QImage::Format_Indexed8 (8-bit unsigned, 1 channel – grayscale).

Unlike conversions from cv::Mat, however, with conversions to cv::Mat we now have to be concerned about memory management. Keep in mind that QImage uses implicit data sharing, so when assigning one QImage to another, it keeps track of how many QImages are using the same data and releases it when nobody is using it. Because our conversions here rely on us pointing at that internal data, and because we have no way of telling the QImage we’re pointing at it, we either have to copy it, or ensure the the lifetime of the new cv::Mat does not exceed that of the original QImage.

If you know (programmatically) that the lifetime of the cv::Mat is shorter than the QImage, then you can be more efficient by passing  false to QPixmapToCvMat or QImageToCvMat for the  inCloneImageData argument. This will share the QImage data. For example you might have a function like this:

In this case you know the QPixmap exists longer than the cv::Mat, so there’s no reason to clone the data.

Exceptions to this are the QImage::Format_RGB32 and QImage::Format_RGB888 formats.  In these cases, we need to create a temporaries to convert the data so we don’t modify the QImage data.  This means we can’t just point at its data like we do with the others, so we are forced to clone it.  If this function is called with an image in either of these formats and inCloneImageData is false, we generate a warning and return a clone anyways.

If you are unsure, it is safest to let these functions clone the data.

I hope that’s useful to somebody.  Please feel free to point out any errors or offer a better solution!

(The complete file, along with an example, is available on github.)

10 March 2018:

  • added information about endianness at the top of the header
  • check for compilation on big endian systems and complain
  • changed handling of QImage::Format_RGB32 -> CV_8UC3 conversion – instead of a format conversion and a channel swap on the QImage side, we just strip the alpha channel on the OpenCV side
  • added a test program and test data in the github repo
  • the test program also shows how to split a cvMat and output the red, green, blue, and alpha channels into their own files

15 February 2017:

  • fixed RGB swapping for QImage::Format_RGB32 case (thanks Matthias!)
  • fixed CV_8UC1 color table initialization bug
  • adds a more succinct conversion from CV_8UC1 using QImage::Format_Grayscale8 for Qt >= 5.5

24 July 2016: Incorporated some of the feedback from below, updated the post, and put the code up on github.

Leave a Reply for André Cancel Reply

Write a Comment

Comment

37 Comments

  1. This is a good thorough answer better than the ones on Stack Overflow. I am still wondering if there is a way to get a QPixmap without an extra conversion, or how to get one with minimal copying.

    “The CV_8UC4 conversion to QImage::Format_ARGB32 is straightforward – simply construct a QImage and return it. Since QImage uses implicit data sharing, we don’t need to worry about any memory management issues.”

    This isn’t quite true and from my experience and reading the documentation leaves a dangling pointer that’s hard to track down if the QImage outlives the cv::Mat.

    “The buffer must remain valid throughout the life of the QImage and all copies that have not been modified or otherwise detached from the original buffer.”

    This is fine in your 3-channel implementation because rgbSwaped() copies the data. The 4-channel implementation doesn’t. But then, don’t you also need to swap BGR->RGB for 4 channels?

    • Thanks Yuriy!

      It’s been a long time since I worked with this, but I agree it looks like there’s a CV_8UC4/cv:Mat lifetime problem. How… unfortunate.

      As to the BGR->RGB issue – what you’re saying sounds reasonable. I thought I’d tested all the cases I present here, but I will take some time to look at them again.

    • Yuriy:

      I’ve spent some time looking at this.

      For the lifetime issue I am unable to get it to crash. I allocated the cv::Mat in a separate function on the stack, returned a QImage, and wrote it out. I even called release() and deallocate() on the Mat explicitly. Maybe this isn’t complex enough to reproduce it. If you can create a small example could you please add it as a GitHub issue?

      As for the BGR->RGB issue in the CV_8UC4 case, if I swap them I am returned an incorrect image. You can try this yourself by adding “.rgbSwapped()” to the return and running the example in the repo. I’m not 100% sure why yet – maybe this has something to do with endian-ness of the data?

      I verified that the CV_8UC4 Mat created by QImageToCvMat() in the QImage::Format_ARGB32 case is correct without swapping by splitting the Mat and writing out the individual channels.

      If you have more issues with this, it might be easier to communicate in a GitHub issue.

      • Just a quick followup – the channel swapping is indeed related to endianness. Some QImage formats are endian-dependent, others are not.

        I added an issue with more info about QImage and endianness.

        Aside: Looking into this has given me a better way to handle QImage::Format_RGB32 -> CV_8UC3. Instead of a conversion and then a channel swap on the QImage side I can just strip the alpha channel on the OpenCV side.

  2. Setting the colortable breaks my image. In cvMatToQImage() for the format there is a color table set on the image. Acutally without doing this my image is already fine. Setting the color table breaks my image.

  3. Your code has a mistake “swapped” is overwritten if the format is RGB32:

    What you actually wanted to write was:

    Please tell me if I can use your conversion code? And under which licence terms?

    best regards

    • Thanks Matthias! (I don’t use this code anymore, so I never ran into this issue…)

      I fixed the code here and on github.

      There is no license – you can do whatever you want with it.

  4. I am getting a warning C4267: ‘argument’ : conversion from ‘size_t’ to ‘int’, possible data loss at line where is is written QImage::Format_Indexed8. Is there anyone who can help me to debug this any tell me where I am running into this conversion problem.

    Thanks

    • I guess on Windows OpenCV’s step is a size_t. The QImage constructor requires an int.

      I don’t use MSVC, but your options are probably:

      1. turn off the warning
      2. ignore it
      3. (and I don’t recommend this one) an old C-style cast might work

      Aside: I hate size_t – it is such a pain to work with for portable code. I avoid using it as much as possible.

  5. Note that the initialization of the color lookup table could be more efficient. First of all, make sure you create the vector with the right size in one go instead of pushing back and causing reallocations. As QRgb is just a typedef for unsigned int, it is also easy to just initialize it once to right contents and be done with it. That saves a bunch of unneeded shifts and AND operations. I just generated a 256-element list and used brace initialization to initialize the vector with it. That also saves the if statement.

    I wrote:

    • You’re right – if you’re doing something where the initialization time is having an impact on your app then it makes sense to optimize it.

      Thanks André!

  6. Thank you VERY much for this. It is exactly what I need. I really appreciate you taking the time to explain the internals of the cv::Mat format and providing translation code.

  7. Awesome walkthrough of how it’s done. Much simpler than some other solutions I’ve seen for the same thing. Thanks for this.

  8. There seems to be an error on the third #include line, it reads:
    #include “opencv2/imgproc.hpp”
    to my understanding this should be:
    #include “opencv2/imgproc/imgproc.hpp”

  9. Thanks for the code. But I’d like to know why you are converting a CV_8UC4 image (which has four 8-bit channels) to a Format_RGB32 image (which should have three 8bit channels) and the other way around.
    The QImage documentation states that RGB32 images are saved as 0xffRRGGBB, which I read as one byte for blue, one for green and one for red and the first one is always 255.

    If I open for example a png saved with transparency (should have 4 channels) as QImage it has got the format ARGB32, which is not considered in your code. If I open one without transparency it is opened as RGB32 (so only 3 channels, or the alpha channel is white).

    Can you please explain this to me?

    • As I mentioned above it’s been a long time since I looked at any of this and I’m not using it in any active projects anymore, but it seems that I’m missing two things:

      1) CV_8UC4 should be converted to QImage::Format_ARGB32 in cvMatToQImage

      2)QImageToCvMat should add another case for QImage::Format_ARGB32 which does the same thing as QImage::Format_RGB32

      This is probably what Jassie was referring to in the comment above (though I don’t know about his crashing problem).

      I’ll have to experiment when I get the chance.

      Thanks for the input!

      • Ok, I’ve been trying around a little bit and changing the line to

        QImage(mat.data, mat.cols, mat.rows, mat.step, QImage::Format_ARGB32);

        works fine (alpha channel is correctly converted).

        I added some more cases to the conversion function from QImage to cv::Mat.

        If the QImage is Format_ARGB32 or Format_ARGB32_Premultiplied it can be converted by:

        cv::Mat(image.height(), image.width(), CV_8UC4, const_cast(image.bits()), image.bytesPerLine()).clone();

        If it is Format_RGB32 then the problem is, that the image has got an alpha channel which is completely white. So you can either convert it to a CV_8UC4 image, or the other way would be doing the following:

        QImage converted = image.convertToFormat(QImage::Format_RGB888).rgbSwapped();

        mat = cv::Mat(converted.height(), converted.width(), CV_8UC3, const_cast(converted.bits()), converted.bytesPerLine()).clone();

        Then the resulting image has only got the three colour channels. Unfortunately it has to be converted which might be expensive, I don’t know.

        If you like you can have a look at the complete code here:

        http://goo.gl/UEAQrb

        Just have a look at the two last functions in the file. Note: I’m converting QPixmap to cv::Mat. If that would be possible without first converting to a QImage, that would also be very nice.

        Best regards
        Bastian

  10. Thanks for your article. But I have a problem when I convert Qimage to mat. When I get a image which format is Format_ARGB32, and I want to keep alpha channel.But it always crashes.Is there any problem? Here is my code:

    cv::Mat mat( image.height(), image.width(), CV_8UC4, const_cast(image.bits()), image.bytesPerLine() );

    Could you please help me ?

    • I haven’t been working in this area in a very long time, but I would start by looking at two things:

      1) Make sure the lifetime of the cv::Mat is managed properly so you aren’t working on a temporary (see comments towards the end of the post).

      2) Check that a QImage::Format_ARGB32 can actually be converted to CV_8UC4. I suspect this is incorrect since what you have here is the same code as for QImage::Format_RGB32. If this is wrong it shouldn’t crash though – it should give you an incorrect image.

  11. Thanks for your article 🙂 It helped me a lot 🙂
    To which QFormat would you convert a CV_32FC1 image ? Format_Indexed8?
    And how would you do it, would you first convert CV_32FC1 to CV_8UC1 using opencv function convertTo(), or do something else ?
    Thanks again 🙂

    • You’re welcome! Glad you found it useful.

      Keeping in mind that I’m not an expert on this, I think your proposal seems like a good solution.

      First convert to CV_8UC1 using OpenCV’s convertTo(), then convert to QImage::Format_Indexed8 as I do in ASM::cvMatToQImage() above.

  12. This is the best walk-through online! Most of the stuff on the internet regarding QT and OpenCV is messy, thanks! It may be out of the scope of what you’re doing, but could you discuss the image formats OpenCV uses? Im new to image processing and im also learning both QT and OpenCV at the same time. One thing I was wondering is, what does it mean to have 1 or 2 or 3 or 4 channels?? The data representation used by OpenCV is pretty clear, CV_{U|S|F}C(number of channels)…but the last bit of information, the channels, and what they mean to me, the guy whose doing the processing on these images, isnt too clear.

    • alem:

      You can think of the channels as the individual components that are combined to create an image. In the cases we’re dealing with here, they are combined with the bit size to define the format of the image – the way the bits are laid out in memory.

      The simplest is 1 channel. This is simply a greyscale (or black & white) image.

      As far as I know, 2 channel does not exist… If it does, it’s not useful for our discussion 🙂

      The most common 3 channel format is RGB, so it will have one channel for red, one for blue, and one for green. Combining them will create your colour image.

      Another common 3 channel format is HSV (hue, saturation, value) which stores the colour info differently.

      RGB can be extended to RGBA – where A is alpha – to create a 4 channel image. CMYK (cyan, magenta, yellow, and black) is another 4 channel image format.

      What my switch statements in the code above are doing are mapping the QImage formats to the OpenCV formats and vice-versa.

      With OpenCV you almost certainly need to know the specifics of the format you’re dealing with if you’ll be doing any pixel manipulation. So when you are processing an image, you will need to know whether you’re working with greyscale, RGB, RGBA, HSV, or CMYK.

      On the Qt side of the fence, they don’t talk in channels, they just define the format including its bit layout explicitly, like QImage::Format_RGB888. Almost all the QImage functions are high-level enough that you probably don’t care what the internal format is.

      Some more info can be found on the Wikipedia page (but I find it’s not as clear as it could be).

      Hope that helps and doesn’t further confuse the issue.

      Thank you for taking the time to comment!