A blog about anything a software engineer might encounter during his programming efforts.

26 October 2012

Overlay transparent image in OpenCV

It seems OpenCV does not provide very much support for transparent images. It has images with 4 channels, which are treated as BGRA images, with the fourth channel being the Alpha channel. It also supports saving these images using the PNG format, yet it does not support properly displaying these images in highgui, nor does it support combining these images.

I have looked a while for a way to combine a (BGR) background image with a (BGRA) foreground image, with semi-transparent images, but was unable to find one. So, I wrote my own and since I couldn't find a solution on the internet, I'll bet I can save some people some time by sharing my implementation, which is as follows:

// start at the row indicated by location, or at row 0 if location.y is negative.for(int y = std::max(location.y , 0); y < background.rows; ++y) {int fY = y - location.y; // because of the translation // we are done of we have processed all rows of the foreground image.if(fY >= foreground.rows)break; // start at the column indicated by location,

// or at column 0 if location.x is negative.for(int x = std::max(location.x, 0); x < background.cols; ++x) {int fX = x - location.x; // because of the translation.// we are done with this row if the column is outside of the foreground image.if(fX >= foreground.cols)break; // determine the opacity of the foregrond pixel, using its fourth (alpha) channel.double opacity = ((double)foreground.data[fY * foreground.step + fX * foreground.channels() + 3])

/ 255.;

// and now combine the background and foreground pixel, using the opacity,

The output image will always be of the same size as the background image, in BGR colour space. The position parameter determines how the foreground is placed on top of the background. A position of (100, -50) will move the foreground 100 pixels to the right and 50 pixels up.

It might not be the best solution as it was written in a hurry, so please feel free to improve.

Explanation of the code:

We first copy the background to the output, so we can continue with only the pixels where foreground and background overlap.

We start at the first scanline where they overlap, being either the top row of the background (y=0) or the top row of the foreground (y = position.y), so we take the maximum of these two. We do the same for the column value (x). We use fX and fY (foreground-x and foreground-y) as the current coordinates in the foreground image.

We determine the opacity of the foreground image using its fourth (alpha) channel. If it is > 0, we combine the pixels of the background and foreground for all channels.

We are done if we passed all overlapping pixels and output will now contain an image of the background, overlayed with the transparent foreground image.

Could you check if both images are read correctly by showing them in a window before creating the overlay? I have seen reports by people having problems reading png files because of some issues with libpng.

Hi Michael, As a posted earlier that you code is working correctly but i have got an issue, Actually i am inserting the BGRA overlay google onto the Video , in my case openCV returns BGRA Image to process, As a result after you code runs the red Overlay image becomes blue, Actually i am very new to OpenCV and after doing doing lot of changes could not figure our what exactly should i do to make it work correctly. I tried to convert the BGRA image to BGR but in that case overlay image become correct but the background image becomes blue. Can you please help me out..

All images in OpenCV are in BGR or BGRA space by default. If you want to show them in the viewer in OpenCV, only BGR will show the correct colours.You can convert colour spaces using the appropriate method (use cvtColor), but these other colour spaces will not show up correctly when using imshow, as it expects BGR.

Hi Michael, Actually i am using your code in my iPhone application,In iphone using native API i am opening iPhone video Camera then i am processing each of the frame of Video with you code to insert an overlay image. I am getting BGRA format image from Video Camera and the output will be displayed in BGRA colorspace. But as you clearly mentioned that to run properly we need to have BGR image as background and BGRA image as foreground and the output will be in BGR color space. Can you please suggest any link or code which will overlay BGRA background with BGRA foreground and will output BGRA color space.That will solve my problem and it will be great help for me. Waiting for your reply.

From GBRA to GBR basically just means removing the fourth (transparency) channel.But to be honest, I can not imagine your camera producing GBRA images, as the A is the transparency channel. No camera can record transparency as such, so no camera can include it in its imagery.My code uses a non-transparent background (BGR) and overlays it with an image that has transparency (BGRA), producing a completely filled image, in which the whole transparency channel would be wasted, so it's BGR again.

Hi Michael, You were absolutely correct as i my camera is giving image in RGB colorspace. I have just change it to BGR color space and your code works perfectly now. Thank you so much for showing me the right way...

Opacity depends on the png overlay image. Every pixel with Opacity = 255 should appear completely opaque.If it seems to be transparent, I think the A-channel is off somehow. You could split the image and show the opacity (last) channel as a grey value image. Pixels that should not appear transparent should be pure white. If not, I think there is something "wrong" with the png image. You could also try opening the png image with an image editor and saving it again, OpenCV seems to be very picky as far as transparent images go, maybe the png contains some information that is somehow misinterpreted by OpenCV.

Hey, I have recently started out on OpenCV and this is a problem that i am trying to solve. This isn't much literature that I could dig up online about transparency in OpenCV apart from this post. I tried implementing. The code is getting succesfully build, but the exe is crashing once it tries to run.

This is the code that I am using

http://pastebin.com/f5bGzL4B

It would be of really awesome help if you could point out what mistake I am doing.

Assuming your glasses are properly centered in their image, I would suggest finding the center point between the two eyes.If you know the (approximate) locations of the eye centers, you can take the average of those two points, which would be the point exactly between those two. Now you can place your glasses with the center of the transparent image over the point between the two eyes. You will need to calculate the needed translation and pass it to the overlay function.

OpenCV Error: Unspecified error (The function is not implemented. Rebuild the library with Windows, GTK+ 2.x or Carbon support. If you are on Ubuntu or Debian, install libgtk2.0-dev and pkg-config, then re-run cmake or configure script) in cvShowImage, file /home/dinesh/vtr/opencv/modules/highgui/src/window.cpp, line 501terminate called after throwing an instance of 'cv::Exception' what(): /home/dinesh/vtr/opencv/modules/highgui/src/window.cpp:501: error: (-2) The function is not implemented. Rebuild the library with Windows, GTK+ 2.x or Carbon support. If you are on Ubuntu or Debian, install libgtk2.0-dev and pkg-config, then re-run cmake or configure script in function cvShowImage

Have you tried the solutions as specified in the error message? It seems the problem lies not in the rotation code, but in the ShowImage call. I see you are on a Linux or BSD (or othr *nix system) and you probably need certain libraries in order for the highgui libraries to operate properly - such as Gtk+ for instance).

Oh my!!! You are just too true to be good. How do you manage to write and research on such wonderful things? You have inspired me to work harder now. I shall try as much as possible to enjoy life to the fullest and be satiated with the wonderful things that are around me, which I have been unaware of until now.

I know this thread is a bit old but I thought I'd post a copy of the function that modifies an existing Mat instead of creating a new output Mat in case anyone finds it useful: https://gist.github.com/maximus5684/082f8939edb6aed7ba0a

Thank you for your code, i am currently using this code for a proyect in which i had to develop an overlay for drone cameras but i couldn´t make it work with opencv addWeighted method because i wanted an opaque overlay.

You will just have to use this code for every frame in the real-time video. So basically, grab a frame, call the function to do the overlay and then show, send or encode the frame (or whatever you need to do with the frame afterwards).

Hi Michael, i got a problem. Your code runs too slow for my needs.I need to process two images per frame in a 30 fps videooverlayImage is at worst 10ms processing time in my cpuI need to paralelize your method. Is there a way to paralelize it?

You could just run every scanline in a separate thread for instance. So that if you have 8 cores, you can do 8 scanlines at once. Or you could see if you could divide the whole images in 8 equal parts, so for 800 scanlines, you'd get 8 parts of 100 scanlines each and then run those in parallel.