Introduction

Image alignment is the process of matching one image called template (let's denote it as T) with another image, I (see the above figure). There are many applications for image alignment, such as tracking objects on video, motion analysis, and many other tasks of computer vision.

In 1981, Bruse D. Lucas and Takeo Kanade proposed a new technique that used image intensity gradient information to search for the best match between a template T and another image I. The proposed algorithm has been widely used in the field of computer vision for the last 20 years, and has had many modifications and extensions. One of such modifications is an algorithm proposed by Simon Baker, Frank Dellaert, and Iain Matthews. Their algorithm is much more computationally effective than the original Lucas-Kanade algorithm.

In the Lucas-Kanade 20 Years On: A Unifying Framework series of articles, Baker and Matthews try to classify image alignment algorithms and divide them into two categories: forwards and inverse, depending on the role of the template T. Also, they classify algorithms as additive or compositional, depending on whether the warp is updated by adding parameters increment or by composing transformation matrices. Following this classification, the Lucas-Kanade algorithm should be called a forwards additive algorithm, and the Baker-Dellaert-Matthews algorithm should be called an inverse compositional algorithm.

In this article, we will implement these two algorithms in C language using the Intel OpenCV open-source computer vision library as the base for our implementation. We also will try to compare these two algorithms, and will see which one is faster.

Why is this article needed? First of all, after writing this article, I have a better understanding of image alignment algorithms myself (at least I hope so). There are a lot of articles that provide tons of information on image alignment. Most of them are oriented on advanced readers, not on beginners. Among those articles, I've found several ones that, from my point of view, describe difficult things more simply. But what I really missed is code examples to experiment with. So, the other two purposes of this article are a relatively simple explaination of how image alignment works and providing sample source code.

Who will be interested in reading this? I think, if you remember some mathematics from your university program, are familiar with C or C++, interested in learning how to track an object on video, and have a little patience, you can read this.

Compiling Example Code

It is very easy to run the sample – just download demo_binary.zip and run the executable file.

But to experiment with the code, you will need to compile it yourself. First of all, you need Visual Studio .NET 2005 to be installed. Visual C++ Express Edition also fits our needs.

The next step is to download and install the OpenCV library from here. Download OpenCV_1.0.exe and run it on your computer.

The auxfunc.h and auxfunc.cpp contain auxiliary image warping functions. The forwadditive.h and forwadditive.cpp files contain an implementation of the Lucas-Kanade algorithm. The invcomp.h and invcomp.cpp contain an implementation of the inverse compositional algorithm. And, the main.cpp contains the main function.

What does it do?

Saying in a few words, this program tries to align the template (the small butterfly image) with the big image (butterfly on the flower). It is not limited to using a butterfly as the template. For example, you can use a face image as a template and determine where the face moves on the next frame of the video sequence.

During the process of alignment, we perform warping operations on the template. We assume that there are three allowed operations: in-plane rotation of the template, and its 2D translation in X and Y directions. These operations are called the allowed set of warps.

We define the allowed set of warps by defining the warp matrix, . The matrix W depends on the vector of parameters, p=(wz, tx, ty). Here tx and ty are translation components and wz is the rotation component.

It is even possible to use much more complex sets of warps, for example 3D translation and rotation, scaling, and so on. But, here we use a relatively simple set of warps for simplicity.

Now, about the logic of the application.

After you compile and run the demo, you will see a console window, where we will display the diagnostic information. Enter the components of the vector p=(wz, tx, ty).

After that, you can see two windows containing the template T and the incoming image I. You also can see a white rectangle on the image I. It is an initial approximation of the butterfly's position. We just assume that the butterfly is somewhere near that area.

Press any key to start the Lucas-Kanade image alignment algorithm. This algorithm tries to search where the butterfly is located, iteratively minimizing the difference between the template and the image I. In the console window, you will see how it works. You can see the current iteration number, parameter increments, and the mean error value.

After the process of error minimization converges, the summary is written to the console. The summary includes the algorithm name, the calculation time, the resulting mean error , and the parameters approximation. You can also see the white rectangle that displays how the template is aligned on the image I.

Now, press any key to run the same process, but using an inverse compositional image alignment algorithm. You can see the current iteration number and the mean error value.

The summary is displayed for the inverse compositional algorithm. Press any key to exit the program.

In OpenCV, matrices are stored in CvMat structures. Let's define a pointer to our warp matrix W, and initialize it with zero.

// Here we will store our warp matrix.
CvMat* W = 0; // Warp W(p)

Now, create two windows for the template T and for the image I. OpenCV has the basic support for the GUI. The GUI support is implemented as a part of OpenCV called the highgui library. The cvLoadImage() function allows to load an image from a JPG file.

Now, allocate memory for the images using the cvCreateImage() function. The first parameter is the size of the image in pixels, the second one is the depth, and the third is the number of channels. RGB images have a depth IPL_DEPTH_8U and three channels. IPL_DEPTH_8U means to use an unsignedchar as the image element. IPL_DEPTH_16S means to use a shortint as the image element.

We also need to convert our photo to gray-scale format, because image alignment algorithms work with gray-scale images only.

Now, let's allocate memory for our warp matrix W using the cvCreateMat() function. The first two parameters define the matrix size (3x3) and the last one is the type of the matrix element. CV_32F means we will use float as the matrix element type.

First of all, let's include the headers for the functions we will use.

#include<spanclass="code-keyword"><stdio.h></span>#include<spanclass="code-keyword"><time.h></span>#include<spanclass="code-keyword"><cv.h> // Include header for computer-vision part of OpenCV.</span>#include<spanclass="code-keyword"><highgui.h> // Include header for GUI part of OpenCV. </span>#include<spanclass="code-string">"auxfunc.h" // Header for our warping functions. </span>

Let's define the function that will implement the forwards additive (Lucas-Kanade) algorithm. The template image pImgT, the template bounding rectangle omega, and another image pImgI represent the parameters for the algorithm. In OpenCV, the images are stored as IplImage structures, and a rectangle can be defined using the CvRect structure.

Now, let's allocate memory for our matrices using the cvCreateMat() function. The first two parameters define the matrix size (3x3 or 3x1), and the last one is the type of the matrix element. CV_32F means we will use float as the matrix element type.

Allocate memory for images using the cvCreateImage() function. The first parameter is the size of image in pixels, the second one is the depth, and the third is the number of channels. Gray-scale images have a depth IPL_DEPTH_8U and one channel. IPL_DEPTH_16S means to use shortint as the image element.

// Get current time. We will use it later to obtain total calculation time.
clock_t start_time = clock();

Pre-compute the image gradient . Note that cvSobel() function produces an enhanced image gradient (becase we use 3x1 or 1x3 Gaussian kernel), so we should call cvConvertScale() to normalize the gradient image.

Implementing an Inverse Compositional Image Alignment Algorithm

In an inverse compositional algorithm, the template T and the image I revert their roles.

Pre-compute:

Evaluate the gradient of image T(x).

Evaluate the Jacobian at (x;0).

Compute the steepest descent images, .

Compute the Hessian matrix,

Iterate:

Warp I with W(x;p) to compute I(W(x,p)).

Compute the error image I(W(x;p))-T(x).

Warp the gradient with W(x;p).

Compute

Compute

Update the parameters

until .

The code of this method is located in invcomp.h and invcomp.cpp. Let's see what invcomp.cpp contains.

Include all required headers:

#include<spanclass="code-keyword"><stdio.h></span>#include<spanclass="code-keyword"><time.h></span>#include<spanclass="code-keyword"><cv.h> // Include header for computer-vision part of OpenCV.</span>#include<spanclass="code-keyword"><highgui.h> // Include header for GUI part of OpenCV.</span>#include<spanclass="code-string">"auxfunc.h" // Header for our warping functions.</span>

Here we have implementation of Baker-Dellaert-Matthews (inverse-compositional) algorithm:

IPL_DEPTH_16S means to use short as gradient image element. Here we use short, because cvSobel produces not normalized image, and uchar overflow may occur. To normalize the gradient image we use cvConvertScale().

Bilinear Interpolation

In both algorithms we use bilinear pixel interpolation. Interpolation helps to calculate the intensity of a transformed pixel with better accuracy. Transformed pixels usually have non-integer coordinates, so by interpolating their intensities we take intensities of neighbour pixels into account. This also helps to avoid mean error oscillations and improve overall minimization performance.

Conclusion

Image alignment has many applications in the field of computer vision, such as object tracking. In this article, we described two image alignment algorithms: the Lucas-Kanade forwards additive algorithm and the Baker-Dellaert-Matthews inverse compositional algorithm. We also saw the C source code for these algorithms.

The calculation time for the forwards additive algorithm is 0.157 sec. The calculation time for the inverse compositional algorithm is 0.078 sec. As we see, the inverse compositional algorithm works faster, because it is more computational effective.

See Also

You may also want to read Image Alignment Algorithms - Part II, where we implement the forwards compositional and inverse additive image alignment algorithms and compare all four implemented algorithms.

About the Author

Comments and Discussions

I am looking for image alignment algorithm for merging different exposure images to HDR images. There is famous MTB for it but it needs too much memory and i think it is slow. I was wondering how does your code works compared to MTB alignment and would be work on different exposure images too?