OpenCV Viola & Jones object detection in MATLAB

15 February 2010

In image processing, one of the most successful object detectors devised is the Viola and Jones detector, proposed in their seminal CVPR paper in 2001. A popular implementation used by image processing researchers and implementers is provided by the OpenCV library. In this post, I’ll show you how run the OpenCV object detector in MATLAB for Windows. You should have some familiarity with OpenCV and with the Viola and Jones detector to work through this tutorial.

Steps in the object detector

MATLAB is able to call functions in shared libraries. This means that, using the compiled OpenCV DLLs, we are able to directly call various OpenCV functions from within MATLAB. The flow of our MATLAB program, including the required OpenCV external function calls (based on this example), will go something like this:

cvLoadHaarClassifierCascade: Load object detector cascade

cvCreateMemStorage: Allocate memory for detector

cvLoadImage: Load image from disk

cvHaarDetectObjects: Perform object detection

For each detected object:

cvGetSeqElem: Get next detected object of type cvRect

Display this detection result in MATLAB

cvReleaseImage: Unload the image from memory

cvReleaseMemStorage: De-allocate memory for detector

cvReleaseHaarClassifierCascade: Unload the cascade from memory

Loading shared libraries

The first step is to load the OpenCV shared libraries using MATLAB’s loadlibrary() function. To use the functions listed in the object detector steps above, we need to load the OpenCV libraries cxcore100.dll, cv100.dll and highgui100.dll. Assuming that OpenCV has been installed to "C:\Program Files\OpenCV", the libraries are loaded like this:

You will get some warnings; these can be ignored for our purposes. You can display the list of functions that a particular shared library exports with the libfunctions() command in MATLAB For example, to list the functions exported by the highgui library:

The above output shows that MATLAB has successfully loaded the cascade file and returned a pointer to an OpenCV CvHaarClassifierCascade object.

Prototype M-files

We could now continue implementing all of our OpenCV function calls from the object detector steps like this, however we will run into a problem when cvGetSeqElem is called. To see why, try this:

libfunctions('cxcore100', '-full')

The -full option lists the signatures for each imported function. The signature for the function cvGetSeqElem() is listed as:

[cstring, CvSeqPtr] cvGetSeqElem(CvSeqPtr, int32)

This shows that the return value for the imported cvGetSeqElem() function will be a pointer to a character (cstring). This is based on the function declaration in the cxcore.h header file:

CVAPI(char*) cvGetSeqElem( const CvSeq* seq, int index );

However, in step 5.1 of our object detector steps, we require a CvRect object. Normally in C++ you would simply cast the character pointer return value to a CvRect object, but MATLAB does not support casting of return values from calllib(), so there is no way we can cast this to a CvRect.

The solution is what is referred to as a prototype M-file. By constructing a prototype M-file, we can define our own signatures for the imported functions rather than using the declarations from the C++ header file.

This will automatically generate a prototype M-file named proto_cxcore.m based on the C++ header file. Open this file up and find the function signature for cvGetSeqElem and replace it with the following:

An example face detector

We now have all the pieces ready to write a complete object detector. The code listing below implements the object detector steps listed above to perform face detection on an image. Additionally, the image is displayed in MATLAB and a box is drawn around any detected faces.

@Vladimir – Thanks, I’ve never tried using the .NET interface (I’m still using MATLAB R2006a). If you were to take that approach, I suppose you would want to first create a .NET wrapper around the OpenCV libraries. Take a look at OpenCVDotNet – I don’t think the project is still active, but might give you some ideas anyway.

@dong – cvLoadImage returns a null if the image fails to load. There can be lots of reasons – the filename may be incorrect, the image may be invalid, etc. I’d suggest you should try getting a simple C++ version of your code working first, and then port it over to MATLAB.

The idea of adding to Matlab some of the functionalities of OpenCV is awesome, but I can’t find the libraries in my OpenCV folder I should share in Matlab. I’m using OpenCV 2.2 for Windows and Matlab 7.11, and I am afraid you are using another version and maybe for Linux. Which version are you using? Should I use Windows or Linux?

From other Matlab/Flash project, I got cxcore100.dll and cv100.dll, but I don’t have the header files of these dlls, so I can’t shared the libraries in Matlab. Can I follow your guide and succeed if you send to me these files?

Thanks – I used Windows but I used an older version of OpenCV than you (I originally wrote this code in 2008). I have just downloaded version 2.2 for Windows and all of the DLL files seem to have been reorganised and renamed. You could try downloading a pre-2.0 version of OpenCV (older versions are available on Sourceforge, I think I used 1.1pre1), or you could try to work out what the relevant libraries and functions for face detection are in the latest version. I’d be interested to hear if you get it working with version 2.2!

A very informative article. Thanks. Is it possible to convert cvImage to a MATLAB image to display inside MATLAB with imshow()? On the other hand, how to convert a MATLAB image to cvImage to be process by any OpenCV function through Calllib()? How to pass CV_RGB(r, g, b) (that is a macor/in-line function) as an argument in Calllib()?

Take a look at the documentation for the IplImage structure. The raw image data is stored in the imageData variable and with a bit of fiddling it should be possible to convert this raw data into a MATLAB matrix. Be warned though – JPEGs come in a lot of different varieties (colour depth, number of channels, interlaced vs progressive, etc), so you might find that any code you write to process the raw images may break when you try it out with JPEGs from different sources. The OpenCV FAQ contains some code showing how to decode raw data which might be of help.

As to your question about C/C++ macros, these cannot be used in MATLAB unfortunately.

loadlibrary(…
fullfile(opencvPath, ‘bin\cxcore100.dll’), …
fullfile(opencvPath, ‘cxcore\include\cxcore.h’), …
‘mfilename’, ‘proto_cxcore’);
This code was run, but proto_cxcore.m was not generated. this code is not error! Can you explain it? thanks!

I liked the article
but when I tried to apply the code
I downloaded the opencv binary file and installed and built it using VS2010 as the instructions
but I can’t these three files
cxcore100.dll, cv100.dll and highgui100.dll

That looks like you are using a newer version of OpenCV (since your build directory contains “python” and the Python bindings only came in version 2, AFAIK). I originally wrote this code using OpenCV version 1.1pre1. However in the latest version of OpenCV, all of the libraries seem to have been reorganised and renamed. To get this working you could either download version 1.1pre1, or try to figure out what all of the libraries and functions have been renamed to in the latest version of OpenCV.

Warning: Warnings messages were produced while parsing. Check the functions you
intend to use for correctness. Warning text can be viewed using:
[notfound,warnings]=loadlibrary(…)
> In loadlibrary at 399
Error loading library intermediate output follows.
The actual error is at the end of this output.
*********

Found on line 853 of input from line 393 of file C:\Program Files\OpenCV\cxcore\include\cxtypes.h

Type ‘_IplTileInfoPtr’ was not found. Defaulting to type voidPtr.

Found on line 853 of input from line 393 of file C:\Program Files\OpenCV\cxcore\include\cxtypes.h

Eval of const expression 32 ][ 2 failed with error Unmatched right square bracket at (eval 6) line 1, at end of line
syntax error at (eval 6) line 1, near “32 ]”
Missing right curly or square bracket at (eval 6) line 2, at end of line
.
Found on line 1261 of input from line 811 of file C:\Program Files\OpenCV\cxcore\include\cxtypes.h

Type ‘CvMatND’ was not found. Defaulting to type error.

Found on line 1261 of input from line 811 of file C:\Program Files\OpenCV\cxcore\include\cxtypes.h

Type ‘CvGraphVtxPtr’ was not found. Defaulting to type voidPtr.

Found on line 1697 of input from line 1292 of file C:\Program Files\OpenCV\cxcore\include\cxtypes.h

Type ‘CvMatPtr’ was not found. Defaulting to type voidPtr.

Found on line 2298 of input from line 133 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatPtr’ was not found. Defaulting to type voidPtr.

Found on line 2305 of input from line 140 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatPtr’ was not found. Defaulting to type voidPtr.

Found on line 2305 of input from line 140 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatPtr’ was not found. Defaulting to type voidPtr.

Found on line 2308 of input from line 143 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatPtrPtr’ was not found. Defaulting to type voidPtrPtr.

Found on line 2312 of input from line 147 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatPtr’ was not found. Defaulting to type voidPtr.

Found on line 2357 of input from line 192 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatPtr’ was not found. Defaulting to type voidPtr.

Found on line 2357 of input from line 192 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatPtr’ was not found. Defaulting to type voidPtr.

Found on line 2362 of input from line 197 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatPtr’ was not found. Defaulting to type voidPtr.

Found on line 2362 of input from line 197 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatPtr’ was not found. Defaulting to type voidPtr.

Found on line 2369 of input from line 204 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatPtr’ was not found. Defaulting to type voidPtr.

Found on line 2369 of input from line 204 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatPtr’ was not found. Defaulting to type voidPtr.

Found on line 2380 of input from line 215 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatPtr’ was not found. Defaulting to type voidPtr.

Found on line 2380 of input from line 215 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatPtr’ was not found. Defaulting to type voidPtr.

Found on line 2390 of input from line 227 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatPtr’ was not found. Defaulting to type voidPtr.

Found on line 2390 of input from line 227 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatNDPtr’ was not found. Defaulting to type voidPtr.

Found on line 2399 of input from line 236 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatNDPtr’ was not found. Defaulting to type voidPtr.

Found on line 2402 of input from line 239 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatNDPtr’ was not found. Defaulting to type voidPtr.

Found on line 2406 of input from line 243 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatNDPtr’ was not found. Defaulting to type voidPtr.

Found on line 2406 of input from line 243 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatNDPtr’ was not found. Defaulting to type voidPtr.

Found on line 2415 of input from line 252 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatNDPtr’ was not found. Defaulting to type voidPtr.

Found on line 2415 of input from line 252 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatNDPtr’ was not found. Defaulting to type voidPtr.

Found on line 2466 of input from line 303 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatNDPtr’ was not found. Defaulting to type voidPtr.

Found on line 2477 of input from line 315 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatPtr’ was not found. Defaulting to type voidPtr.

Found on line 2543 of input from line 387 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatPtr’ was not found. Defaulting to type voidPtr.

Found on line 2543 of input from line 387 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatPtr’ was not found. Defaulting to type voidPtr.

Found on line 2559 of input from line 412 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatPtr’ was not found. Defaulting to type voidPtr.

Found on line 2559 of input from line 412 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatPtr’ was not found. Defaulting to type voidPtr.

Found on line 2808 of input from line 680 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatPtr’ was not found. Defaulting to type voidPtr.

Found on line 2808 of input from line 680 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatPtr’ was not found. Defaulting to type voidPtr.

Found on line 2834 of input from line 707 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatPtr’ was not found. Defaulting to type voidPtr.

Found on line 2834 of input from line 707 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvMatPtr’ was not found. Defaulting to type voidPtr.

Found on line 2839 of input from line 712 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvFileNodePtr’ was not found. Defaulting to type voidPtr.

Found on line 3699 of input from line 1619 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvFileNodePtr’ was not found. Defaulting to type voidPtr.

Found on line 3705 of input from line 1625 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvFileNodePtr’ was not found. Defaulting to type voidPtr.

Found on line 3705 of input from line 1625 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvFileNodePtr’ was not found. Defaulting to type voidPtr.

Found on line 3710 of input from line 1630 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvFileNodePtr’ was not found. Defaulting to type voidPtr.

Found on line 3710 of input from line 1630 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvFileNodePtr’ was not found. Defaulting to type voidPtr.

Found on line 3758 of input from line 1678 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvFileNodePtr’ was not found. Defaulting to type voidPtr.

Found on line 3770 of input from line 1690 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvFileNodePtr’ was not found. Defaulting to type voidPtr.

Found on line 3778 of input from line 1698 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvFileNodePtr’ was not found. Defaulting to type voidPtr.

Found on line 3782 of input from line 1702 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h

Type ‘CvFileNodePtr’ was not found. Defaulting to type voidPtr.

Found on line 3785 of input from line 1705 of file C:\Program Files\OpenCV\cxcore\include\cxcore.h
*********
??? Error using ==> loadlibrary at 480
There was an error running the loader mfile. Use the mfilename option
to produce a file that you can debug and fix if needed. Please report
this error to the MathWorks so we can improve this function.

I have just started using opencv,completely new to this language.It seems the opencv which i installed (OpenCV-2.3.1-win-superpack) neither has a bin folder nor the i.e bin\cxcore100.dll . Moreover,there are several versions of cxcore.h which are present in C:\Program Files\OpenCV\opencv\include and C:\Program Files\OpenCV\opencv\build .

Hey alister, I am using OpenCV2.1. Can you please point out the location your ‘proto_cxcore’ is made? Everything runs fine till that point and no error is given. Get the feeling that it is created somewhere.

Hey it’s me again. To anybody having the same problem as me, I finally figured out what the problem was. You just have to make sure to unload cxcore100 library (cxcore210d in my case) before running the proto_cxcore line.

hello, I am using opencv 242 and matlab 2011b. When i try to load the library, it loads. But when I try to call (libfunctions) it I get an error. It does not find it!
I have no idea why this is happening. Would you please help me.

load
Warning: Warnings messages were produced while parsing. Check the functions you intend to use for
correctness. Warning text can be viewed using:
[notfound,warnings]=loadlibrary(…)
> In loadlibrary at 347
In load at 4
Warning: Function D:\tracking\load.m has the same name as a MATLAB builtin. We suggest you rename the
function to avoid a potential name conflict.
> In loadlibrary>@()cd(savedir) at 299
In onCleanup>onCleanup.delete at 61
In loadlibrary at 418
In load at 4
Error using loadlibrary (line 421)
There was an error loading the library “C:\OpenCV\bin\cxcore100.dll”
C:\OpenCV\bin\cxcore100.dll is not a valid Win32 application.

Error in load (line 4)
loadlibrary(…

Caused by:
Error using loaddefinedlibrary
C:\OpenCV\bin\cxcore100.dll is not a valid Win32 application.