The leJOS 0.9.1 release added the OpenCV vision processing library. What better way to test it than build a face-detecting robot! First step is to figure out how to use the OpenCV libraries on EV3 to detect a face in an image.

It turns out this is really hard to do in practice. Lighting is one of the problems; any strong reflections or highlights in the image and OpenCV will fail to detect the face. The other issue is the processing speed of the EV3; face detection is a CPU-heavy task! The EV3 has a fast processor, but it struggles to do reliable face detection in almost real-time to control a robot.

Are comfortable logging in via ssh to the command line prompt on the EV3

How does face-detection work?

First of all, this code will detect a face-shaped object in an image stream, but it won’t distinguish who it is. That is termed face recognition and is a lot harder to do (trust me – I built these systems for a living!) How does OpenCV detect a face? It takes an image and then appliers a series of ‘classifiers’ to it to extract face-like features from the image. If the image contains enough evidence of face-like features then the OpenCV code returns the coordinates of the bounding rectangle around what it thinks is the face.

OpenCV in leJOS 0.9.1 has two types of face classifiers available: Haar classifiers and LBP feature classifiers. The Haar classifiers are slow but more accurate. In my tests the Haar classifier could reliably detect multiple faces against a bright background, but it was slow! So slow that I was seeing detection times of the order of 1 second per frame (i.e. a frame-rate of 1 fps). This is too slow to reliably control a robot’s motion. The LBP feature classifiers are faster, but less accurate. With an LBP feature classifier I can get on the order of 8-10 fps on the EV3 on a 160×120 image from a webcam. The problem is face detection is very unreliable at this frame-rate, with faces rarely detected even when perfectly lit.

Reading from the webcam

I’m using a cheap webcam plugged into a USB hub plugged into the host port on the EV3. leJOS has native support for webcams since the 0.9.0 release, and with OpenCV that support is now present in the OpenCV library too.

We start by initialising and opening the camera device. You’ll have to set the image width and height here too. I set it to a very small size of 160×120 pixels. The camera can return larger images, but the bigger the image the slower it is to process on the EV3!

VideoCapture vid = new VideoCapture(0);

vid.set(Highgui.CV_CAP_PROP_FRAME_WIDTH, 160);

vid.set(Highgui.CV_CAP_PROP_FRAME_HEIGHT, 120);

vid.open(0);

System.out.println(“Camera open”);

Once the camera has been opened reading from it is as simple as:

Mat frame = new Mat();

vid.read(frame);

if (!frame.empty()) {

…

}

A Mat is an OpenCV data structure akin to a matrix – it is like a ‘magic bucket’ that you can store almost any data in for OpenCV to process.

Detecting faces

The OpenCV library in Java provides the detectMultiScale() function to detect face images. It takes a number of parameters, and you’ll have to play around with the parameters to get it to work reliably. Face detection is more of an art than a science!

The first thing we want to do is convert the colour image from the webcam into a grayscale image, and then equalise the histogram of the image. While OpenCV can process colour images it seems you’ll get more reliable results from a grayscale image.

I’m using the lbpcascade face detector for this example because it runs quickly on the EV3 CPU and gives reasonable (but not great) accuracy. You will need to copy the file lbpcascade_frontalface.xml from the opencv/data/lbpcascades directory into the same directory on the EV3 as the jar file you are running (usually into /home/lejos/programs).

That’s a lot of parameters. So what do they all mean?

mGrey: This is the input frame that we want to detect a face in. In this case it’s the grayscale image.

faces: A matrix of rectangles; OpenCV returns each face as a rectangle as the bounding box around where it thinks the face is located.

scaleFactor: how much scaling is applied to the image. The lower the number the more accurate the detection, but as a consequence the slower it goes. I started with a value of 2.0 for scale factor, and then reduced it down to 1.8 to get better accuracy but a reasonable detection time.

minNeighbours: controls accuracy. I just leave this at 2

flags: leave this at 0

minSize: the minimum size for a face. I set this to 20×20 to allow small faces to be detected.

maxSize: the maximum size for a face. I set it to 160×120 in case the face takes up the whole image.

Interpreting the output and Drawing faces

Once you’ve run detectMultiScale() you’ll need to find the faces, if any were detected. But how? Recall the output is stored in the faces matrix, so we convert that into an array and see if it has any elements in it. If it does we iterate over each element, which is a rectangle, and then draw a rectangle on the image frame so we can display it in a web browser.

What does the output look like?

I opened up a web browser and connected to my EV3 on the wifi. The images look look like this (yes I usually look like that as I’m trying to watch the screen while holding my head still 🙂

As I mentioned face detection is very sensitive to lighting and background in an image. In this case I had to move the camera so the skylight above my head wasn’t in the image. At night I found that the lights in the room were creating highlights which confused the camera. An ideal lighting setup is to have a dull matt background with soft lights coming from behind the camera at your face.

3 Comments

Steve BarrattNov 01, 2017 @ 20:18:45

Hi Mark
Thanks for this. It is very intriging, but 0.9.1 includes OpenCV so what is required by …
“Have downloaded the opencv-2.4.11 release to copy some needed files onto your EV3: http://opencv.org/downloads.html”
could you reveal what files and where to copy them?
I have the openCV webstreaming working and would dearly love to be able to face track, (I’m a hobbyist not a student or professional programmer)
Eclipse doesnt indicate any missing libraries or dependencies but I get this error when I run it
java.lang.NullPointerException
at barratt.steve.DetectFaceDemo.run(HTTPFaceDetect.java:109)
at barratt.steve.HTTPFaceDetect.main(HTTPFaceDetect.java:70)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at lejos.internal.ev3.EV3Wrapper.invokeClass(EV3Wrapper.java:62)
at lejos.internal.ev3.EV3Wrapper.main(EV3Wrapper.java:46)

Hi Steve – it looks like the openCV native libraries aren’t being found. Are you able to just simply do a test program with
System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
in it to verify that openCV is finding the libraries?