Face Detection in Android with Google Play Services

This is the first installment of a series of articles on SitePoint covering the API libraries that constitute Google Play Services. Each article in the series will be self contained, so you don’t have to read the others to follow along.

The Google Play Services APIs we’ll cover include:

Face Detection

Barcode Scanner

Location and Context

Analytics

AdMob

Maps

Identity

Play Games Services

Introduction to the Mobile Vision API

The recent release of Google Play Services 7.8 added new Mobile Vision APIs which provide a framework for finding objects in photos and videos. The framework includes detectors, which locate and describe visual objects in images or video frames, and an event driven API that tracks the position of those objects in video.

--ADVERTISEMENT--

Currently, the Mobile Vision API includes face and barcode detectors, which can apply separately or together. In this article, we’ll look at the Face API which can find human faces in images, live streams and videos. It can find and track positions of facial landmarks such as eyes, nose, and mouth, and provide information about the state of facial features e.g. are the eyes open, is there a smile, etc.

Note

To run the app, you should have a device that runs Android 2.3 (Gingerbread) or higher and includes the Google Play Store. This means that unless you have installed Google Apps on your virtual device, you won’t be able to test the code on the emulator. The default Android emulator doesn’t support installing Google Apps on the device, but if you use an emulator like Genymotion you will be able to (despite the team discontinuing support for Google Apps).

Face Detection

Face detection is not a new feature on Android. Previously it was possible to perform face detection using the FaceDetector.Face API introduced in Android API level 1. The face detection framework provided by the Mobile Vision API is an improvement of this API.

You should note that at the moment, the Google Face API only provides functionality for face detection and not face recognition. It can detect the presence of human faces in an image or video, but it cannot determine if two faces correspond to the same person. It can infer from changes in the position frame to frame that faces in consecutive frames of video are the same face. If a face leaves the field of view and re-enters, it isn’t recognized as a previously detected face.

To get started with the API, create a new project named FaceDetectionDemo select the defaults in the Next screens of the Android Studio project creation wizard, selecting API level 15. The Mobile Vision API is backward compatible for all devices that support Google Play Services (i.e. Android 2.3 Gingerbread and higher).

Add an id to the TextView that is in the generated activity_main.xml file as shown.

android:id="@+id/face_count"

Run the app and you should see the number of faces found in the image.

Actually, if you ran the app, you probably got a wrong result from the above. We’ll soon see why, first lets look at the code just run.

We first create a Bitmap from the image labelled image01.jpg. Then we create a detector for finding faces in a supplied image. The FaceDetector is created via an associated builder class, specifying the relevant detection options. In the above example we set tracking to false which is recommended for detection with unrelated individual images (as opposed to video or a series of consecutively captured still images). For detection of unrelated individual images, this will give a more accurate result. For detection on consecutive images (e.g. live video), tracking gives a more accurate (and faster) result.

The face detector can run with low resolution images (e.g. 320×240). Running face detection on low resolution images is faster than high resolution images. Keep in mind that at lower resolutions, the face detector may miss some smaller faces due to having less detail.

We then create a frame from the bitmap and run face detection synchronously on the frame.

Next we change the text of the TextView in the view layout with number of faces detected.

When finished with the detector, release it to free native resources.

The first time an app using Face API runs on a device, GMS downloads a native library to the device. Usually this completes before the app runs for the first time. But if that download has not yet completed, then the call to detect faces will not detect any faces. So you might have run the app and got an output of 0 detected faces.

You can use isOperational() to check if the required native library is available and take some action if it isn’t, like notify the user. You could, for example, do something like the below.

Still in onCreate(), change the statement that opens the resource file as shown.

InputStream stream = getResources().openRawResource(R.raw.image02);

Here we use image02.jpg instead of image01.jpg. The same image could be used, but since image01.jpg contains more faces, there will be a lot of markings when we mark all detected landmarks. image02.jpg contains one face and so it will be better suited for this use.

The above iterates through the landmarks on a detected face and draws a mark at the point of the landmark.

Call the above method in onDraw() instead of the drawFaceRectangle() function.

drawFaceAnnotations(canvas, scale);

Run the app and you should see the different Landmarks of the detected face. The Landmarks mark the eyes, nose, mouth and cheeks.

Classification

Classification determines whether a certain facial characteristic is present. For example, a face can be classified with regards to whether its eyes are open or closed. Another example is whether the face is smiling or not.

To detect facial characteristics, change the statement that creates a FaceDetector in MainActivity as follows.

We use drawText() to draw text onto the canvas. We use face.getIsSmilingProbability() which returns a value between 0 and 1 giving a probability that the face is smiling

Call this function in onDraw() instead of calling drawFaceAnnotations().

detectFaceCharacteristics(canvas, scale);

Run the app and you should see the smiling probability of the different faces. As you can see, the grinning face on the top right has the highest probability.

Face Tracking

Face tracking extends face detection to video sequences. Any face appearing in a video for any length of time can be tracked. That is, faces detected in consecutive video frames can be identified as being the same person. Note that this is not a form of face recognition, this mechanism just makes inferences based on the position and motion of the face(s) in a video sequence. We won’t look at this in this tutorial, but you can read the official guides for more information.

Conclusion

We’ve looked at some of the features of the Face API. To find out more about the Mobile Vision API, you can find code for sample apps that use the API here.