How Facial Recognition Tech Could Help Trace Terrorism Suspects

Face match: A crowd of spectators near the site of the first bomb blast.

The FBI appealed to the public Thursday for help identifying two men shown in pixilated photos and video footage who are suspected of involvement in Monday’s bomb attacks in Boston.

The two men, now identified as Tamerlan Tsarnaev and Dzhokhar Tsarnaev, brothers originally from Chechnya, were involved in a dramatic shootout with police in Cambridge, Massachusetts, on Thursday night. The pair robbed a 7/11 and killed an MIT police officer before hijacking a car and engaging police in pitched battles in the suburb of Watertown. The older of the two men, Tamerlan Tsarnaev, was killed during a shootout with police while his younger brother, Dzhokhar, remains on the run as of Friday morning.

Experts say the FBI could have used images from the scene of Monday’s bombing—together with facial recognition software—to search through identity databases. The approach is likely to become more common in the future as new technology makes using facial recognition on surveillance and bystander imagery more reliable.

Deploying facial recognition software in the Boston investigation isn’t straightforward because the images available are very different from the evenly lit, frontal, passport-style photos stored in law enforcement databases. Such mug shots can be matched with about 99 percent accuracy, says Anil Jain, a professor at Michigan State expert who works on facial recognition, a figure that falls to about 50 percent for images of good quality but with added complications such as a person wearing a hat or glasses.

Attempting facial recognition on images like those released by the FBI Thursday is out of the question, Jain says. However, there may be other images and videos available that contain a better view that could be high quality enough, he says. “You could search all the other images based on clothing,” he says, “[and then] you could locate the same person and collect multiple images.”

Such a search could be done manually, but the FBI also likely has access to software that could speed the process by matching images and video footage that show the same scene or area, says Brian Martin, director of biometric research at MorphoTrust, a company that provides facial recognition technology to the FBI and the U.S. Department of Defense.

An image found amongst the many provided by witnesses and surveillance cameras wouldn’t have to be a perfect mug shot, either, says Martin. “There are numerous techniques to clean up an image,” he says. “You could improve the resolution, correct shadows, or rotate the pose of the face.”

Facial recognition algorithms struggle once a person’s face is turned by more than about 20 degrees, says Martin, but software from his company can correct turns of up to 45 degrees. It does this using built-in knowledge of facial geometry and by filling in the hidden side of a face by copying from the visible side.

Still, even if the FBI is able to find a photo to submit to its facial recognition search system, it won’t return just a single name, even if the person is on file. “With this type of situation you’re trying to generate leads,” says Martin, and agents would expect to manually screen a list of tens or hundreds of possible matches.

The FBI and other law enforcement and security agencies will see a growing opportunity to use facial recognition, as the volume and quality of surveillance camera and bystander imagery from cell phones grows. That trend is encouraging, and sometimes directly funding, companies like MorphoTrust and academics like Jain to work on technologies that could see facial recognition used routinely in criminal investigations both major and minor.

Martin’s team at MorphoTrust is working on making software better able to handle the kind of images that appear in surveillance and bystander data. “In a case like this you don’t typically get a good frontal view that’s well-lit,” says Martin. “We’re trying to push the boundaries so you can compensate for things like a face more than 45 degrees off to the side.”

With funding from the FBI, Jain at Michigan State is working on software for matching faces from low quality surveillance video against existing image databases. Another project is developing a system that can search a database of faces for a match with a sketch drawn by a forensic artist or a partial or outdated photo.

Other researchers are testing more fundamental rethinks of facial recognition algorithms. Marios Savvides, an assistant professor at Carnegie Mellon and director of its Cylab Biometrics Center, has developed technology that can create an accurate high-resolution image of a face from a poor resolution one, and which can correct for faces turned partly away from the camera.

Savvide’s software matches faces turned to the side by working out what the faces on file would look like when turned by the same angle, and also by tracking features that are still visible. That avoids having to assume the hidden side of a face matches the visible one, as with in MorphoTrust’s technology, says Saviddes.

“Many cases today, like in Boston and other crimes, law enforcement have low-resolution, off-angle images they can’t do anything with,” says Saviddes, “but we can change that.”