Humans Trump Machines in Facial Recognition

My print column this week examined facial-recognition software, and the confusing numbers about its accuracy. A recent study seemed to suggest that the technology has become highly reliable even at identifying people from a pool of Facebook photos, but a close look at the results shows they are less impressive than they seem.

The U.K. police have learned as much when they tried using such software to identify people filmed by surveillance cameras during the riots last month. A person familiar with the matter said the police only see it as one weapon in the arsenal and not a major one, at that. That is because the success is dependent on already having a picture of a suspect on file, and for the CCTV image to be a good one. Half a face, or a blurry one — which many of these images are — won’t do, this person said. Relying on human identification, from the public and the police, remains the best way to identify suspects.

Simon Lubin, a spokesman for the British Transport Police, said the department has tried the software in the past but declined to confirm whether it was being used in the riots. The trouble, he said, is it isn’t yet very good at picking out one person in a crowd of thousands. “You’ve got thousands of people going through the top of an escalator” in London’s underground train system, Mr. Lubin said. “It’s not sophisticated enough yet to be operationally worthwhile. That may change. Technology is changing all the time.”

For now, experts agree, the technology has a lot of trouble with imperfect images. “Facial recognition only ‘works’ (depending on what your criteria are) under optimal acquisition conditions,” Arnout Ruifrok of the Netherlands Forensic Institute wrote in an email. “As soon as pose, lighting and occlusion are included, the performance quickly deteriorates.”

Alessandro Acquisti, an information-systems expert at Carnegie Mellon University and co-author of the recent study on IDing people from a database of Facebook photos, said the technology has progressed but has a long way to go. “The observation that face recognizers’ ability to detect and recognize faces is improving is quite undeniable,” Acquisti wrote in an email. “The observation that they still significantly underperform humans at that task, however, is also undeniable.”

Paul Schuepp, president and chief executive of the facial-recognition company Animetrics Inc., noted his company’s high accuracy rates with manageable photos but added, “Face recognition gets more and more difficult as you move away from a controlled cooperative face in a camera with studio condition lighting.” He said that such complicating factors as “hats, terrible lighting and dark sunglasses” can “cause face recognition to fail.” It is, he said, “No different than trying to take a fingerprint with a person wearing gloves.”

Angela Sasse, a computer scientist at the University College of London, said that these barriers have important implications for security. “That is a particular problem for the ‘watchlist’ scenarios in facial recognition, where unfavorable lighting means you can’t capture enough of the face at sufficient quality, or you have an old mug shot or low-quality CCTV [closed-circuit television] image as enrollment sample, and the verification sample comes from a CCTV camera.”

Tests that have pitted facial-recognition software against human pattern-matching have found that people do better. Researchers at the University of Massachusetts, Amherst’s Computer Vision Lab, assembled a database of news photographs, with multiple images of each newsmaker, and then other researchers used various tools, either software or people recruited online, to perform matching tasks. People trumped machine dramatically in the tests.

“Essentially, we wanted to find some approximation to the natural distribution of faces a person might see in everyday life,” said Gary Huang, who worked on the study. “So we basically used a collection of news photographs as a proxy for this distribution.”

Separately, I wrote about Wikipedia and its quest for more editors. The English-language site had 3.7 million articles and nearly 40,000 editors in June — impressive numbers but representing slowing growth and a decline, respectively. By way of comparison, I asked officials at Encyclopaedia Britannica how many entries and editors they had. Spokeswoman Orly Telisman did provide some numbers, but first she sent a caveat.

“Comparing the Encyclopaedia Britannica to Wikipedia is not comparing apples to apples — it’s more like comparing apples to chairs,” Telisman said. “We’re just different. Both companies come from very different beginnings, our products and values are different — we are editorially based, they are open-sourced (again, nothing wrong with either execution, but it seems difficult to me to make a one-on-one comparison).”

With all that said, Telisman and Dale Hoiberg, senior vice president and editor in chief of Britannica, said the encyclopedia had 140,000 articles containing over 75 million words, plus links to more than 800,000 articles for further research. The content has increased roughly 10% to 15% annually.

Meanwhile, I asked representatives of Wikimedia Foundation, which runs Wikipedia, about numbers for its new system to rate articles. Many online rating systems yield very high numbers — a grading curve encompassing the whole Web. For instance, the average rating in the Power Reviews network is 4.17 out of five, a spokeswoman said. Wikipedia raters so far are a bit more selective, with an average rating of 3.7. And the tool is fulfilling one of its goals, drawing in people who don’t normally edit articles. “During the feedback tool trial phase over 90% of the raters had never edited before,” said Wikimedia spokesman Jay Walsh.

About The Numbers

The Wall Street Journal examines numbers in the news, business and politics. Some numbers are flat-out wrong or biased, while others are valid and help us make informed decisions. We tell the stories behind the stats in occasional updates on this blog.