Face-reading software to judge the mood of the masses

Systems that can identify emotions in images of faces might soon collate millions of peoples' reactions to events and could even replace opinion polls

IF THE computers we stare at all day could read our faces, they would probably know us better than anyone.

That vision may not be so far off. Researchers at the Massachusetts Institute of Technology's Media Lab are developing software that can read the feelings behind facial expressions. In some cases, the computers outperform people. The software could lead to empathetic devices and is being used to evaluate and develop better adverts.

But the commercial uses are just "the low-hanging fruit", says Rana el Kaliouby, a member of the Media Lab's Affective Computing group. The software is getting so good and so easy to use that it could collate millions of peoples' reactions to an event as they sit watching it at home, potentially replacing opinion polls, influencing elections and perhaps fuelling revolutions.

"I feel like this technology can enable us to give everybody a non-verbal voice, leverage the power of the crowd," el Kaliouby says. She and her colleagues have developed a program called MindReader that can interpret expressions on the basis of a few seconds of video. The software tracks 22 points around the mouth, eyes and nose, and notes the texture, colour, shape and movement of facial features. The researchers used machine-learning techniques to train the software to tell the difference between happiness and sadness, boredom and interest, disgust and contempt. In tests to appear in the IEEE Transactions on Affective Computing, the software proved to be better than humans at telling joyful smiles from frustrated smiles. A commercial version of the system, called Affdex, is now being used to test adverts (see "Like what you see?").

Collecting emotional reactions in real time from millions of people could profoundly affect public polling. El Kaliouby, who is originally from Egypt, was in Cairo during the uprising against then-president Hosni Mubarak in 2011. She was startled that Mubarak seemed to think people liked his presidency, despite clear evidence to the contrary.

"She thought maybe Mubarak didn't think a million people was a big enough response to believe that people are upset," lab director Rosalind Picard said at the lab's spring meeting on 25 April. "There are 80 million people in Egypt, and most of them were not there. If we could allow them the opportunity to safely and anonymously opt in and give their non-verbal feedback and join that conversation, that would be very powerful."

Pollsters could even collect facial reactions on the streets, or analyse the reaction of an audience listening to a politician's speech. Picard's group recently ran an MIT-wide experiment called Mood Meter, placing cameras all over campus to gauge the general mood. To preserve privacy, the cameras didn't store any video or record faces - they just counted the number of people in the frame, and how many were smiling.

Frank Newport, editor in chief of political polling firm Gallup, headquartered in Washington DC, says such software could be useful. "There's no question that emotions and instincts have an impact in politics," he says. "We're certainly open to looking at anything along those lines." But he'd want to know how well facial responses predict actual votes.

Picard worries that the technology might have a dark side. "My fear is that some of these dictators would want to blow away the village that doesn't like them," she says. It would be important to protect the identities and IP addresses of viewers, she says.

Like what you see?

Since launching a project to record viewers' facial reactions to Super Bowl adverts in February, they have collected more than 40 million frames of people responding to what they see. Facial expressions and head position are picked up by the user's webcam and then processed to gauge emotion.

The adverts that were tested can be viewed on the company website, as can graphs of the audience response, grouped by age. The idea is to give advertisers a fast, accurate response to campaigns.

If you would like to reuse any content from New Scientist, either in print or online, please contact the syndication department first for permission. New Scientist does not own rights to photos, but there are a variety of licensing options available for use of articles and graphics we own the copyright to.