We like to think we can read people like a book, relying mostly on tell-tale facial expressions that give away the emotions inside: the way the brows lift slightly with alarm, or the crow’s feet that crinkle with a wide smile. But when it comes to the strongest emotions, we read much less from facial expressions than we think we do. In fact, even though we believe it’s the face that tells the story, we’re typically reading something very different: body language and social cues.

That’s the new, counterintuitive finding from a study published this week in the journal Science. Researchers from Princeton, New York University, and the Hebrew University of Jerusalem presented volunteer study participants with a series of pictures showing people experiencing extreme emotion, either positive or negative. The images included professional tennis players who had just won or lost a point in a major match, as well as people undergoing nipple piercing, and those in the throes of orgasm.

In some of the images, researchers would only show the study participants a face; in others, only a body; and in others still, both the body and the face. You might think it’d be obvious from a face whether someone is in pain (having a nipple pierced) or whether he has just won Wimbledon. But it turns out it isn’t.

“The striking finding was that our participants had no clue if the emotion was positive or negative, when they were judging isolated faces,” says lead study author Hillel Aviezer from Hebrew University in an email response discussing the findings. “By contrast, when they were judging the body (with no face), or the body with the face, they easily differentiated positive from negative expressions.”

The findings are doubly surprising because the study participants themselves were convinced that they recognized the emotions from the faces, not from body language or contextual cues.

“They even had their own ‘mini theories’ about what part of the face was most important – but this was all an illusion,” Aviezer says.

He adds that we do, of course, read a great deal of salient day-to-day emotional information from faces — but only in certain situations. The reliability of that transmission, for example, appears to break down when emotions are at their strongest. The face contorts. We can tell that something major has happened, but it’s tough to tell that something is dramatically positive or devastatingly negative.

In the article, Aviezer and his colleagues liken the muscles of the face to an audio speaker. As the volume is pushed to its maximum, the quality of the signal becomes poor and the message becomes harder to pick out.

There is a similar analogy to the emotional signals that we hear. We know that the sound of gentle laughing conveys happiness and a gentle sob conveys sorrow. But most of us find it difficult to distinguish a shriek of joy from a shriek of fear. Since we often hear sounds without another contextual clue to explain them — overheard from an adjacent room — we tend to accept that we can’t tell positive from negative emotion in all cases. But we almost never see faces without body language and a setting to accompany them. So we may fail to realize that the face is less informative than usual when emotions run high.

Aviezer says his results suggest that the classic textbook distinction between positive emotions and negative emotions may need to be revised to incorporate the possibility that they may share more underlying physiological features than we thought. That could have implications for theories that link emotions to economics, social psychology, and neuroscience.

“I also think the findings may have some clinical applications,” he says. “Consider populations such as individuals with autism or various neuropsychiatric disorders. We know these people often have difficulties with recognizing facial expressions,” he says. “Until now we have been trying to help them by training them to better understand the isolated faces. But our work suggests that perhaps we should zoom out a bit and teach them how to recognize emotions from the full person in context.” That could provide a broader range of therapies from which autistic children can draw, and, perhaps even benefit.