In 1999, the US broadcaster ABC commissioned a pilot for a new TV series set in Los Angeles. It starts with a brunette emerging from a car crash. There’s $125,000 in her bag, but she has no memory of who she is or where she was going.

Anyone hoping for a standard story arc to unfold at this point would have been disappointed. Many scenes are left unresolved and seem disconnected. And there are eerie, fantastical elements, such as the appearances of a dwarf with oversized prosthetic arms and a tiny head, and an enigmatic cowboy who is totally at odds with the film’s setting in modern Hollywood.

Although the network dropped the series before it aired, the director turned the pilot into a feature film that enjoyed a worldwide release. He had not resolved all its incongruities, however, with one critic of the film complaining that ‘nothing makes any sense … There’s no purpose or logic to events.’ Salon spent a large portion of their review asking: ‘What the fuck is going on in this movie?’

Yet today, David Lynch’s Mulholland Drive (2001) is considered one of the best films of the 21st century (appearing at the top of a poll by BBC Culture in 2016) – and the very elements that bewildered some critics are considered part of the film’s genius.

Many works of art deliberately challenge our understanding of the world in this way, including other films by Lynch, the writing of Franz Kafka and the humour of Monty Python, to name but a few. All feature illogical and incongruous elements and the uncanny juxtaposed with the familiar.

Clues to the appeal of this kind of art come from recent work by psychologists, who are beginning to understand the strange effects it can have on the brain. According to research on the ‘meaning maintenance model’ of human reasoning, surreal and absurd art can be so unsettling that the brain reacts as if it is feeling physical pain, yet it ultimately leads us to reaffirm who we are, and sharpens the mind as we look for new ways to make sense of the world. The findings also suggest new ways to improve education, and even help to explain our responses to some of the more absurd political events of recent years.

The meaning maintenance model was first proposed by three psychologists – Steven Heine, Travis Proulx and Kathleen Vohs – in 2006. They were inspired by the French-Algerian philosopher Albert Camus, who argued that the human mind continuously attempts to construct a view of reality as a single, coherent whole – an urge he described in The Myth of Sisyphus (1942) as ‘nostalgia for unity’.

Heine and his team proposed that our mental representation of the world is like a delicate web of interconnected beliefs, documenting the relations between ourselves and the people, places and objects around us. When we are confronted with an apparently inexplicable event that appears to break that framework, we feel profound uncertainty – the ‘feeling of the absurd’.

Using these ideas as a launch board, the psychologists described three ways in which the mind might mitigate that feeling. The most drastic would involve building a new mental representation to incorporate the inexplicable event. Alternatively, we could reinterpret the event so that it fits our existing mental model. Or we could strengthen other beliefs and values, even those relating to a completely unconnected domain – a phenomenon that the psychologists described as ‘fluid compensation’. This involves ‘retreating to a safe place where the world makes sense again’, Heine told me.

Earlier psychology studies already provided evidence for fluid compensation. In one, students who performed badly on a test were more likely to affirm their support of a winning sports team, presumably as a way of seeking solace for the threat to their self-esteem. Another showed that people who felt socially isolated became more defensive of their broader national identity. Heine and his colleagues reasoned that, if the meaning maintenance theory is correct, then surreal or absurd works or art – created to challenge our view of the world as a logical, coherent place – should have similar effects.

In one study designed to test the theory, the psychologists asked volunteers to read An Imperial Message (1919), a parable by Kafka that features a confusing, paradoxical conclusion, in which the protagonist finds that, no matter how far he walks through a palace, he can never reach the exit. Afterwards, participants rated how much they valued their birth country, nationality, and mother tongue. As predicted by the meaning maintenance model, people who read the Kafka story compensated for their feelings of uncertainty by reaffirming their cultural identity, significantly more so than people who had read a straightforward fable by Aesop.

The researchers later turned their gaze to visual art. They asked volunteers to view either Landscape With a Double Rainbow (1812) by John Constable, the abstract expressionist piece Untitled XVI (1976) by Willem de Kooning, or a surrealist painting, René Magritte’s The Son of Man (1964), which depicts a man in an overcoat with an apple hovering in front of his face – juxtaposing familiar elements in an unfamiliar way. Afterwards, all the volunteers completed the same questionnaire examining their ‘personal need for structure’, which directly measures how much people value order and familiarity over spontaneity and unpredictability.

Once again, the psychologists saw evidence of fluid compensation, with the Magritte painting provoking a much greater ‘need for structure’ than the other artworks. Amazingly, the need for structure scores of those who looked at the Magritte painting were as high as another group of people who wrote about their own death. The unexpected juxtaposition of the man and the apple was apparently as much of a threat to people’s mental representation of the world as thinking of their own demise.

At a neural level, the discomfort prompted by the absurd appears to share the same processing as physical pain. In another experiment, Heine and his collaborators gave volunteers the painkiller acetaminophen or a placebo before either watching a clip from Lynch’s surreal film Rabbits (2002) or writing a short essay about death. In both cases, the volunteers who took the painkiller showed reduced fluid compensation afterwards, compared with those who took a placebo.

‘They are becoming extra-vigilant to the world, trying to make sense of what doesn’t make sense’

Paradoxically, humour can similarly diminish the psychological effects of the absurd. In another of Heine’s studies, volunteers read a story based on an iconoclastic Monty Python sketch. As expected, they demonstrated fluid compensation as a result (in the form of harsher moral judgments) – but only if they didn’t understand that it was a joke. ‘When you “get” a joke, you’re resolving an expectancy violation by recognising that it was meant to be funny,’ Heine says. ‘Those who didn’t realise that it was a joke weren’t able to resolve it in this way, and hence turned to fluid compensation.’

As evidence for the meaning maintenance theory mounts, other psychologists hope it could have positive practical implications in the classroom. Similar to other negative feelings, such as anger or stress, the discomfort triggered by the absurd can sharpen our thinking, including improving our pattern-recognition skills. For instance, after reading a Kafka story, volunteers are quicker to detect and learn the rules determining the ordering of (apparently random) strings of letters. ‘They are becoming extra-vigilant to the world,’ explains Heine. ‘[And they’re] trying to make sense of what currently doesn’t make sense.’ This appears to result from a slight increase in working memory, giving the brain extra resources to find orderly connections in the chaos around us.

This raises the possibility that teachers could deliberately create feelings of uncertainty to prompt students to look harder for meaning in the material they’re studying. For example, it’s easy to see how a physics lesson on the Big Bang could serve this purpose, if teachers emphasised its truly unsettling ramifications. In other subjects, teachers could overtly challenge common misperceptions and myths. ‘Just about anything that is counterintuitive could work,’ says the social psychologist Jason Martens at Capilano University in Canada. Rolf Reber, professor of cognitive psychology at the University of Oslo, agrees, suggesting that you could ‘begin with a surprising fact, something that at first sight does not make sense’ to prompt students to look for new connections.

Heine believes that the meaning maintenance model could even help to explain our responses to recent political upheavals. After so many unexpected turns of events, many people feel like the world no longer makes sense – and that uncertainty could be prompting us to become more entrenched in our political views. ‘I often wonder if this is one thing that’s contributing to the increasing polarisation that we see,’ he says. ‘You want to retreat to a safe place, doubling down on your existing beliefs about the way the world should be.’

But can the meaning maintenance model help us understand why people turn to the surreal and absurd for entertainment in the first place? Heine thinks so: ‘I think people turn to the work [of people] like David Lynch for the same kinds of reasons that they turn to horror films. A horror movie is a safe place to experiment with your fears – you can explore your emotional reactions while knowing that you’ll be OK in the end. I think that some people also enjoy exploring their existential anxieties in a similar kind of safe space.’

The TV executives who first commissioned the Mulholland Drive pilot appeared not to appreciate this. As the show’s producer told TheNew Yorker at the time: ‘ABC thinks viewers are too stupid to want to figure things out, to have a bit of a surreal experience.’ In fact, the very incongruities that alienated the TV executives have powerful psychological effects that explain the enduring appeal of the film Mulholland Drive. Whether it’s the work of Magritte, Kafka or Lynch – a little dose of the surreal and the absurd helps us wrestle with the grander questions of our own place on Earth, and catalyses our search for meaning within it.

Sign-up to our weekly newsletter

Psyche is a new digital magazine from Aeon that illuminates the human condition through three prisms: mental health; the perennial question of ‘how to live’; and the artistic and transcendent facets of life.