The better you are at combining visual and auditory information, the better you can remember what you’ve learned. This conclusion reached by neuroscientists at UNIL demonstrates the effectiveness of teaching methods which simultaneously make use of multiple senses, like that developed by Maria Montessori. This could also improve the rehabilitation of stroke victims.

Imagine that you’re with your friend, Julie, when you both witness a burglary in which one of the robbers speaks while the other one remains silent. Will this difference in behaviour allow you to recognise the first burglar more easily in an identity parade at the police station? Probably, because it seems that multisensory events – those which engage sight and hearing – enhance memory and create more vivid memories. However, you have to be able to combine visual and auditory information, which may not be the case for Julie. This is the conclusion of the research conducted by Micah Murray, associate professor at the UNIL Faculty of Biology and Medicine, Director of the Laboratory for Investigative Neurophysiology (LINE) at the Lausanne University Hospital and of the EEG module at the Center for Biomedical Imaging (CIBM).

Our senses interact so that we can perceive information

We perceive our environment, in the broadest sense of the term, through the signals sent to use by our five senses (sight, hearing, smell, taste and touch). How does this different information interact to allow our brain to recognise a face or learn a foreign language? This question has always fascinated Murray and his interest has led this “expert in multisensory interactions,” as he defines himself, to conduct research in many areas, ranging from blindness to memory.

Does the brain function like Switzerland?

Before you can store and retrieve the memory of somebody you have met or something you have seen, your brain must encode the information it receives from the senses. Yet it was long believed “that the brain areas which process for example visual and auditory information worked independently of each other,” subsequently sending the information they hold to a higher area of the brain. “Basically,” summarises the UNIL professor, according to this interpretation, it’s like in Switzerland where “the cantons operate independently, with each one processing its own information, then they meet in the capital city Bern to summarise”.

Watching to better understand speech

“This was forgetting,” he continues in the same vein, “that there are direct rail lines connecting the different cantons”. Similarly, research in the past few decades has shown “that these different brain areas communicate with each other” and that the visual areas can help the auditory areas to process information, or vice versa. Murray illustrates his point with the example of speech. “When you listen to someone who speaks your native language, you don’t need to watch it being spoken to understand, and if you do, you make eye contact with the speaker. But if you don’t have a good command of the speaker’s language, you look at his mouth to better understand his words”. We’ve all intuitively experienced this multisensory perception when two of our senses work together to help us better understand reality. However, “there has been only little scientific research into this phenomenon”.

Can you benefit from multisensory information?

Murray and his team therefore decided to look into the subject to find out “whether you can benefit from this multisensory interaction”. Previous work in this field in fact went in the opposite direction and concluded that it was counterproductive to engage multiple senses simultaneously when performing a task. “These conclusions were hard to believe” because they contradict our everyday experience. And the researcher has identified the bias. These surprising results are explained by the fact that “up until recently, neuroscientists conducted their studies on groups of volunteers”. Their results had statistical value, but did not take into account the differences between individuals.

A sound can help to remember an image…

Murray and his colleagues consequently repeated the experiment, recruiting healthy UNIL students. Their task was simple: they had to look at drawings and say whether or not it was the first time the drawings were shown to them. Sometimes the image was also accompanied by a “meaningless” sound like a ‘beep’”. At the same time, researchers recorded the brain activity of the volunteers using electroencephalography (EEG).

They then found that for some students, remembering images worked better in the presence of a sound while in others the opposite was true. “Some benefited, while others were put at a disadvantage,” says the neuroscientist.

… and an image can help to remember a sound

To ensure that this phenomenon did not concern just visual information, the researchers repeated the experiment with another group of students which they asked to listen to a series of recognisable sounds: “sounds from everyday life like a passing truck or a mewing cat,” sometimes with an abstract drawing. Again, the effect varied according to the individual; the image helped some volunteers to remember the sound but impeded others. This is probably because “some people can assimilate multisensory perceptions (visual and auditory in this case), whereas others focus on the task at hand, ignoring other stimuli”.

Foreseeable capacities

Better still, thanks to the EEG, the researchers found they could predict the memory capacity of the volunteers depending on each one’s ability to combine visual and auditory information. ”We’ve established for the first time that there’s a direct link between brain activity in response to multisensory information at a given moment and the subsequent ability to recognise an object through sight or hearing,” says Murray.

Breaking down entrenched ideas

These observations are laying to rest entrenched ideas in psychology that memory is better if you learn and recollect in the same context. It’s always been thought, for instance, “that a diver who’s learned something underwater remembers it better when he’s under the same conditions”. Yet the neuroscientist points out that “our study shows how one single presentation of an image in a context where multiple senses are stimulated improves memory more effectively than when you act in a purely unisensory context”. So you don’t need to dive back underwater to remember what you’ve learned.

Scientific confirmation of the Montessori method

Murray also sees his work as “scientific confirmation of the soundness of the Montessori Method,” named after Maria Montessori, and Italian physician and educator who developed this learning method in the early twentieth century. Her approach was to “treat children as individuals, to give everyone some degree of freedom to find out the best way of learning for them and to provide different students with resources appealing to their sense of touch and sight etc”. In other words, her teaching was based on multisensory education, the benefits of which have been demonstrated by UNIL researchers.

Experiments on pupils

Convinced that his studies “pave the way for a particularly effective learning strategy,” Murray, together with his Geneva colleagues, has also undertaken to repeat his experiments with students from a Montessori school and pupils at a state school. He aims to check if “one of the teaching methods strengthens children’s ability to assimilate multisensory information and improves their memory”.

The role of art, music and dance

But without waiting for the results of this study – due to be published in a few months’ time – Murray believes that education should give more consideration to each child’s needs, taking into account that some children “have a visual memory while others have an auditory memory, and still others have a multisensory memory”. He also recommends giving greater prominence “to art, dance, music and other disciplines which appeal to multiple senses” which, beyond mere entertainment, could improve young people’s learning abilities.

Rehabilitating stroke patients

This work could also have clinical implications, suggests the neuroscientist, specifically “for the rehabilitation of people who have had a stroke”. For stroke patients, the environment becomes very “noisy”; it’s overloaded with stimuli of all kinds and they can’t isolate the most important ones. Since some students tested at UNIL were able, in the presence of multisensory information, to isolate noise interference and focus on the visual task at hand, it might be possible to train stroke patients do the same. “We’ll see,” says Murray.

Can you improve your memory?

The perception study therefore opens up many avenues and it’s tempting to ask whether they could allow the average person to improve his or her memory. “It’s easy to test yourself to find out if you have an auditory or a visual memory,” says the neuroscientist. There’s nothing stopping us either from repeating the experiment here with students to try to determine whether, in the presence of a sound, we remember images better or worse. “If you feel like it, have fun,” says the researcher. But bear in mind that this is only a game and you shouldn’t expect any big effect. “If truth be told, we don’t know if someone who benefits from multisensory stimuli to perform a particular task will react the same if they have to do another type of exercise,” says Murray. The mysteries of perception are therefore far from all being explained.