When Zinedine Zidane, the then French captain, headbutted Italy’s Marco Materazzi during the 2006 World Cup final, the clash quickly became one of the most infamous incidents in football history. What was not clear was what sparked the Frenchman’s ire – Zidane said his mother had been insulted, a charge that Materazzi vigorously denied.

The head-butt got Zidane sent off and Italy won the game. However, had there been technology there to identify what was said, the result could have been very different, Helen Bear believes. “If a machine lip-reader was in existence, the other player [could] have got sent off too so it would have been 10 men against each other in a World Cup final,” she argues.

Bear is one of a number of researchers at the University of East Anglia focusing on ways to teach computers to read people’s lips, technology that could be used in artificial intelligence applications.

A three-year study at the university’s school of computer studies could prove a significant advance in the science behind automated lip-reading, which is still in its early stages. The technology could help people who have recently lost their hearing and, on a more basic level, it could improve our interactions with gadgets that are usually controlled by hand.

“For those who suffer post-lingual hearing loss, it is a a lot harder for them to learn to lip-read than someone who was deaf from birth, purely because if you have to learn from birth you are surrounded by all of this visual information. Some sort of technology that could help with that would be invaluable,” she says. In practical terms, the system could work using the camera on a smartphone to read the speaker’s lips and then carry out commands.

One of the main problems faced by researchers is that some of the sounds that are made when people talk relate to a very similar facial expression. These shapes that the mouth makes are known as visemes. However, there are many more sounds, or phonemes, made during speech.

This means that the viseme can have a number of meanings. A human lip-reader needs to figure out what is the actual meaning and also relies on other information such as the context of what is being talked about and body language. In a similar way, machines working to pinpoint what is being said by analysing the movement of the mouth have the same problem when the different sounds have similar facial appearances. “This is where we get this confusability,” says Bear, who has recently completed her PhD.

Her breakthrough has been in finding a new way to distinguish the sounds, which appear similar on the face, by identifying subtle differences which computers will be taught to recognise . By doing this, the different words can then take shape and the computer can lip-read what a person is saying.

The development is significant, says Prof Richard Harvey, who has been working with Bear on the research. “Lip-reading is one of the most challenging problems in artificial intelligence so it’s great to make progress on one of the trickier aspects, which is how to train machines to recognise the appearance and shape of human lips,” he says.

The possibility of machines being able to lip-read could allow people to control devices without using their hands, such as when driving. A smartphone being used as a satnav could still pick up commands even if background or engine noise drowns out the speaker. And for someone who is outside using a phone whose call is being disrupted by wind noise, the camera could switch itself on and pick up what is being said.

In other areas of research, using advances in technology has meant that volume controls in cars can now be manipulated using gesture controls similar to those of games machines, and cookers may soon have the ability to be turned on and off without being touched.

Bear says that while the reality of this happening is some way off, it is possible. “I don’t see any reason why lip-reading technology will not exist at some point,” she says.

With the possibility that a machine can read someone’s lips and then the words are displayed on a screen will inevitably raise privacy concerns about the possible uses for this new strand of artificial intelligence.

“The whole point is that the machine is learning something that as humans we have not been able to do by ourselves, which is quite exciting. I know some people are wary [about artificial intelligence]. My personal opinion is that if you are careful, if you properly develop something and test it as far as you can, everything should be OK. It is all about having good software engineering practices and principles,” Bear says.

• For more information on how Big Innovation Centre supports innovative enterprise in Britain and globally, go to our archive of the innovators columns on the Big Innovation Centre website.

This article was amended on 25 April 2016. Helen Bear has completed, but not yet been awarded her PhD. As such, she cannot yet officially be referred to as a doctor.