Can a Brain-Computer Interface Convert Your Thoughts to Text?

ECoG and audio data are recorded at the same time. Speech decoding software is then used to determine timing of vowels and consonants in acoustic data. ECoG models are then trained for each phone individually by calculating the mean and covariance of all segments associated with that particular phone. Image courtesy of Christian Herff

Ever wonder what it would be like if a device could decode your thoughts into actual speech or written words? While this might enhance the capabilities of already existing speech interfaces with devices, it could be a potential game-changer for those with speech pathologies, and even more so for “locked-in” patients who lack any speech or motor function.

“So instead of saying ‘Siri, what is the weather like today’ or ‘Ok Google, where can I go for lunch?’ I just imagine saying these things,” explains Christian Herff, author of a review recently published in the journal Frontiers in Human Neuroscience.

While reading one’s thoughts might still belong to the realms of science fiction, scientists are already decoding speech from signals generated in our brains when we speak or listen to speech.

In their review, Herff and co-author, Dr. Tanja Schultz, compare the pros and cons of using various brain imaging techniques to capture neural signals from the brain and then decode them to text.

The technologies include functional MRI and near infrared imaging that can detect neural signals based on metabolic activity of neurons, to methods such as EEG and magnetoencephalography (MEG) that can detect electromagnetic activity of neurons responding to speech. One method in particular, called electrocorticography or ECoG, showed promise in Herff’s study.

This study presents the Brain-to-text system in which epilepsy patients who already had electrode grids implanted for treatment of their condition participated. They read out texts presented on a screen in front of them while their brain activity was recorded. This formed the basis of a database of patterns of neural signals that could now be matched to speech elements or “phones”.

When the researchers also included language and dictionary models in their algorithms, they were able to decode neural signals to text with a high degree of accuracy. “For the first time, we could show that brain activity can be decoded specifically enough to use ASR technology on brain signals,” says Herff. “However, the current need for implanted electrodes renders it far from usable in day-to-day life.”

So, where does the field go from here to a functioning thought detection device? “A first milestone would be to actually decode imagined phrases from brain activity, but a lot of technical issues need to be solved for that,” concedes Herff.

The Latest on: Brain-to-text system

November 23, 2018 - Brain implant lets paralyzed people turn thoughts into text EngadgetThree people paralyzed from the neck down have been able to use unmodified computer tablets to text friends, browse the internet and stream music, thanks to ...

November 20, 2018 - With Brain Implants, Scientists Aim to Translate Thoughts into Speech Scientific AmericanScientific American is the essential guide to the most awe-inspiring advances in science and technology, explaining how they change our understanding of the ...

November 27, 2018 - Brain interface lets people with paralysis control tablet computer Futurity: Research NewsA brain-computer interface (BCI) allows people with paralysis to directly operate an off-the-shelf tablet device just by thinking about making cursor movements ...

November 21, 2018 - Brain implants help paralyzed people use tablet apps Science NewsDevices that eavesdrop on neural activity can help paralyzed people command computer tablets to stream music, text friends, check the weather or surf the ...

November 26, 2018 - The Neuroscience of Mind-Control Gaming VarietyNeurable received $2 million to develop software that make it possible for users to control virtual and augmented reality with their mind.

December 14, 2018 - MS brain lesions: Pictures and prognosis Medical News TodayAs multiple sclerosis (MS) progresses, lesions may form in the brain. With magnetic resonance imaging, doctors can use the lesions to track the progress of the ...

December 13, 2018 - New brain circuit that controls anxiety found Medical News TodayUsing genetically modified mice, scientists have uncovered a brain circuit that regulates anxiety and could be a treatment target for related disorders.

December 12, 2018 - FACT CHECK: Did a Brain-Eating Amoeba Kill a Woman Who Rinsed Her Sinuses with Tap Water? Snopes.comA woman who rinsed her nasal passages using a neti pot filled with tap water contracted a fatal “brain-eating” amoeba infection.

November 22, 2018 - This is how your brain predicts future events Medical News TodayBrains learn how to anticipate future occurrences from patterns. This process is called "anticipatory timing," and it allows us to successfully interact with the world ...

via Google News

The Latest on: Brain-to-text system

"That's extremely aggressive," for a such a brain-to-text system, says Wyble. In a keynote speech at a Facebook conference in San Jose, CA, on April 19, Dugan spoke of the brain in terms usually reser... […]

This would be achieved using an optical imaging system which scans the brain 100 times a second. That could allow users to send texts and emails without touching their phone, and could be useful for p... […]

Using brain implants, people can already type eight words a minute, she said. Facebook's goal, working with researchers at several U.S. universities, is to make the system non-invasive, as well as fas... […]

At the same time, these efforts raise hard questions related to patient privacy, as any brain-to-text system will essentially read a person's unspoken thoughts. On the Internet, many reacted with alar... […]

Using brain implants, people can already type eight words a minute, she said. Facebook’s goal, working with researchers at several U.S. universities, is to make the system non-invasive, as well as fas... […]

One method in particular, called electrocorticography or ECoG, showed promise in Herff’s study. This study presents the Brain-to-text system in which epilepsy patients who already had electrode grids ... […]

Researchers at KIT and Wadsworth Center, USA present their "Brain-to-Text" system in the scientific journal ... In Karlsruhe, the methods for signal processing and automatic speech recognition have be... […]

Researchers at Karlsruhe Institute of Technology in Germany and at Wadsworth Center in America are the first to devise a system that can decode brain activity from speaking and actually translate it b... […]