Machine Translates Thoughts into Speech in Real Time

Model of the brain-machine interface for real-time synthetic speech production. The stroke-induced lesion (red X) disables speech output, but speech motor planning in the cerebral cortex remains intact. Signals collected from an electrode in the speech motor cortex are amplified and sent wirelessly across the scalp as FM radio signals. The Neuralynx System amplifies, converts, and sorts the signals. The neural decoder then translates the signals into speech commands for the speech synthesizer. Credit: Guenther, et al.

(PhysOrg.com) -- By implanting an electrode into the brain of a person with locked-in syndrome, scientists have demonstrated how to wirelessly transmit neural signals to a speech synthesizer. The "thought-to-speech" process takes about 50 milliseconds - the same amount of time for a non-paralyzed, neurologically intact person to speak their thoughts. The study marks the first successful demonstration of a permanently installed, wireless implant for real-time control of an external device.

The study is led by Frank Guenther of the Department of Cognitive and Neural Systems and the Sargent College of Health and Rehabilitation Sciences at Boston University, as well as the Division of Health Science and Technology at Harvard University-Massachusetts Institute of Technology. The research team includes collaborators from Neural Signals, Inc., in Duluth, Georgia; StatsANC LLC in Buenos Aires, Argentina; the Georgia Tech Research Institute in Marietta, Georgia; the Gwinnett Medical Center in Lawrenceville, Georgia; and Emory University Hospital in Atlanta, Georgia. The team published their results in a recent issue of PLoS ONE.

“The results of our study show that a brain-machine interface (BMI) user can control sound output directly, rather than having to use a (relatively slow) typing process,” Guenther told PhysOrg.com.

In their study, the researchers tested the technology on a 26-year-old male who had a brain stem stroke at age 16. The brain stem stroke caused a lesion between the volunteer’s motor neurons that carry out actions and the rest of the brain; while his consciousness and cognitive abilities are intact, he is paralyzed except for slow vertical movement of the eyes. The rare condition is called locked-in syndrome.

Five years ago, when the volunteer was 21 years old, the scientists implanted an electrode near the boundary between the speech-related premotor and primary motor cortex (specifically, the left ventral premotor cortex). Neurites began growing into the electrode and, in three or four months, the neurites produced signaling patterns on the electrode wires that have been maintained indefinitely.

Three years after implantation, the researchers began testing the brain-machine interface for real-time synthetic speech production. The system is “telemetric” - it requires no wires or connectors passing through the skin, eliminating the risk of infection. Instead, the electrode amplifies and converts neural signals into frequency modulated (FM) radio signals. These signals are wirelessly transmitted across the scalp to two coils, which are attached to the volunteer’s head using a water-soluble paste. The coils act as receiving antenna for the RF signals. The implanted electrode is powered by an induction power supply via a power coil, which is also attached to the head.

The signals are then routed to an electrophysiological recording system that digitizes and sorts them. The sorted spikes, which contain the relevant data, are sent to a neural decoder that runs on a desktop computer. The neural decoder’s output becomes the input to a speech synthesizer, also running on the computer. Finally, the speech synthesizer generates synthetic speech (in the current study, only three vowel sounds were tested). The entire process takes an average of 50 milliseconds.

As the scientists explained, there are no previous electrophysiological studies of neuronal firing in speech motor areas. In order to develop an accurate neural coding scheme, they had to rely on an established neurocomputational model of speech motor control. According to this model, neurons in the left ventral premotor cortex represent intended speech sounds in terms of “formant frequency trajectories.”

In an intact brain, these frequency trajectories are sent to the primary motor cortex where they are transformed into motor commands to the speech articulators. However, in the current study, the researchers had to interpret these frequency trajectories in order to translate them into speech. To do this, the scientists developed a two-dimensional formant frequency space, in which different vowel sounds can be plotted based on two formant frequencies (whose values are represented on the x and y axes).

“The study supported our hypothesis (based on the DIVA model, our neural network model of speech) that the premotor cortex represents intended speech as an ‘auditory trajectory,’ that is, as a set of key frequencies (formant frequencies) that vary with time in the acoustic signal we hear as speech,” Guenther said. “In other words, we could predict the intended sound directly from neural activity in the premotor cortex, rather than try to predict the positions of all the speech articulators individually and then try to reconstruct the intended sound (a much more difficult problem given the small number of neurons from which we recorded). This result provides our first insight into how neurons in the brain represent speech, something that has not been investigated before since there is no animal model for speech.”

To confirm that the neurons in the implanted area were able to carry speech information in the form of formant frequency trajectories, the researchers asked the volunteer to attempt to speak in synchrony with a vowel sequence that was presented auditorily. In later experiments, the volunteer received real-time auditory feedback from the speech synthesizer. During 25 sessions over a five-month period, the volunteer significantly improved the thought-to-speech accuracy. His average hit rate increased from 45% to 70% across sessions, reaching a high of 89% in the last session.

Although the current study focused only on producing a small set of vowels, the researchers think that consonant sounds could be achieved with improvements to the system. While this study used a single three-wire electrode, the use of additional electrodes at multiple recording sites, as well as improved decoding techniques, could lead to rapid, accurate control of a speech synthesizer that could generate a wide range of sounds.

“Our immediate plans involve the implementation of a new synthesizer that can produce consonants as well as vowels but remains simple enough for a BMI user to control,” Guenther said. “We are also working on hardware that will greatly increase the number of neurons that are recorded. We expect to tap into at least 10 times as many neurons in the next implant recipient, which should lead to a dramatic improvement in performance.”

Overall, the work marks a milestone in the development of a permanent neural prosthesis that requires no major external hardware beyond a wireless receiver and laptop computer. Previous brain-machine interfaces for communication applications are very slow, producing only about one word per minute. The new system has the potential to enable real-time conversation, and help minimize the social isolation that accompanies profound paralysis.

Related Stories

A particular resonance pattern in the brain’s auditory processing region appears to be key to its ability to discriminate speech, researchers have found. They found that the inherent rhythm of neural activity called “theta ...

Researchers have identified regions of the brain where speech sounds are perceived as having abstract meaning, rather than as just a stream of sensory input. They said their identification of the regions demonstrates that ...

Researchers from the Hebrew University of Jerusalem have succeeded for the first time in devising a model that describes and identifies a basic cellular mechanism that enables networks of neurons to efficiently decode speech ...

When someone speaks to you, do you see what they are saying? We tend to think of speech as being something we hear, but recent studies suggest that we use a variety of senses for speech perception - that the brain treats ...

(PhysOrg.com) -- A McGill-led study has found that the perception of speech sounds is modified by stretching facial skin in different directions. Different patterns of skin stretch affect how subjects perceive different words.

Inspired by the way sound is processed in the brain, researchers at Carnegie Mellon University (CMU) have devised a new, highly efficient scheme for converting sounds to digital form. The payoff could be a ...

Recommended for you

A team of researchers at the IRCM led by Frédéric Charron, PhD, in collaboration with bioengineers at McGill University, uncovered a new kind of synergy in the development of the nervous system, which explains an important ...

Memory loss has recently been associated with excessive silencing of genes through a process called methylation. Researchers at the University of Louisville investigated the effects of a diet rich in methionine—an amino ...

(MedicalXpress)—A team of researchers with the University of Maryland and Mt. Sinai School of Medicine has found that brain regions in female rodents associated with sexual behavior are feminized by repression ...

Neuroscientists are taking inspiration from natural motor control to design new prosthetic devices that can better replace limb function. In new work, researchers have tested a range of brain-controlled devices ...

The decades worth of data that has been collected about the billions of neurons in the brain is astounding. To help scientists make sense of this "brain big data," researchers at Carnegie Mellon University ...

one step closer to telepathy. the mind is still a black box, theories are how the auditory and speech system may be obscured by plastic adaptation of the neuronal system of the subjects brain to the device that is implanted in it, however, as long as the theory helps the progress and development of the technology, it's utility warrants its explanatory legitimacy, particularly with regard to the mechansim of the action in the artificial speech device (as opposed to natural speech) .

dan42day

Perhaps I'm a mad scientist, but it seems this procedure could be performed on other species on the planet, such as chimpanzees or dolphins, heck maybe even a horse. Wouldn't that be cool if there was a breakthrough like this to talk with lower species.

scientists implanted an electrode near the boundary between the speech-related premotor and primary motor cortex

I'm not sure why they chose that site. Was it because it was easier to insert the electrode there, or was it because it supported their "auditory trajectory" hypothesis?

I honestly hope it was the former.

*After* the primary motor cortex would have made learning to speak way easier for the patient. Weeks, instead of half a year! Learning to ski, skate, drive, karate, whistle, they're all simply reprogramming the motor cortex. That's what we do. We're actually better at it than any other species.

earls

The internal dialogue... the conversation we are always having with ourselves. We can hear it, we know it exists, and yet there is no evidence that it occurs... yet. The old New Age (?) philo/anthro/huckster? Carlos Castenada said his Yaqui brujo Don Juan Matus taught that the first step on the path to becoming a 'man of power' was learning to turn off the internal dialogue. But we want to be able to broadcast it instead. His methods also included peyote, psilocybin and various other natural herbs and spices; but only in the first book.

One could imagine with the evolution of this technology, a nano computer system implanted in the brain(or perhaps less invasively, rather as an article of clothing in contact with the electro magnetic brainwave spectrum)such that communication would be possible between parties by thought alone and translated from their native languages automatically. Frankly, this scenario seems no further off than say harnessing matter/anti matter reactions to power inter stellar flight shielding humans from radiation and using inertial force as an equivalent for gravity via effect continuous acceleration deceleration.

Donovan's brain will be real if this continues. So will tele-robots that work at the command of a person wearing a thinking cap. It seems that mind reading devices are much closer to reality than anyone thought. The singularity is approaching and its picking up speed.

What a fascinating development!However, I can already imagine some of the potential negative consequences if this was to morph into a new form of lie detection, and/or thought police tool as we push into a world of more control.

Lies should be detected. Cheaters should be punished dont you think? Why do people think that the ability to get away with something or to get preferential treatment are unalienable rights? I say cameras on every corner to catch dumbass speeders, tailgaters and red light runners. They restrict my freedom to drive in safety and comfort. Tech will soon make lawyers technically obsolete but people will resist this because they will want to retain the chance to beat the system. This is WRONG.

Sometimes maybe the system is wrong? I don't feel the speed limit should be 60, I want to drive 65-70 and I am perfectly capable of doing so without putting anyone in any more danger. However, it's the slow drivers who don't look where they're going that cause a lot of the accidents.

@plasticpowerSee? Like I said- we don't usually need to break laws but we feel it's our right to do so. We WANT to, just as any animal wants to find a way out of the cage. Instinct. I heard 2 lawyers talking yesterday about a simple case with an inevitable conclusion which nevertheless dragged on for years, causing needless hardship and expense, only because the defendent had some influence. Look up the story on the Guess Jeans magnate. Over principle he has ruined lives not to prove any point but because he is an unsettled paranoid. With millions to burn. Enough is enough. Only automated justice is truly blind.

Actually, the gov is decades ahead of this. They have been reading ALL of our thoughts for at least 2 decades. Technological Telepathy has long been in use. But of course they just label (like me) people as schizophrenic who say they hear voices or are the victim of gov. mind control programs. Too much for you?

What is it- 11.5mhz or something, the frequency emanating from those police repeaters on water towers and which resonates with our brain waves? I read icke too. I doubt if most peoples thoughts are worth reading but they are certainly worth influencing. MK Ultra.

This technology is in no way attempting telepathy nor to teach speech. Once areas of the brain are dead they cannot be reanimated (or at least this study is not attempting to do so). The brain distinguishes everything we do. Internal thought is not the same as external verbalization, and the intent and implementation for speech are entirely different. The electrode is placed exactly were it can pick up intents to speak. It could not be placed after the primary motor cortex because that is what is damaged. If the volunteer had impulses going to that area, then he would be speaking, in very rudimentary terms.

Anyone remember the Gilligan Island episode where Gilligan eats something and can read the thoughts of others? This story reminds me of that. What happens when we say exactly what we think? "Hey, Ginger, nice t*ts!"

This will come as a major relief to those poor individuals trapped inside their bodies as they will be able to make their wishes clear - including the wish to end all life support.

I personally think that given that people have the right to end all treatment - effectively choosing a slow a painful death - we should also give them the right to choose a quick and painless death. After all, murderers die by painless lethal injection, so why do we force innocents to suffer more?

I hasten to add that this should be a personal choice - but I know which one I'd make.

I'd like to know how to deactivate micro chip brain implants. Since 2001, my famly and I have been harassed continually by the "Big Brother" and the city where I live in here Italy. Even all my co-workers have access to me. I have no privacy. Please send comment by e.mail at clifbank2003@yahoo.comThan you.

If the volunteer had impulses going to that area, then he would be speaking, in very rudimentary terms.

Actually, thats not technically true. The impulses to the motor cortex are working fine (this is why putting the electrodes in between the pre-motor and the motor cortexes found signal) but the connections from the motor cortex to the actual MUSCLES was destroyed. This is why he was paralyzed (spinal cord damage).

Interestingly, the signals through your brain to cause movement (in a normal, healthy human) are almost identical to the signals sent while merely THINKING about moving, minus the primary motor cortex (which initiates the movements).

Please sign in to add a comment.
Registration is free, and takes less than a minute.
Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.