Microsoft may have co-opted Star Trek a full century early by demonstrating an honest-to-goodness universal translator — one that not only renders what you’re saying into another language in real time, but that manages to sound like you while doing so.

In fact, assuming everything really is as it appears in the video above, Microsoft just pulled off something pretty amazing — much more impactful, in theory anyway, than a mere voice recognition app like Apple’s Siri.

Microsoft’s global head of research, Rick Rashid, demonstrated the surprisingly mature technology to a crowd of 2,000 students and teachers on Oct. 25 at the 14th annual Computing in the 21st Century Conference, held in Tianjin, China (the fourth largest city in the country).

Standing onstage with a large screen above him, Rashid’s speech was at first rendered on the screen as English text, the words appearing as spoken with near-perfect accuracy. A “recognizability” percentile in the lower-right-hand corner indicated how identifiable Rashid’s speech patterns were, operating well above 70% for most of the presentation.

After walking through a few watershed moments in speech-recognition research, Rashid shifted to live speech translation, explaining that Microsoft’s approach to the process happens in three steps. First, the company converts spoken English word-by-word into Chinese text. Next, the words are rearranged, since the word order of a Chinese sentence is different from its English analogue. Last, the newly translated Chinese text is converted back into speech, and — here’s the really clever part — made to sound as if the original speaker were vocalizing in the translated language (you can hear this yourself, starting around the video’s 7:30 mark).

“In the realm of natural user interfaces, the single most important one — yet also one of the most difficult for computers — is that of human speech,” wrote Rashid in a followup blog post. “For the last 60 years, computer scientists have been working to build systems that can understand what a person says when they talk.”

In the course of its research, Microsoft says it’s been able to reduce errors by 30% — an increase, according to the company, from one word in four to five being incorrect, to just one word in seven to eight. Rashid calls that “the most dramatic change in accuracy since the introduction of hidden Markov modeling in 1979.” (Markov modeling is a math concept dealing with probability theory.)

And with its speech-language translation engine, Rashid argues:

…we may not have to wait until the 22nd century for a usable equivalent of Star Trek’s universal translator, and we can also hope that as barriers to understanding language are removed, barriers to understanding each other might also be removed. The cheers from the crowd of 2000 mostly Chinese students, and the commentary that’s grown on China’s social media forums ever since, suggests a growing community of budding computer scientists who feel the same way.

So that’s Microsoft pitch — impressive, real enough and clearly promising for the future of engagement between speakers of different languages.

But would a universal translator also have downsides?

I can think of one in the Nicholas Carr “Is Google Making Us Stupid?” vein — the notion that externalizing so much of what we do mentally with computers and via the Internet is making us shallower, cognitively speaking. And academic research into the Internet’s role as an extension of our brains suggests that the more we’re sure of having access to information in the future, the less we’re able to summon it from memory.

So what happens if we outsource our brains linguistically? Would a universal translator render language instruction obsolete? Why, if you could just clip something onto your shirt, Star Trek-style, would you bother to actually learn a second or third or fourth language, when a computer could just play wingman and save you the effort? Doesn’t learning another language actually increase our brainpower? Would externalizing that diminish us somehow?

You’ve probably heard how learning a second language can be a serious brain booster. In an interview on the subject, Therese Sullivan Caccavale, president of the National Network for Early Language Learning, references a 2007 Harwich, Massachusetts study, which she explains “showed that students who studied a foreign language in an articulated sequence outperformed their non-foreign language learning peers on the Massachusetts Comprehensive Assessment System (MCAS) test after two-three years and significantly outperformed them after seven-eight years on all MCAS subtests.”

She continues:

Furthermore, there is research … that shows that children who study a foreign language, even when this second language study takes time away from the study of mathematics, outperform (on standardized tests of mathematics) students who do not study a foreign language and have more mathematical instruction during the school day. Again, this research upholds the notion that learning a second language is an exercise in cognitive problem solving and that the effects of second language instruction are directly transferable to the area of mathematical skill development.

Of course futurists and brain augmentation wonks will argue that we’re fast approaching a point at which the distinction between brains and computers becomes irrelevant. But in the meantime, amazing as technology like this is — in particular its promise to let us speak any language (estimates put the number of “living languages” in the world at just under 7,000) — it raises interesting new questions about the role of cognitive externalization: the pros and cons of handing increasingly more of what we used to do with our biological “human tech” over to computers.

Allow me to paint a scene of the future: This invention will allow for a much quicker acceptance of a one world-known language. Under UN mandates, parents will be required to set their government-provided devices to English on Monday, Chinese on Tuesday, Spanish on Wednesday, and Arabic on Thursday, while spending the rest of their time speaking their native language. Thus, children around the world will learn 4-5 languages as first languages. This could be groundbreaking.

Also, it might change the accepted grammar rules by reimposing the computer's grammar over what is accepted by native speakers, so that what we speak changes because of how computers translate things....

There are 2 schools of thought, on the one hand, certainly the brain needs exercise in order to expand its functionality, on the other hand learning things that are not necessary, or are redundant or outdated likely uses up space that could have been put to better use.

I always had a resistance to learning additional languages because I thought it was difficult enough to master the intricacies of one without trying to compound the problem and possibly additionally because learning them proved difficult for me also possibly because of the preconception noted above.

My belief is that computers, the Internet and soon apparently language translators free us from having to learn unnecessary or non-productive information and provide greater "space" for learning about the ever increasing complexities of our modern world and how to interact with it successfully.

It does not seem we have been made intellectually lazy by these advances, we have just adjusted to make use of them and move on.

@GaryMcCray ha ha .. see , if you knew how to speak another language, you wold talk differently.

Learning another language is not about the language, it's about the culture it comes with.

Learning another language is not only not going to be useless, it will actually expend your brain.

The concept that learning a second or third language takes up space shows you don't know how the brain works.

Gosh there are so many reasons u r wrong .. imagine a world where the only way for different cultures to communicate is using a translator system that maps one culture onto the other .. it defies the definition of culture and language.

I could give you one million examples why u r wrong.

A language is alive ... and it evolves. Start Trek universal translator will never exist unless we manage to invent a thinking computer. Never say never .. but if that happens, then the ramifications are going to be a lot more significant than learning languages

I spent four years under essentially forced Spanish instruction, went to Mexico several times and could communicate fairly well with the locals.

Nonetheless, in order to learn about or to interact with foreign cultures for most of us it is not necessary to learn their language, and how much richer experiences if you can interact freely with a much larger assortment of cultures regardless of your language deficiencies.

The million examples you can give me are from then, not from now.

Our primary problems are incorporated in our inability to understand each other at a time when information transfer is for all practical purposes globally instantaneous.

Trending to a common language and in the mean time accurate computer translators are probably necessary for our survival, if not for convenience.

Without massive global cultural convergence we are all doomed.

BTW I am sincerely hoping for a thinking computer and view that as one of the main possibilities for continued human existence.

When I say I am looking forward to a thinking Computer I am referring to Ray Kurzweil and the singularity which hopefully will see fit to integrate humans, but if not, that's OK too.

As for China, certainly they will have enough economic force to be able to extend their language(s), but that isn't what is going to happen, aside from being technologically unwieldy in the extreme, China is cursed (you might say blessed) with having an incredibly diverse and not mutually decipherable batch of dialects.

English is much easier to learn both spoken and written and unless there is a completely unforeseen cultural backlash it will continue to be adopted there, eventually displacing their own languages. And China is probably the most resistant culture.

Finally, although you certainly can enhance or possibly improve your understanding of a culture by learning their language, it is not in any way essential, and in the new global world, the necessity is to interact with far more cultures than you could ever possibly learn the languages for.

No culture that is alive will stick to play the intermediary role, it will become adopted and mix into the culture/languages of those who adopt it as a mean to intercommunicate. Most known examples, James Bond.

U can't adopt a language without learning its culture, that's what you don't seem to understand, because without the cultural layer, you lose the meaning.

The massive global culture convergence is already happening and it is using English now .. but the Chinese will not adopt more English than they do now ... the more China develops the less the need for them to learn English ... u can find so many Americans who can't speak but English , because everybody else has done the effort to learn English, so there is no need for Americans to make the effort... but trust me, it will change, So far not speaking Chinese has not given much disadvantage to anyone, but knowing Chinese, that is understanding the culture, will start giving advantage to those who do and so the role of English as intermediary language is destined to change.

Here is the issue with a thinking computer ... how do you know it is telling you the truth? That's the difference between a technical translator that will not get you laid when visiting China (unless you go for prostitution services where language is zero, money does the talking).. and one that thinks, to make sure it is telling you the truth, it has to be your buddy ..

Please note that to get laid with a prostitute when in China, you still have to be able to express the need for sex, and that is culture .. funny, isn't it?

When u r saying u look forward to a thinking computer, u are saying u look forward to artificial consciousness .. that is what Star Trek isn't telling you ... the thinking computer able to think needs to have an opinion .. and that is no longer a translator, it's a new life form, a new culture.

Adopting its service for translation, means creating a new intermediary not human culture at the service of two other culture who can't talk to each other,

A truly thinking computer wouldn't take long to understand it has power. See what I mean?

All that said .. technical support for translation is welcome, but will not replace the need to learn the culture and its language .

By the way ... believe it or not, you don't need to be literate to learn a new language. That is the definitive proof that true translation is not a technical issue.

>First, the company converts spoken English word-by-word into Chinese text.

Wouldn't it be more useful to convert English into some form of pivot language ( http://en.wikipedia.org/wiki/Pivot_language ) first? With a pivot language, the number of translations goes up linearly rather than quadratically, as it is in 1:1 translations between all languages.

Plus, if you create a pivot language that can be written and spoken by humans, especially one with simplified and logical grammar and spelling, you can have the basis for a universal second language that people can use without the need for an external devices (which would somewhat mitigate the fears of universal monolingualism).

@snarfangel They tried creating simplified languages and experiments proved that no matter how simple you design an idiom it will be always harder to learn a language known to few than a language known to many. It's called collective cognitive. When you teach a lab rate the solution to a labyrinth all subsequent lab rats in the world will master the labyrinth faster if you design a new labyrinth with equal number of walls, dead ends and path length to goal it will happen again the more rats learn the way out the quicker each subsequent mouse will find the solution. Structural linguists have observed this phenomenon with language design too, the more people speak a language the easier it becomes for subsequent generation to master, many endangered indigenous languages have a very simple structure and vocabulary

Microsoft did take on one of the most severe cases, Chinese (any dialect) and English are near diametric opposites in pretty much every way imaginable.

I do recall Esperanto which could sort of qualify as a simplified "pivot" language, but the problem then was everybody had to learn it, nobody already spoke it. Really hard to get it going.

The reality is that there are 2 forces currently at work.

There is already a very strong drift towards the language where most global interaction takes place, English and towards the easier to learn Spanish / Latin based languages also largely accepted in International commerce.

Russians are much more likely to now English than Americans are to know Russian.

This drift seems likely to continue and eventually English will become the commonly accepted global language, not because it is better, simply because it is what is already there.

Language translators can also help with this transition, by removing global cultural boundaries and establishing the value of a common communication medium even more deeply than it is now.

In the end, language in the modern connected world is something everybody needs in common.

@snarfangel theoretically yes, the problem is that a language is culture dependent ...meaning it takes meaning only if it used ... so the pivot language works in theory only because nobody will ever learn it. Americans don't learn a second language ... the culture wart will start when the Chinese will stop learning English to assert their culture.

Heh, it posted a picture of the quadratic form from the Wikipedia article I linked. You can compare the formula above with (N-1). For 1000 languages, you are comparing ~500,000 possible translations vs. ~1000 for the pivot language.

Interesting philosophical question you raise, Matt, regarding downsides. It's important we always look to technology as a great assistant that enables humans to expand our own brain functions and unique abilities further. In other words, we humans will need to resist the urge to become complacent and let technology take over. It is up to us to use technology as a leverage for greater thinking, greater collaboration.