How Google's Ear Hears

If you own an iPhone, you can now be part of one of the most ambitious speech-recognition experiments ever launched. On Monday, Google announced that it had added voice search to its iPhone mobile application, allowing people to speak search terms into their phones and view the results on the screen.

In designing the system, Google took on an enormous challenge. Where an automated airline reservation system, say, has to handle a relatively limited number of terms, a Web search engine must contend with any topic that anyone might ever want to research--literally.

Fortunately, Google also has a huge amount of data on how people use search, and it was able to use that to train its algorithms. If the system has trouble interpreting one word in a query, for instance, it can fall back on data about which terms are frequently grouped together.

Google also had a useful set of data correlating speech samples with written words, culled from its free directory service, Goog411. People call the service and say the name of a city and state, and then say the name of a business or category. According to Mike Cohen, a Google research scientist, voice samples from this service were the main source of acoustic data for training the system.

But the data that Google used to build the system pales in comparison to the data that it now has the chance to collect. "The nice thing about this application is that Google will collect all this speech data," says Jim Glass, a principal research scientist at MIT. "And by getting all this data, they will improve their recognizer even more."

Mobile phones are assuming more and more computational duties; in much of the world, they're people's only computers. But their small screens and awkward keyboards can make text-intensive actions, like Web search, frustrating. While mobile browsers are getting better at predicting your search terms, and thereby reducing the amount of typing, nothing is quite as easy as speaking directly into the phone.

Speech-recognition systems, however, remain far from perfect. And people's frustration skyrockets when they can't find their way out of a voice-menu maze. But Google's implementation of speech recognition deftly sidesteps some of the technology's shortcomings, says Glass.

"The beauty of search engines is that they don't have to be exactly right," he says. When a user submits a spoken query, he says, Google's algorithms "just take it and stick it in a search engine, which puts the onus on the user to select the right result or try again." Because people are already used to refining their queries as they conduct Web searches, Glass says, they're more tolerant of imperfect results.

Even after the search application loads, the voice-recognition system kicks in only when the user puts the phone to her ear, as determined by its built-in motion sensors. "If you're listening all the time, then you trigger false positives," Glass says. "The typical solution is to make you push a button," but the motion-activated system is easier and more intuitive, he says.

The search application also uses the iPhone's built-in location-awareness system to prioritize results. For instance, if you search for Bank of America, one of the results will be a map of local branches. This saves users from having to include location terms--which can be open to misinterpretation--in their queries.

While Google won't disclose details about how its voice-recognition system works, it probably hasn't done anything too radical, says Nelson Morgan, director of the International Computer Science Institute, in Berkeley, CA. "Nearly everybody who does speech recognition has a system that looks about the same," he says. First, the system analyzes frequency characteristics of the voice input. Then, based on probabilities drawn from a huge number of real-world examples, it correlates them with words. Finally, those words are fed into a language model that uses common combinations or sequences of words to resolve ambiguities. For instance, if you say, "president of the United," it's likely that the next word is going to be "States."

While Google isn't announcing plans to use its voice-recognition technology for other services, the potential is easy to see. "Now we have tech to take spoken words and convert it to text," says Gummi Hafsteinsson, a senior product manager at Google. "There are a lot of options." Currently, there's no way to use your voice to access Google's calendar or e-mail applications or to write an e-mail or a text message. But that could change in the future. "I think this opens up a whole new dimension," Hafsteinsson says.

FOREVER BRUCE LEE

About Me

Great Minds Have Similar Thoughts

Champions aren't made in gyms, champions are made from something they have deep inside them - a desire, a dream, a vision. They have to have last-minute stamina, they have to be a little faster, they have to have the skill and the will. But the will must be stronger than the skill.-Muhammad AliI'll be more enthusiastic about encouraging thinking outside the box when there's evidence of any thinking going on inside it.- Terry PratchettNot to be absolutely certain is, I think, one of the essential things in rationality.- Bertrand RussellWhat we think, or what we know, or what we believe is, in the end, of little consequence. The only consequence is what we do.Sometimes what's right isn't as important as what's profitable.- Trey Parker and Matt StoneThere are only two kinds of people who are really fascinating: people who know absolutely everything, and people who know absolutely nothing.- Oscar WildeSometimes I lie awake at night, and I ask, "Where have I gone wrong?"/ Then a voice says to me, "This is going to take more than one night."- Charles M. SchulzThere is nothing worse than aggressive stupidity.- Johann Wolfgang von GoetheThe significance of man is that he is insignificant and is aware of it.- Carl BeckerA lie can travel halfway around the world while the truth is putting on its shoes.- Mark Twain"If you know how to spend less than you get, you have the philosopher's stone." So said Benjamin Franklin more than 200 years ago. How much easier it is to be critical than to be correct.- Benjamin DisraeliOf course the game is rigged. Don't let that stop you--if you don't play, you can't win.- Robert HeinleinAbility will never catch up with the demand for it.- Malcolm ForbesNo man remains quite what he was when he recognizes himself.- Thomas MannNo man needs a vacation so much as the man who has just had one.- Elbert HubbardThere is no pleasure in having nothing to do; the fun is in having lots to do and not doing it.- Mary Wilson LittleBooks to the ceiling,/ Books to the sky,/ My pile of books is a mile high./ How I love them! How I need them!/ I'll have a long beard by the time I read them.- Arnold LobelLeif Ostling said in a statement that his comments about Germany had been "interpreted in a way that was not intended."If a man will begin with certainties, he shall end in doubts; but if he will be content to begin with doubts he shall end in certainties.- Sir Francis Bacon"It's not the voting that's democracy, it's the counting."- Tom StoppardElections are won by men and women chiefly because most people vote against somebody rather than for somebody.- Franklin P. AdamsInvention is the mother of necessity.- Thorstein VeblenDon't try to solve serious matters in the middle of the night.- Philip K. Dick