Supporting Synonyms and Multi-Language Voice Input in Android

My Android Wear app, Resistor Decoder, takes voice input from the user. I wanted to support several ways that users could say the “same” thing while maintaining architectural support for multiple languages.

Synonymous Inputs

In my app, people speak the names of colors that they see on a resistor. In English, most of the relevant colors have a single name, but a user could reasonably describe this coloras either “violet” or “purple,” and the app should behave the same when given either name for what it thinks of as “violet.”

Android developers are already encouraged to use String resource files to tokenize Strings. The same basic mechanism works great for supporting synonymous inputs, with a slight modification.

The normal way to use String resources looks like this:

strings.xml:

<string name="violet">violet</string>

You might look for “violet” in your input like so:

// assumes you have a list of Strings from
// your speech input
String violet = this.getResources().getString(R.string.violet);
// check to see if the next input String is the
// word for violet

// assumes you have a list of Strings from
// your speech input
for (String violet : myresources.getStringArray(R.array.violet)) {
// check to see if the next input String is
// one of the possible words for violet
}

Each word or phrase that your app supports is sorted into arrays with other synonymous words or phrases. Each <item> under a given array name will be treated as if it’s the same input by the app. I recommend doing this even for words or phrases that have no synonyms, since this may not be true in other languages.

In my case, my inputs are just colors, but in your case, maybe it’s, “show me the forecast”, “what’s the weather going to be”, “what’s it going to be like tomorrow” etc. You can easily expand this list as you get feedback from users, or possibly from Analytics, about other phrases your users want to say.

Multi-Language Support

Instead of producing translations for one-to-one mappings between tokens and strings, as you’d typically do, you now produce translations for your string arrays. Because each language will have its own words and phrases to express the concepts of your original language, you must provide your translators with special instructions here. They are used to providing a single translation for a single word or phrase; this is quite a shift for them.

For example, “orange” is the only reasonable way to say “orange” in English, but in Korean, there are three ways. If you provided your translator with your string array that mapped the token orange to the word orange, they may just provide a single way to say “orange” in Korean. Likewise, asking a translator to translate both “violet” and “purple” into their language may seem like a strange request if they only have one word for that color. Take care to provide clear instructions so that the translator understands the task.

Once this is done, the code that supports synonymous inputs “just works” and automatically supports the arrays provided for any other language. For example, my app supports one way to say “red” in English but three in Korean.