Using Speech APIs in Windows Phone 8

In the previous article in this series on Windows Phone 8 app development, I explored the improved speech experience introduced in Windows Phone 8 and started developing an app that takes advantage of new extensible voice commands. In this article, I explain how to retrieve parameters when it is necessary to execute the voice commands and how to update phrase lists within the app. I also explain how to use speech recognition and text-to-speech features. This way, you will be able to combine voice commands, speech recognition, and text-to-speech to enable a voice-driven UX.

Checking Parameters and Executing Voice Commands

The Recipes app discussed in the first two parts of this series recognizes voice commands, but fails when it has to navigate to the target pages because the pages don't yet exist in the project. It is time to add the pages for each of the voice commands. I'll start with the simple AddNewRecipePage page that the app must display when executing the CreateNewRecipe voice command. To keep the sample app simple and focused on speech-related features, I won't use a Model-View-ViewModel (MVVM) approach  I'll just add the necessary pieces of code to the code-behind files. However, take into account that a MVVM approach is definitely suitable for these kinds of apps.

Then, enter AddNewRecipePage.xaml in the Name textbox and click Add. Visual Studio 2012 will create and display the preview for a new portrait page (Figure 2).

Figure 2: The initial contents for the new Windows Phone Portrait Page with the ContentPanel control selected.

A StackPanel, also known as the title panel, contains the name of the application and the page title. Obviously, I don't want the recipes app to display "MY APPLICATION" and "page name" in the page that adds a new recipe. You will notice the following lines that define the title panel don't use resources to retrieve the app's title. Thus, if you change the app's title in the app manifest, nothing will change in the pages you added to the project, because they will still display "MY APPLICATION:"

I have to add another page to handle the other voice command, so I want to use resources to define the localized app name. Double click on AppResources.resx within the Resources folder. You will notice the value for ApplicationTitle is "MY APPLICATION" (Figure 3). You can change it to your desired apps's title and use the static resource in the different pages. In this case, it is necessary to set "RECIPES" as the value for ApplicationTitle.

Figure 3: The resources defined in AppResources.resx.

Add a new resource named AddNewRecipePageTitle, set its value to "add a recipe" and change the lines that define the title panel in AddNewRecipePage.xaml to make the two TextBlock controls use the localized string resources and follow best practices:

Now, add a Loaded event handler named AddNewRecipePage_Loaded to the AddNewRecipePage page. Then, add the following code to the event handler and establish a breakpoint at the if statement:

private void AddNewRecipePage_Loaded(object sender, RoutedEventArgs e)
{
//// Check whether a voice command launched this page
if (this.NavigationContext.QueryString != null
&& this.NavigationContext.QueryString.ContainsKey("voiceCommandName"))
{
//// A voice command launched this page and I want to retrieve the command name
//// included in QueryString
var commandName = NavigationContext.QueryString["voiceCommandName"];
//// Now, you can execute code according to the commandName value
//// and retrieve values for additional parameters required by the commandName
}
}

Build and start debugging the app in the emulator. After the debugging session executes the app, activate GSE and say "Recipes create new recipe." The phone should recognize the command, launch the Recipes app, and navigate to the AddNewRecipePage. Thus, the AddNewRecipePage_Loaded event handler will be fired and the debugger will stop at the breakpoint. Inspect the value for the NavigationContext.QueryStringDictionary<string,string> (see Figure 4). Visual Studio displays two key-value pairs for the dictionary, as shown in Table 1:

Key

Value

voiceCommandName

CreateNewRecipe

reco

Recipes create new recipe

Table 1: Key-value pairs.

Figure 4: Inspecting the NavigationContext.QueryString Dictionary<string,string> for the AddNewRecipePage page after the voice command launches the app.

In this example, the phone navigated to the page as a result of the execution of a voice command. However, the same page might be loaded in other scenarios where voice commands aren't involved, so you cannot assume that NavigationContext.QueryString is always going to have values when the Loaded event is fired. That's why the previously shown code uses an if statement to check whether NavigationContext.QueryString is not null and contains the "voiceCommandName" key. If that's true, it means that a voice command made the app navigate to this page.

In our example, there is just one voice command that can navigate to the AddNewRecipePage. However, it is possible to have many voice commands that can navigate to the same page and perform different actions in the page. That's why it is important to check the value for the "voiceCommandName" key. In addition, you can check the complete recognized speech by retrieving the value of the "reco" key. Depending on the value of the "voiceCommandName" key, you might need to retrieve additional parameters whose values are included in additional key-value pairs for the dictionary.

Retrieving Parameter Values for the Voice Commands

The SearchRecipe voice command defined in the previous article is an excellent example of a voice command that requires you to retrieve many parameters to perform the search. Thus, it is necessary to add the page that this command requires. Right-click on the Recipes project in Solution Explorer, select Add | New Item… and select Windows Phone Portrait Page. Enter SearchRecipesPage.xaml in the Name textbox and click Add. Visual Studio 2012 will create and display the preview for a new portrait page.

As done with the previously created page, I want to use resources in this page. Double-click on AppResources.resx within the Resources folder. Add a new resource named SearchRecipesPageTitle, set its value to "search recipes." Change the lines that define the title panel in SearchRecipesPage.xaml to make the two TextBlock controls use the localized string resources and follow best practices:

Build and start debugging the app in the emulator. After the debugging session executes the app, activate GSE and say "Recipes search sandwiches with tomatoes." The phone should recognize the command, launch the Recipes app and navigate to the SearchRecipesPage. Thus, the SearchRecipesPage_Loaded event handler will be fired and the debugger will stop at the breakpoint. Inspect the value for the NavigationContext.QueryStringDictionary<string,string> (Figure 5). Visual Studio displays five key-value pairs for the dictionary, as shown in Table 2:

Key

Value

voiceCommandName

SearchRecipe

Reco

Recipes search Sandwiches with tomatoes

recipeElements

tomatoes

recipeConnectors

with

recipeTypes

Sandwiches

Table 2. Five key-value pairs.

Figure 5: Inspecting the NavigationContext.QueryString Dictionary<string,string> for the SearchRecipesPage page after the voice command launches the app.

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task.
However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Video

This month's Dr. Dobb's Journal

This month,
Dr. Dobb's Journal is devoted to mobile programming. We introduce you to Apple's new Swift programming language, discuss the perils of being the third-most-popular mobile platform, revisit SQLite on Android
, and much more!