Menu

Building AR Applications with Unity and IBM Watson

Over the last days I’ve enjoyed playing with Unity and the IBM Watson SDK, which allows using cognitive services like speech recognition in Unity projects. With this technology you can not only build games, but also other exciting scenarios. I’ve extended an Augmented Reality application from my colleague Amara Keller which allows iOS users to have conversations with a virtual character.

The picture shows a printed piece of paper with a pattern. When using the app, the 3D character shows up on the paper. Users can have conversations with the character, for example:

The main logic is in this file. Let’s take a look how to use the Speech To Text service as an example. First you need to initialize the service with credentials you can get from the IBM Cloud. The lite account offers access to the Watson services, doesn’t cost anything and you don’t even have to provide a credit card.

The application also showcases how to use Watson Assistant and Watson Text To Speech in addition to Watson Speech To Text. Check out the open source project for details.

One important thing to keep in mind when using the three Watson services together is the timing. For example you should stop recording before playing an audio clip received from Watson Text To Speech so that Watson doesn’t listen to itself. Also you need to make sure to only play one clip at a time.

I’m neither a Unity, nor a C# expert. So I’m sure there are better ways to do this. Below is how I’ve solved this. I start the recording again only after the duration of the audio clip.