A blog for developers programming with Autodesk platforms, particularly AutoCAD and Forge. With a special focus on AR/VR and IoT.

July 25, 2016

Adding spatial sound to our Unity model in HoloLens – Part 1

As mentioned last time, upgrading to Windows 10 has opened the door to spatial sound in our HoloLens model.

The foundational step is to set our Unity project’s audio settings to have the Spatializer Plugin as the “MS HRTF Spatializer”: if you don’t see this option then you may need to upgrade your OS (as I did).

At this point we need to think a little about how best to implement spatial sound in this project, particularly as it literally contains a number of moving parts. I see three main alternatives:

A single sound is assigned to our robot

When the robot stops completely, so does the sound

The same sound is assigned to each of the robot’s parts

When each part stops moving, so does the sound for that part

A different sound is assigned to each of the robot’s parts

When each part stops moving, so does the sound for that part

In some ways option 1 is simplest – we only need to attach a single sound to the root of the robot – but in others it’s the most complex, as we have to keep track of which parts are moving. But we’ll start with that and see how it goes: I have a sneaking suspicion that adding a sound for each part (i.e. options 2 & 3) add unnecessary (and unwelcome) processing overhead for the device.

The first order of business is to add the chosen sound as an AudioSource to our root object (by dragging and dropping the project asset) and set its properties according to the Holograms 101 tutorial:

I’ve added a Buzz.cs script to manage the “buzz” of the robot, based on the movements of the various parts:

using UnityEngine;

using System;

publicclassBuzz : MonoBehaviour

{

publicint parts = 6;

privateint stopped = 0;

privateint max = 0;

privateAudioSource audioSource;

void Start()

{

// Calculate the sum of all possible part flags

max = 0;

for (int i = 0; i < parts; i++)

{

max += (int)Math.Pow(2, i);

}

// Store the audio source for later access

audioSource = this.gameObject.GetComponent<AudioSource>();

}

publicvoid StartPart(int part)

{

// If our part is on the stopped list, remove it

int flag = (int)Math.Pow(2, part);

if ((stopped & flag) > 0)

{

stopped -= flag;

}

// Start the audio if it isn't already playing

if (audioSource && !audioSource.isPlaying)

{

audioSource.Play();

}

}

publicvoid StopPart(int part)

{

// If our part isn't on the stopped list, add it

int flag = (int)Math.Pow(2, part);

if ((stopped & flag) == 0)

{

stopped += flag;

}

// If all parts are stopped, stop the audio

if (audioSource && (stopped == max))

{

audioSource.Stop();

}

}

}

We do this by having a “number of parts” setting that tells us how to populate a “powers of 2” value (much as you might do with enumeration flags you need to keep track of). So if the value is 1, we know that part 0 is stopped, while if it’s 23, we know parts 0, 1, 2 & 4 are stopped (23 = 1 + 2 + 4 + 16 = 20 + 21 + 22 + 24). We can pre-compute the maximum value: all we need to do to check whether all parts are stopped is to see whether the stopped value has reached this maximum. At which point we stop the sound.

Then we need to update our Rotate.cs script to allow for a unique number assigned to each part – something we do in the Unity UI – as well as providing a simpler Start/Reverse/Stop/TogglePart interface:

using UnityEngine;

publicclassRotate : MonoBehaviour

{

// Parameters provided by Unity that will vary per object

publicint partNumber = 0; // Part number to help identify when all are stopped

The PartCommands.cs script then needs to be updated to call through to this new interface, rather than modifying the part directly. Which actually makes it a bit simpler:

using UnityEngine;

using System;

publicclassPartCommands : MonoBehaviour

{

// Called by GazeGestureManager when the user performs a Select gesture

publicvoid OnSelect()

{

CallOnParent(r => r.TogglePart());

}

publicvoid OnStart()

{

CallOnParent(r => r.StartPart());

}

publicvoid OnStop()

{

CallOnParent(r => r.StopPart());

}

publicvoid OnQuick()

{

CallOnParent(r => r.isFast = true);

}

publicvoid OnSlow()

{

CallOnParent(r => r.isFast = false);

}

publicvoid OnReverse()

{

CallOnParent(r => r.ReversePart());

}

privatevoid CallOnParent(Action<Rotate> f)

{

var rot = this.gameObject.GetComponentInParent<Rotate>();

if (rot)

{

f(rot);

}

}

}

Now let’s take a look at this approach in action. This video shows the robot buzzing fairly discreetly – you only really notice that it’s not background noise when the movement stops completely. You don’t really get a sense for the spatial nature of the sound: if you turn your head away from the hologram, you definitely hear the sound behind you. Which is both really cool and apparently an important part of designing a compelling holographic experience.

In the next few posts I expect to continue exploring spatial sound by adding sounds at the part level in our Unity model.