Audio Design For Interactive Narrative VR Experiences

[11.15.18]- Larry Chang

Audio is important in many forms of storytelling. George Lucas once said "Sound is half of the experience" It is even truer when it comes to Virtual Reality because the sound in VR can directly affect the immersion of the experience. For example, if we pick up a bottle in VR and drop it on the ground and it creates no sound or the sound does not match our expectation, then our subconscious will tell us immediatelythat there is something wrong in the virtual world we are trying to create. Because sound doesn't work like that in our daily lives, we have to make sure the audio experience is intuitive enough that the audience won't disconnect them from believing in the virtual world. Without a proper audio design, the narrative experience will lose its charm and the world won't be as convincing.

While helping to create the interactive narrative of The Price of Freedom, I have come up with some ideas about what works and what doesn't. Consider this as a post-mortem for The Price of Freedom, and I am eager to share with you some of the lessons that I learned.

The Price of Freedom is an interactive VR story based on the declassified true event of Project MK Ultra. Project MK Ultra was a highly controversial project that ran from the 1950s to the 1970s in which the CIA experimented with several methods including using LSD to brainwash its subjects. In the story, you take the role of a CIA agent and are given a task to kill Ben Miller, a radical who broke into CIA facilities and stole top-secret chemical weapons research. By exploring in the environment, the player discovers the true reason for their mission and who they really are.

Audio design for interactive experiences can be categorized into four elements: Sound Effects, Music, Ambience, and Dialogue. But how should we prioritize making these elements and where should we place them when it comes to VR?

1. The Importance of Collision & Interaction Sound

Integrating collision and interaction sounds into the experience in a very early stage of the production is a good idea. This can not only help the designer or developer of the team have a more concrete sense about how the virtual world they designed feels, but also help audio professionals examine how things actually sound in the VR environment so that we can spot potential problems and make changes as early as possible.

The sound experience needs more variation to be closer to reality in order for us to feel comfortable. Lacking diversity breaks the immersion because we don't find it natural. So here are the solutions:

At Construct Studio, we designed a sound system that had two features. First, we created a list of common material sound folder like wood, metal, paper, stone and so on. Then we assigned them to every object based on their features. Second, the system receives the info of how strong the object's collision force is each time it collides and then divided them into three groups: light, medium and heavy. Each group has a list of sound clips that will be selected to play randomly each time when the sound is triggered. On top of that, we also added the pitch variation to the sound clips to make sure that each time it would sound slightly different.

We have to apply this sound system to almost every object in the scene. Since the player has the freedom to interact with mostly everything they want in the scene, we have to make sure that everything they touch has satisfying sound feedback that won't break the immersion. But if there are so many audio effects around the players, how are they going to receive more important information and find out what the story is about in VR?