What is Sound “Research”?

This is a guest contribution by Karen Collins. Karen is the Canada Research Chair in Interactive Audio at the Games Institute, University of Waterloo, Canada, and the director of Beep: A Documentary History of Game Sound. She’s been researching game audio for the past fifteen years, and in the process, published four books and nearly 100 research papers on sound. As Albert Einstein said, “If we knew what it was we were doing, it would not be called research, would it?” Karen proudly admits she has no idea what she is doing. @GameSoundDoc beepmovie.com

Don’t we already know what sound is? What do we need research for? I’m often met by surprise or confusion when I tell people I do research in sound. It may help to explain a few research projects that I’ve worked on in recent years to share the types of research that can be done in sound. These were for the most part done in a university setting, although some of the projects received some funding or support from private partners (e.g. Google, Microsoft)—I’ll talk about the academic-industry research crossover below.

1. Surface computing audio research: For a while about seven or eight years ago, surface (table-top) computers were “the next big thing”. We saw these computers on CSI and other TV shows that depicted the latest, greatest cutting edge technology. The use of these computers gave rise to some interesting questions when it came to sound: If there are sonic notifications, for instance, how do we indicate who around the table is being notified? Where should the speakers be positioned, and do we need to change mixing practices if, for instance, there are a group of people playing a game around a table computer? These are the kinds of questions I explored with some colleagues over a number of years, and we built audio-only games as a way to test not only user experiences, but also to test the use of games as a testing method in usability/user experience research.

2. Networked mobile audio research: One year at Project Bar-B-Q: The Interactive Audio Think Tank, I was part of a workgroup looking at what might be the coming in the world of mobile audio. This was just before Google launched their first Android phone. When the Nexus One was launched, I took some of the ideas we proposed in the workgroup and explored them. One of these ideas was to use individual phones as an array of speakers, so I could have one master phone and send out audio to a bunch of slave phones. You can imagine standing in a crowd at an ad-hoc protest where there’s no loudspeaker available, and instead one person could use their phone as a microphone, and the rest could become loudspeakers. We built it and published the work at SIGGRAPH and AES.

3. Gambling sound research: How does sound contribute to gambling addiction in slot machines? Over a number of years, myself and a group of colleagues conducted psychophysical research (EEG, heart rate, sweat response) into how sound contributed to excitement and addiction on slot machines. We looked at sound on/off, negative sounds, and using sound to draw attention. We found that sound did contribute to excitement and potential addiction, and our work has significant relevance to today’s “freemium” mobile games that now often cross over into the realm of gambling.

4. Understanding our spatial relationship to sound in media research: How do we subjectively position a listener through recording and mixing techniques? We’re all familiar with the term “point of view”, but what about the audio equivalent? What is the auditory position of a listener, and how does a change in that subjectivity—a change in our auditory “point of view” change our relationship to the characters on-screen? For instance, a close-mic feels intimate, whereas more distant mic recording puts us at a physical and subjective distance from what we see. Louder sounds in the mix tend to appear closer, and so on. So we can create a virtual subjective space where we position a listener, equivalent to point of view in visual aspects of film or games.Collins, Karen.

5. Multimodal interactions in virtual environments and S3D research: How does sound influence our perception of visuals? There’s lots of research that suggests that sound can smooth out a glitchy animation, can influence product preference, or change the way we perceive time and colour. I’ve been part of a team that for years has explored how sound influences perception in virtual environments—in our case, looking at ambient sound in medical simulations, and looking out how sound influences depth perception in stereoscopic 3D.

6. Historical sound research: I’m still fascinated by game sound. How did it develop, why did it develop the way that it did? How did technology influence the aesthetic of sound in early games, and how did that change over time?

Collins, Karen. Game sound: an introduction to the history, theory, and practice of video game music and sound design. Mit Press, 2008.

[vimeo]https://vimeo.com/157076047[/vimeo]

7. Sound and interactivity research: How does our relationship to sound change when we interact with it? Most of our media consumption historically has been with us as an“audience”. With games, we’re co-creators of that sound in that our actions have sonic consequences. But our theories about sound in media are all based on traditional media. How do we need to change our theories to apply to game sound?

Collins, Karen. Playing with sound: A theory of interacting with sound and music in video games. MIT Press, 2013.

8. Binaural audio/experimental uses of sound: Statistically, more people watch films on tablets than televisions now. How can we better leverage headphone sound, and create film designed for tablets? I created an experimental film using some binaural recordings, and created the film specifically for tablets that are laid flat on a table (like a surface computer!), leveraging some of the earlier work I did on sound for surface computing. The film premiered out in L.A. in 2013, and sounded AWFUL in the theatre, since it had been mixed for headphones and binaurally. But it was still a blast, and I love being able to create non-commercially viable work just for the experimental value.

(forthcoming: K. Collins and R. Dockwray. Experimental Sound Mixing for “The Well”, a Short Film Made for Tablets. Leonardo Journal.)

As you can see, there are lots of different types of research that gets undertaken in relation to sound—and I’m just one researcher and this isn’t a complete list! There are all kinds of interesting research being done around the world relating to sound, and to sound in games. Some of it is immediately practical and relevant to practitioners, and some of it is more basic, theoretical research.

What use is this type of research?

My grandmother once scolded me for not spending my life and my brain on something “useful” like medicine. I’d like to think that my work is useful, but asking myself if what I’m doing is relevant and useful is something that I struggle with regularly, and always try to make my work as widely accessible and “useful” as possible. Although a lot of academic research languishes in obscure journals or remains unpublished, most academics today aim to get their research to a wider audience at some stage. There is increasing pressure on academics to show impact beyond academia as one of the metrics our jobs are measured by. Some of the impact is immediately obvious—for instance, the gambling work I did with my colleagues was used to change public policy in Tasmania and Australia, and used in court cases in the United States—but other research takes longer to percolate through to general knowledge. This can mean appearing in text books a decade down the road, or used to explain concepts to more general audiences. Winifred Philips’ book, A Composers Guide to Game Music, for instance, draws on quite a lot of research that’s been undertaken in areas of music psychology. As much as “useful” is important, in other words, it’s nearly impossible to determine in advance if a research project is “useful”. It may be useful in the future, if not right now. I’m reminded of a medieval studies scholar who recently decided to translate some old Viking texts, and worked with a biologist to try out a “recipe” they found for a health tincture. Turns out, it’s a new anti-bacterial that can work against antibiotic-resistant strains of bacteria. Nobody would have thought translating Viking texts would be “useful”, and people likely would have objected if public “taxpayer” funding was used to pay for such work, but some of the greatest inventions in history have come from just trying to answer a question without knowing where the answer will take us. So don’t judge “useful” as just “obviously useful right now”.

Academic/Industry Collaboration: What’s it all about? What challenges and opportunities are there? Should you engage in research with academics (or industry?)

Much of the research grant funding available in Canada and elsewhere (essential for us academics to get and keep our jobs these days) requires some form of industry partnership. We are required to find industry partners to support the work we’re doing, to put up funds (either cash or “in kind” which can be hours of work, equipment, and so on) or even to provide specific problems that they’d like us academics to help them solve. Finding industry partners and working with industry is increasingly a big part of our job. We can offer all kinds of opportunities for research—both immediate, practical problem-solving because we may have equipment or skills that the company lacks and we can help solve an issue for the company in the short term, but also less immediate, long-term and more experimental research. For instance, if a company comes across a problem that they know can’t be solved in the short-term and they can’t invest completely in the problem (quantum computing, for instance, which would revolutionize so many industries), they may help to fund on-going research in exchange for partial ownership of any results, or just because they’re interested in investing in on-going, curiosity-driven research (which can often yield surprising results).

But there are challenges working with academics. Everything moves very, very painfully slowly in academia as the layers of bureaucracy and red tape stamp any hope of quickly getting a project going. Part of the hold-ups are usually IP (intellectual property) agreements between the company and the university. Who will own any inventions that come out of the research? Will students who are hired to work on the project get some ownership? Will an invention prevent the faculty member(s) from publishing the work for a time (which can slow academic careers)? What are the responsibilities of the academics and the company involved? It’s never easy to answer these questions quickly, and different institutions have different policies. I’ve had more than one project completely scuppered by inability of the legal teams to come to an agreement. Faculty also have many duties—most have research defined as 40% of their jobs, but the other 60% tends to take up more like 100% of the time, so research tends to be done on our own time, or by cutting back on other tasks where we could. But if you’re looking for someone to work “full time” on a project, you’ll have to get the funds to offer a PhD studentship or Post-Doctoral fellow that the faculty member can supervise on the project. Supervising students also takes time, so even hiring a bunch of students to work on a project still requires a faculty member to oversee them.

Who does this sound research?

I’ve talked about academics specifically here, since probably the bulk of sound-related research gets done by academics, but they’re not the only ones who do sound research. Research is undertaken at larger companies—for instance, I recently saw an open position at Oculus looking for people to conduct VR sound research, and there are research positions that open up at Microsoft, Google, Apple and other major tech companies. There’s also on-going sound research at all kinds of companies that service the audio market—creating software and hardware, developing the next generation audio tools, and so on. There are also amateurs who conduct what I would consider sound research—field recordists who document sounds in the changing environment, amateurs who build their own instruments and tools, and so on. Amateurs don’t tend to subject their work to formal peer review, in the traditional sense of having evaluation by other experts in a formal way, but these days the Internet forms its own kind of peer review system, and publishing ideas and work online, in my mind, is as important as more traditional conference-and-papers venues.

Should I do sound research?

What is research, exactly, and should you do research? To my mind, there’s always room for new knowledge, and research is about answering questions and discovering new knowledge. There are so many interesting open questions when it comes to sound, especially sound in media, that there is no reason not to conduct research. It may be helpful for you, however, to check with someone who is trained in more formal methods to make sure your research will stand up. For instance, let’s say I want to conduct some basic research into how many people use Beats headphones to play games. So I post the question “Do you use Beats headphones to play mobile games?” on an internet game forum for casual gamers. I’m going to get results from a very specific group of people if I do that: who uses that forum? How many people are lurking and don’t answer? Not only that, but are people who don’t use Beats headphones more likely to answer, versus those who do? Understanding all the nuances of methodology (not to mention how to analyse the data you collect) can be important to making sure you get the answers that you’re looking for! You can conduct research yourself, of course, but you’re more likely to get accurate (and therefore useful) data if you talk to someone trained in how to conduct that type of research.

Here’s some tips to think about before you embark on sound research:

1. Start by asking yourself: What is the aim of my research? What question(s) do I want to answer? Knowing exactly what you want to answer is the first step in formulating your plan of action.

2. Break it down into smaller steps. You can’t answer “What impact does sound have on VR?” because there are so many aspects to that question, that you could never design an experiment to answer it. But you could break it down into, “What impact do footstep sounds have on the player’s VR experience?” Now you can design an experiment!

3. Design your experiment. We’re doing science here, kids! You can’t do research without going back to your grade 7 (at least, that’s when I learned it) hypothesis/methodology/ results formulation for a science experiment. So let’s say I want to look at footstep sounds in VR. I might hypothesize that footstep sounds help to ground the player in the game world and make the world feel more realistic. My methods are going to be to test players in a game with and without footstep sounds.

4. Build your testing material that you need: Build the game with and without sounds, but make sure to think about how others might poke holes in your argument here. Are there other sounds in the game? Those should be exactly consistent across both tests (sound on/off). How will you make sure that other sounds are consistent? Will you have male and female sounds? Is the character in first person? These things all matter. Accounting for all variables in your experiment is the hardest part, but so important.

5. Build your data gathering. How will I find out if footstep sounds help to ground the player in the game world? Will I do psychophysiological measures like heart rate? Will I film them and analyse and code the results? Will I just ask them? How will I ask them—during, or after they play? You need to figure out the best way to get the answer to your question, and often we’ll use a variety of measures—for instance, psychophysiological (which are objective), and questionnaires (which are subjective). Will I interview them (subjective, and qualitative) or just tally up a Likert scale (“scale on 1 to 7” etc.) results (subjective, and quantitative)… It’s important to know if your results are more qualitative or quantitative, subjective or objective, and acknowledge that, since people have a tendency to lie in subjective responses (What? No, I never had a drink before I became of legal age!)

6. Find your test subjects. This can be trickier than it seems, since you need to think through all the angles here too. What parameters might impact results? Do I need to do a hearing test on my subjects first? Do I need to use experienced gamers? Where will the test take place, and how do I account for external sounds that might impact results? (e.g. a passing siren, etc.) Thinking through all of these many parameters in advance helps to bolster your argument.

7. Analyse the data. Did you ever take Statistics and wonder why you had to do that? Now you know. Have fun with that. You have to find out if there is any statistical significance in all that data you just collected.

8. Write it up and share it. Be careful not to use words like “proof”. You didn’t “prove” anything, since there are always so many factors that come into play in an experiment, that the best you can do is have “indications that” or “show that” your results are valid. You have to describe your methods in such a way that someone else can repeat your results, in order for it to be good science. If it can’t be repeated, you didn’t even “show” anything!

A great place to share it is the new research track at GameSoundCon or here on Designing Sound! Watch for next year’s deadlines and get those experiments done!