We go behind the scenes at Valve, Bungie, and Epic Games to see how user feedback helps shape the game development process.

Inside a dark building high above the Seattle landscape, a group of gamers wait patiently to be strapped to a voltage battery. Thin wires peek out from under their sleeves as tiny round contacts are stuck carefully to each of their hands. Somewhere down the hall, a PC registers the first electrical pulse, rippling across the skin of one of the candidates. Whoever he is, his nerves have just given him away.

The playtesting rooms inside Valve’s headquarters resemble a large, untidy science lab. PC monitors line the wall, spewing out steady streams of graphs, patterns, and charts. Some keep track of candidates’ heart rates; others record their eye movements. A pile of resistors and voltage batteries lie discarded in one corner of the room; candidate information reports, video recordings, audio transcripts, written questionnaires, and the results of thousands of hours of direct observation lie in the other. Valve takes its playtesting seriously.

While playing video games for a living may sound like the dream ticket for any avid gamer, for those involved in the process, it represents much more than just another routine part of game development. Testing games before they are released gives game developers a rare insight into the end user experience, helping them validate the quality of the game and isolate potential problems. We now see games as much more than simple products; we see subjective experiences that affect individuals in different ways. For this reason, video game testing not only has to be more rigorous and pervasive than product testing in other industries, but also more precise. Can feelings be measured? Why does one player enjoy a particular game, while another does not? How can user feedback be used to make games better?

Publishers and developers are constantly seeking answers to these questions, trialing a variety of playtesting methods to get the best results. In this feature, GameSpot will go behind the scenes of three major studios–Valve, Bungie, and Epic Games–to find out how tools like science and psychology are helping game developers better understand the nature of player behavior.

Playtesting room or science lab?

A Healthy Dose of Perspective

Three years ago, Valve hired experimental psychologist Mike Ambinder to head up the playtesting department of its development arm. With a B.A. in computer science and psychology from Yale University and a Ph.D. in psychology from the University of Illinois, Ambinder was the perfect man to lead Valve in a new direction in the field of user research. Using his background in visual recognition, Ambinder began treating Valve’s playtesting sessions as a series of scientific experiments designed to test various game design hypotheses. Since most psychological research revolves around trying to isolate mechanisms of behavior to find out why people are motivated to do the things they do, Ambinder thought game playtesting should strive to do much the same thing: use extractable data to make an altogether better user experience.

“For us, playtesting is the most important part of the game development process,” Ambinder says. “It’s not something we save for the end of the development, or use as a quality assessment (QA) or balancing tool. Instead, it is the dominant factor that shapes our decisions about what to release and when to release it.”

This is a relatively new attitude for the games industry. While playtesting has always been a vital part of the game development process, the role it plays in shaping the final user experience has never been as important as it is now. Traditionally, playtesting methodologies focused on video games as products rather than as variable experiences that can affect players in different ways. Even as video games became more complex and publishers began to employ dedicated quality assessment teams rather than simply having the development teams playtest their own games, user feedback practices remained fixed on highlighting problems surrounding the more objective aspects of game design–coding errors and bugs–rather than exploring the experience of what it’s like to actually play the game. Methodologies like functionality testing (looking for general problems in the game’s overall design), compliance testing (checking that the game complies to publishers’ technical and legal requirements), compatibility testing (testing the game on various configurations of hardware and software), localization testing, public beta testing (which lets users pick up any errors the developers may have missed), and regression testing (testing to make sure previously reported bugs have been eliminated) have become industry standards, with publishers and developers mixing and matching various methods to suit individual project needs.

But things are changing. The growth of gaming audiences and the subsequent push towards more diverse gaming experiences has led some publishers to rethink traditional playtesting methods in favor of something a little more relevant. According to Ambinder, the games industry is starting to move towards more innovative ways of gathering data, willing to spend more time, energy, and resources on its accumulation.

“I think more and more companies are starting to see the value in hiring folks with backgrounds in psychology or related fields that provide skilled training in extracting meaningful data from playtesters,” Ambinder says. “For us, playtesting is crucial, as it is the most effective and honest means of validating our products. We would be foolish to release a game that went through minimal playtesting, as we could have little confidence in its quality had it not gone through rigorous testing. To that end, we start playtesting as soon as we have something playable, and we basically never stop as we are constantly updating our products after shipping.”

When developers first became interested in user research some 10 or so years ago, the standard practice was not to waste too much time on it; therefore, only the first hour of the game would undergo outside playtesting. Things are a little more serious these days. Most publishers would consider it madness to release a game that hasn’t been subjected to hundreds of hours of rigorous playtesting, combing over every single part of the game right up to its release. A decade’s worth of knowledge has come down to one thing: perspective.

After spending years as a playtester himself, Epic Games senior game test manager Prince Arrington saw the value of perspective in the work of those around him.

“I always got a kick out of giving feedback to developers and then seeing my suggestions come to life in-game. But sadly, it has the tendency to be one of those things that can easily be undervalued. It’s very easy for us, as developers, to get too close to our projects and fall into the trap of not realizing that our baby isn’t perfect. This often leads to poor design remaining poor. The value in having outside feedback is that it’s always nice, if you’re open to constructive criticism, to get those checks throughout the development cycle so that you can get the confirmation that you’re making something kick-ass…or otherwise, the painful realization that you’re not.”

Bungie playtested its Halo games with people who had never even seen a shooter.

Just like Ambinder, Arrington’s professional career is rooted in psychology. Earning his psychology degree from North Carolina State University, Arrington left academia to become a contract tester when his father told him that he played games so much that he should probably start making them. During his three years as Epic’s QA manager, Arrington has come to see playtesting as a valuable tool in keeping developers in check.

“Playtests are as much a part of the development cycle as design meetings or code reviews,” Arrington says. “While different in execution, it’s something that should be done daily, with different combinations of participants with varying skill sets willing to give constructive criticism, with the sole purpose of making the game as good as humanly possible.

“Developers work on a project for so long, there’s always the potential to lose perspective on what’s working and what’s not working. Without this form of genuine feedback, developers have a tendency to drink the Kool-Aid and become accustomed to the inefficiencies and flaws of a project, which leads to a failure to explore more suitable paths.”

This perspective is often offered by people who have never played video games before. Both Valve and Epic aim for a wide demographic when deciding whom to bring in, from seasoned gamers, to people who have never played a shooter before, right down to non-gamers. With the gaming audience growing every day, developers have come to understand the value of reaching out to all skill levels, fighting to keep existing audiences while trying to snare new ones.

“We have a sign-up page for playtesters on Bungie, and we bring in a whole range of people: people who are experts, people who play casually, everyone,” says John Hopson, the user research design lead at Bungie. (He too has a psychology degree). “For Halo testing we even bring in people who have never played a shooter before. It’s painful to watch because they have a lot of trouble, especially with things that we don’t even think about anymore, like coordinating the use of two thumbsticks or knowing where the buttons are. But Halo is supposed to be a game that is fun for everyone, so it’s necessary to see how people who have never played a shooter before react to it.”

Arrington says if developers lean too far in one direction they risk skewed playtesting results, ones that don’t take into account information from players who have the potential to account for a large portion of the game’s user base.

“If you fail to account for players new to your game or genre, it’s typically those players that will stop playing, or avoid your game, if the barrier to entry is too high. So this is very important for making the game easily accessible to the casual gamer as well as challenging enough for the veteran, hardcore gamer.” （source:gamespot）