Test Case Scenario: Dr. Deborah Hendersen

Credit: Dr. Deborah HendersonThere’s often a deep divide
between a game designer’s goal and a player’s expectations, and that’s where
behavioral researcher Dr. Deborah Hendersen comes in. Hendersen is charged with
the not-so-simple task of figuring out if users are connecting with Microsoft’s
broad array of games and if so, how and whether they’re having a gameplay
experience that matches with the designer’s intentions. Using a mix of science,
observation, and deductive reasoning, Hendersen and the user researchers at
Microsoft help the studio find ways to help players grasp game concepts more
intuitively and work to distill down what makes a game fun…or not. Over the
past few years, Microsoft has changed the way that it researches character and
narrative development. What Dr. Hendersen has found may surprise you, and help
you build a better, more memorable game.

Get In Media: Walk me through the process of doing behavioral
research on a game. What are you looking at in terms of the developer end and
what are you looking at from the user end?

Dr. Deborah Hendersen: The first thing that happens, obviously, is a
conversation with design. We ask them, what are you trying to do with this? How
we test things depends on the questions that the designer has. Really early on,
we generally do tests that are focused around something we refer to as the core
mechanics of the game. If it’s a racing game, can you drive the car? Can you
figure out how to accelerate? Can you break? If you’re in a fighting game, can
you actually hit things with your sword? Do you understand when you’re getting
hit? These questions are not about is the game fun or is the experience fun,
but rather about just are they functional? Can they work? Do they make sense?
Do they seem to promise things that the game isn’t actually trying to promise
or imply? The first question is, really, can we figure out whether the game is
communicating with players in a way that makes sense to them and also in a way
that’s very functional for them?

The next thing we like to
test are the learning systems. When I’m testing for mechanics, I’ll do this on
a one-on-one situation where I have a player come in and I have them do a bunch
of tasks with whatever build we have in whatever state it’s in. What’s great
about that is we can get a lot about understanding because we ask people to
think aloud and we can really understand what the player thinks is going on and
identify any sort of mismatches between the way design wants it to be working
and the way it’s actually working. However, in order to get at questions around
emotional things like is a game fun? Is it exciting? Is it fast in terms of its
pace? Whatever those goals are for the designer, we need to be able to test
lots of people at once because the data we need to collect differs.

Once we get that worked out,
we start looking at what the experience looks like. We start off with a small
snippet of it, generally the initial experience because that first hour is
really important. It sets the tone for a game. Then we move into extended
experience testing where we’ll test the entire game. We’ll bring people in for
multiple days and have them play through as much of the game as they can
possibly get through, breaking at various points and asking them survey
questions. That’s typically how we work in a production cycle.

GIM: How long is that production cycle in terms of what
you do?

DH: Oftentimes we engage with a particular product for a
full two years. It depends on whether it’s something quick like an XBLA [XBox
Live Arcade] game that that has a much shorter production cycle or if it’s
something much longer like a AAA title.

GIM: In doing this research, what discoveries have surprised
you?

DH: Oftentimes you get really surprised by little things
that people interpret the wrong way or get confused. An example of this is in a
game called Toy Soldiers: Cold War. They had this mechanic where you
were supposed to get a whole bunch of kills then you would unlock something
really awesome and it would be really amazing, but the way that they built the
[game] out, it looked like the gun was overheating. Instead of killing a whole
bunch of people and getting this awesome reward, people started deliberately
avoiding that. It was because the icon that they were using was red [so] they
switched it to a blue and made it look a little bit different. There are small
moments like that that we see all the time where there’s just a really consistent
misinterpretation that if you’ve been working with a game for a really long
time, you don’t see because of course you already know what the answer is.

GIM: [Microsoft has recently made some serious improvements
in narrative testing.] What have you discovered about how users are connecting
with narrative?

DH: We’ve found a couple of consistent patterns. One is
when you talk to players about the games that they love most, character really
comes through. They have these great relationships with characters; these
moments where the narrative goals and the gameplay goals come into conflict and
they’ll pick things that are narrative driven because they have a personal
relationship with these characters. That’s really powerful and wonderful.

The other thing that we saw
is that players have a lot of trouble following plots in games. What we did is
asked players to retell the narrative, so tell me from beginning to end the
story, your favorite game’s story, whatever they cared about the most. That’s a
hard question to judge whether or not somebody’s doing it well, so we also
asked them about a story in another media and we looked to see what the
differences were between those two things. When we did that, we found that
people were really struggling to remember game narratives. There are a lot of
reasons why that might be, but for us we began to think about whether it was a
comprehension problem. That it was just hard to get all the pieces to fit
together into a coherent narrative. We began to look at it like that and see if
there was a way to help game developers think about that early in the process
when they still had time to fix things. Many of the ways we tell story in games
are very expensive to do, like cut scenes and voice-over, things like that. We
had sort of a two-part goal. One was to get this improvement in player
understanding, but the other is to get that improvement early enough that game
teams could actually fix things and respond to issues.

GIM: For new developers who may not have the resources of
Microsoft, what do you recommend they think in terms of narrative?

DH: One of the most useful things for a designer to do is
to write down the story, not from the sense of what they want from an
experience perspective, but rather how are they going to tell it? Really be
extremely crisp about that … instead of “Here’s the story that you’re
supposed to take away from it,” but rather “Here’s the description of
what’s going to be on the screen.” [Then] you can really highlight a lot
of areas where people are going to end up confused. … The real thing is about
understanding that what the user gets and what a narrative designer needs when
they’re building the story are oftentimes a little bit different. Where a
narrative designer might build out these wonderful characters with this rich
backstory, if you’re not really careful in how you present that to the user,
the user may miss out on all that information. It’s really about how you
present the information to the user so that they too can have this great narrative
experience that you’ve designed for them.

GIM: Were there any particular aspects of character that
stood out in terms of what users remembered?

DH: It was less about the character and the importance of
the character in the overall story arc, but more about the strength of the
characterization. Is everything that the character is doing making perfect
sense in the environment that they’re in? Are they interacting with the player
in a way that makes sense? It’s something where [characters] weren’t giving you
these generic lines, but they were instead giving you scenes that really felt
tailored to the experience the player was having. That tended to be when
character really, really shone and you got a sense of consistency, a sense that
I really know who this is and now I can interact as a person rather than as
this [character] construct.

GIM: Can you give me an example of one of those smaller
moments?

DH: A great example of this is we asked one participant
what one of his favorite game narratives was or a game that had a lot of great
story in it. He wanted to talk about Skyrim. We talked about the main
story arc and things like that, but when I asked him for a really strong moment
from this game, one that really just stood out as just being amazing for him and
really, really powerful, he talked about a very, very small moment. He talked
about a woman he referred to as “the scary lady,” but when we dug
into that a little bit deeper, he talked about this character who was an
ambassador who’s throwing a party that you essentially crash. You sort of talk
your way in. What I loved about this moment is she is a very minor character
but she does everything right. You walk in the door and she does exactly what
you’d expect a host to do in that situation, which is she comes up and says,
“Sorry, who are you?” and you have to introduce yourself. It’s a
moment that just simply works for this player because everything that she did,
her tone of voice, the way she approached you, the fact that he couldn’t
move… Everything fits together and you’re like, “Oh, no. I’m about to
get caught.”

What was great about this is
it made the player not just think about the moment as it was happening, but
about the moment that would follow. What is going to happen next? And he got
very predictive in this. That’s oftentimes what we see when players get really
engaged with things. One of the things that happens when they retell stories
from other media that we weren’t seeing from games is that they would burst out
with this, “Oh, my gosh, I didn’t know what was going to happen next. I
was super surprised by that. I was texting my friends. We were all trying to
figure out what was going to happen.” Those sorts of moments. And those, I
think, are really wonderful and powerful and really demonstrate when you’ve
really given someone an extraordinary experience.

GIM: For new developers, what do you recommend they think
about in terms of character?

DH: It’s about making sure that your characters are
reacting sensibly to the player. When the player makes a decision, it needs to
be something that is reflected in the people around them because otherwise it
just ends up feeling like a construct. It really is in those tiny, tiny little
details where the character acknowledges what the player has just done and
talks about it with their own vent, really getting that moment of, “Yes, I
know you just did this and I, as this particular character, have this
particular take on it.” I think that’s one of the great ways that you can
elevate the character. You can move it from being a sort of empty AI to being a
person.