Report Comment to a ModeratorOur Moderators review all comments for abusive and offensive language, and ensure comments are from Verified Users only.Please report a comment only if you feel it requires our urgent attention.I understand, report it.Cancel

The Psychology of The Uncanny Valley

It's a fine line between realistic and eerie - do video game characters live in or around the uncanny valley?

A few weeks ago my wife and I decided to fill in one corner of our geek credentials by checking out this Doctor Who show that everyone was going on about. We chose as our entry point the episode entitled Rose, which kicked off the show's 2005 relaunch. Unfortunately, about five minutes in my wife stood up, muttered "Nope, nope, nope" and walked out of the room. The reason? This was an episode where department store mannequins came to life, which she apparently found way too creepy.

"The uncanny valley is often cited as one reason why the cartoon robot Wall-E is appealing, yet Arnold Schwarzenegger's Terminator is not"

She's not alone, and the psychology behind her unease has several direct lessons for video game developers and artists - even more so now that upcoming consoles promise to make game characters more and more realistic.
The 'uncanny valley' is an idea originally from the field of robotics, formalised in the 1970s by Japanese roboticist Masahiro Mori.

The gist is that if you plot people's comfort level with a robot you'd find that as you add more human-like characteristics, people are more likely to find it appealing - but only up to a point. The robot's likeability plummets when it gets a little too humanlike. Make the robot more and more humanlike and it eventually gets appealing again, but that dip in likeability is the eponymous 'uncanny valley' where the vocabulary people use to describe the thing starts to include words like "eerie," "weird," and "oh god it's looking right at me."

The idea has clawed its way out of robotics and into other fields. The uncanny valley is often cited, for example, as one reason why the cartoon robot Wall-E is appealing, yet Arnold Schwarzenegger's Terminator or the dead-eye CG characters in the Final Fantasy: The Spirits Within movie are not. And, of course, this all has applications in the design of characters in video games. It's often cited as one of the reasons why designers choose to go with more stylized aesthetics so that they can create appealing characters who rest comfortably to the left of the uncanny valley instead of spending tonnes of money and time in attempts to traverse it and scramble out the other side. Not everybody can pull off Nathan Drake from Uncharted, but characters like Sackboy from Little Big Planet or the robed traveller from Journey are a lot more practical and potentially more appealing.

The graph above is taken from the Wikipedia article on the uncanny valley and is based on Mori's work. The up/down axis represents how favorable our reaction is to an artificial creation, while the left/right axis represents how much it resembles a real human. At the far left are things like factory robots, and as you move along the right you get progressively more appealing creations like R2-D2 or Bender from Futurama. Add human-like qualities, though, and people start to dislike the creation at best and find it creepy at worst. Think zombies and the conductor character based on Tom Hanks in the animated movie The Polar Express.

But is the idea of the uncanny valley and the design lessons that artists take from it backed up by psychological science? Yes, turns out they are. Let's take a look at just two studies that speak to the design of character faces and body animation to see how.

First, Substantial research has shown that the mugs of human characters matter the most. We have evolved to pay special attention to the faces of other people as a way to do everything from empathise with them, communicate with them, and even look for signs of disease. So it shouldn't be surprising that faces are one of the most important things determining whether or not a video game character will live in the uncanny valley.

"When faces are more realistic, it doesn't take much tweaking to make them look creepy. When the faces are more stylized, a wider amount of facial distortion is acceptable"

One study by Karl MacDorman, Robert Green, Chin-Chang Ho, and Clinton Koch published in the journal of Computers in Human Behavior suggests this is true and provides some specific guidelines for those character creation tools we love to see in RPGs. In one of their studies, the researchers took a realistic 3D image of a human face based on an actual person. They then created eighteen versions of that face by adjusting texture photorealism (ranging from "photorealistic" to "line drawing") and level of detail (think number of polygons). Study participants were then shown the 18 faces and asked to adjust sliders for eye separation and face height until the face looked "the best."

The result? For more realistic faces with photorealistic textures and more polygons, participants pursued the "best" face by tweaking the eye separation and face height until they were pretty darn close to the actual, real face the images were based on. But for less realistic faces with lower polygons and less detailed textures, the ranges of acceptable eye separations and face heights were much larger. In a follow-up experiment the researchers did the same thing, except they asked the participants to adjust the sliders to produce "the most eerie" face instead of the best one. Again, when faces were more realistic looking, it didn't take much tweaking to make them look creepy, but when the faces were more stylized and less detailed, a wider amount of facial distortion was acceptable before things looked eerie.

But faces aren't all that matter. Other research suggests that the way a on-screen character moves also affects how uncanny we think it is. Or, more specifically, whether it moves how we expect it to. Researcher Ayse Pinar Saygin and her colleagues hypothesised that the uncanny valley effect might be because one of the things our brains have evolved to be really excellent at is making note of prediction errors - when something in our environment doesn't do what we expect it to.

Saygin and her colleagues tested this idea through a clever experiment with a functional magnetic resonance imaging (fMRI) machine and video clips of the famous-in-certain-circles "Repliee Q2" - a very realistic looking android from Japan that can easily be mistaken by a person at first glance.

The Repliee Q2, face-to-face with a human.

The researchers hooked study participants up to the fMRI so that they could examine brain activity while looking at three video clips: an android with its exterior shell stripped away so that it was obviously a robot, the realistic looking Repliee Q2 android, and the actual human woman upon whom the android's appearance was based. They found that the "action perception system" in the brain - that is, the bits attuned to perceiving movement - were much more active when looking at the realistic android. A robot moving like a robot? No problem. A human looking all human-like? Not remarkable. But the bits of grey matter really got to work when something they thought would move like a human instead moved like something non-human. This, they argue, may be the neurobiological basis for the uncanny valley effect: a prediction error in how something ought to move.

So, what does this mean for game designers and artists? One clear implication is that there's only a small margin for error when you're trying to make something look like a real human on screen. Certain facial features may only be off by a very little bit before things look weird. So if you've got a game like Tiger Woods PGA Golf or Mass Effect that lets players customize an avatar, you may want to place some narrow boundaries on the "randomise" feature unless you want players grimacing and jamming on the "cancel" button. You may also want to test drive character faces and allow testers to tweak them to make them less weird.

Similarly, movement matters if you're trying to be realistic. Some games spend a lot of money to get this right and they really stand apart from others. It's not just about having characters doing human things like shifting their weight or changing poses. It's about having them do it really smoothly and realistically. Double for facial animations.

The flipside of all this, though, is that if you don't have the budget, expertise, or time to craft super high resolution skin textures or do motion capture, going with a more stylized art direction may keep your characters out of the uncanny valley. Dishonored comes to mind as a recent example, with its exaggerated character models that can get away with more because the game's design doesn't try to make them look too real.
So artists and designers have choices. Not every game needs to shoot for super realistic, no matter what the hardware manufacturers' marketing folks claim is possible.

Games Jobs

10 Comments

A really cool example of the uncanny valley for motion is "reverse walking," which is a cheap horror movie trick. You film a person who is actually walking backwards, doing their best to pretend to be walking forwards, but then you run the footage in reverse, so that instead they look like they are just casually walking forwards across the frame, only. . . wrong. It's extremely creepy, even though it's just a normal human moving, because the way the body is moving just does not look right to the subconscious.

For me the uncanny valley with characters has always something to do with animation and never down to simple looks. Its the small details like the eyes, the head or other body parts move that makes something look creepy or not.

As the saying goes, eyes are the window to the soul. And it seems to me that even in the best character models, the eyes are always off. Getting the eyes absolutely perfect would go a long ways towards possibly eliminating the uncanny valley I think.

The interesting thing I've been seeing for years and is still not quite "right" is how CG and in-engine movies in some games "get" it right, but playable characters come off as less "realistic" and not as fluid or interesting because they can't be animated as flawlessly as they would be when not under a player's control. Stylized art works best for me when it comes to seeing how different dev teams can make those digital people "act" a lot more than ones that are supposed to look as close to human as possible.

This has been a huge issue for me in current gen games. Motion Capture is a very useful tool, however it needs to be manipulated right so we get the most out of the timing and the spacing of the characters action and in order to do this you need to understand animation values. I for one would rather animate everything, however with tight schedules, it isn't possible.
If the process of cleaning up motion capture is rushed we end up with a lot of uncanny valley characters, It shouldn't be looked upon as the cheap option, if it's given the time that required the result can be stunning. I think as an industry we you should be striving for feature film quality level, when it comes to mocap clean up. The tools can help us, but we need to get are hand dirty and adjust and favour the timing of the movement to sell the believability

Its really incredible to see how realistic some tech is - I mean it literally blows my mind how real we can make something look, but the second it moves, its gone. We're too good at reading each other and any matter of advancements can't seem to hide it. If there's one thing we're experts at, its being human, hard to fool us. So why invest so much in it? I'd rather abstraction anyday, far more freedom. Reality is a restriction.

A stylized look is what games should strive for. Games that try and mimic realism, get superseded by themselves and their competitors every year on their path to make everything ultra realistic. But it will never look realistic. It will never look like real life. Simply because of the nature that its a game and you press a stick on the controller, and the character moves. The quest for realism dates these games (what looks incredible today, will look laughable in X years)

Whereas a stylized look (eg. Darksiders, Team Fortress 2) will never look outdated, they have their own distinct charm and a timelessness almost (while their competitors battle for realism) Just keep things stylized. Also gives us a chance to create very unique looks for games.

I don't know, games will look realistic eventually. We're getting closer and closer. You do make a good point though that maybe game developers shouldn't be trying too hard to do it now, because we aren't there yet. I mean, if I were in charge of determining the art direction for a game in development right now, I would definitely aim for a cool stylistic approach, Bioshock's about as realistic as I'd aim for, rather than hard realism, because you're right that anything created today will inevitable look dated in a couple of years, while anything stylish can hold up for much longer. Even first-gen WoW still looks good today, while EQ2 looked sad from the very beginning.

But still, we'll get there eventually, a lot of the photographic tools are already there, the lighting and texture and all that. It's quite doable to make perfectly realistic static architecture and even human models (although probably not ones that work in real time even on next-gen hardware). The real gap is in the subtlety of animation, but we'll get there eventually. It will have to involve AI that works at the muscular level, causing tiny shifts to the entire face in reaction to every little event, and it'll require a lot of careful study of human behavior, so that developers will know what they're doing wrong so that they can fix it. I kind of think that it's probably not worth any specific game developer's time to take the final leaps that will be necessary, and it'll fall more to social scientists and that sort of pure academic to develop the final toolbox of detailed movement cues, which game developers will then use to make some truly realistic games.

Now if the technology existed, of course we would use it. I mean, what's not to love about being able to make movie tie-ins that look EXACTLY like the movie, down to the actors, without turning them into hollow-eyed monsters? You could even use actors that can't do the role anymore, like making a classic Bond game that uses Sean Connery, from the 60s, and yet anything you do in the game, with the UI removed, would look indistinguishable from a scene in a "long lost" 60s Bond film. How cool would that be? We aren't there yet, but we will be.

That doesn't mean that "cartoony" would be dead, of course, there will always be an artistic place for stylized work, even when photo-realism becomes effortless.

Hi Tim, lets see where the industry goes, I guess. But still, a big focus on tech (man hours, priority etc) has the danger of overtaking gameplay. I wish developers would put all their effort, time and resources into crafting raw, fun, gameplay. It's missing more and more these days.