Crossing the Uncanny Valley: Javier von der Pahlen

Javier von der Pahlen may be the
first to take us where no one has gone before—beyond the “uncanny valley.” Many
have entered the valley, but no one has ever made it to the other side, the place
where photorealistic facial animation stops being, well, kind of creepy.

At the 2013 Game Developer’s Conference, Javier von der Pahlen, director
of R&D for the Central Studios division of Activision Publishing, and his
colleague Jorge Jimenez of Activision Blizzard unveiled their next-generation real-time
character rendering. It was jaw-droppingly close to the other side of the valley.

The project was a collaboration between von der Pahlen’s team at
Activision and Paul Debevec’s
at the University of Southern California Institute for Creative Technologies.

Von der Pahlen, no stranger to
crossing frontiers, was born in California but grew up in a half dozen
countries in Europe and South America. He spoke to Get In Media about the
uncanny valley, his fascination with the human face, and what he looks for when
he’s hiring for his cutting-edge team.

Get In Media: How did you first get involved
in animation?

Javier von der Pahlen: It began with computer graphics,
which I became interested in during the late ’80s before it was taught as a
subject. The people of my generation who got into computer graphics came either
from a background of engineering or, as in in my case, a background in architecture.
I studied at Cornell University, and they had a very strong computer science
department that spurred much of the early innovation in computer graphics.
That’s how I was exposed. We were doing architectural renders of buildings.

Initially, rendering an image for a single frame was an
overnight experience. It was not like what we’re accustomed to now, where hundreds
of milliseconds is too long for any impact unless it’s quite groundbreaking. Then
in the ’90s you saw Toy Story come
around, you saw Jurassic Park, and
all of a sudden photorealism seemed to be an achievable goal.

In the beginning of the 2000s, I became interested in faces,
so I started researching this with the intention of getting photorealistic
faces. I thought, okay, maybe it would take six months to do. And years later
I’m still doing it.

GIM: What is it about photorealistic
faces that have so captivated you?

JVDP: Think about this—when you first
open your eyes when you’re a baby, the first thing you see is the face of your
mother. Faces are so present in our lives. It’s the topography we know the
best. And we are deeply attuned to it. From an evolutionary point of view, we
have to understand faces. Even if we’re not conscious of the fact, we’re all
experts in faces. Therefore, they are extremely hard to do, because we’re
impossible to fool, because there are seven billion experts out there. I
suppose it’s a bit of a quixotic enterprise, trying to reach for the
impossible. But it’s not impossible. I think that our GDC presentation proved
that not only is it not impossible,
but we’re close to getting there in real time, which is to me extremely exciting.

GIM: The demonstration was fascinating,
but it does bring up the question of the uncanny valley.

JVDP: We want to cross the uncanny
valley, but we’re not all the way across yet. There are things that are still
missing and people are working on them. We’re missing hair. We’re missing a
certain fleshiness. We have to add the body. In a sense, the reason why a face
is such a good starting point is that a face is so complicated and so subtle
that a tenth of a millimeter difference will affect its look and make a
difference between a mean face and a loving face. It’s just absolutely weird.
It’s a very, very subtle thing. But if we can address all the problems in
faces, then all these solutions that we find are probably—not directly, but indirectly—applicable
to the rest of the environment. Because we have to deal with such a level of
subtlety and we have to deal with this in real time.

GIM: Can you tell us a little about
how your work evolved to this point?

JVDP: About five years ago, I was lead
engineer on Softimage Face Robot, which you guys had early on at Full
Sail. We tried to do this automatic face animation, and I realized how deep into the uncanny
valley we fell when we got there. So reflecting upon this and looking at the
great examples of face animation of that time—that is, Benjamin Button and the work that was being done with Gollum—the
question for me was, “Well, these guys are doing a great job. It’s not perfect,
but it’s very convincing, so how can I leapfrog them.” And the way to leapfrog
them was to go real-time.

Now, the thing about film—and I’m not putting film down in
any shape or form, it’s beautiful the work they’re doing—but in film you render
a face and a scene and everything from just one camera perspective. That allows
you a lot of control and a lot of cheating. If something doesn’t work, just
throw a little explosion in there or blur it, change a frame, move a point,
paint it. You can do that. But with games we have to be realistic from 360
degrees. The light changes, the perspective changes and it’s even more than
that. Those two things will affect a single face, but we don’t have a just single
face, we have multiple faces and multiple hours of face animation per game.

So this is an extremely complex
enterprise. The presentation at GDC was captured at ITC, compressed with
proprietary technology we developed at Activision, and rendered with our
proprietary techniques. What’s really, really hard is actually putting
everything together. So it’s important to note that I’m not alone in this.

GIM:
Can we expect to see this new photorealistic animation in a particular new
game or in a new generation of an existing game?

JVDP: All I can say
is that we are not making this research just for the love of art. So it will be
in our games sometime in the future. I cannot tell you which games we’re
working with. But I can tell you, yes, this is not just going to sit on my laptop.

GIM: When you’re
hiring a new generation of innovators, what are you looking for?

JVDP: My intention is
to hire the geniuses out there—and there are geniuses, I’m not just using this
word lightly—people who have been looking at the face as long as I have and who
are obsessed with one particular aspect of it, whether it be tracking of the
pore or rendering the pixel. There are multiple disciplines that go into this.
The team that produced this face is about ten or 15 people. And I think the
most important quality of every single one of them is their ability to judge
the art and understand the engineering and the same time. That is the unique
profile I’m searching for.

GIM:
I also get the feeling you’re looking for people who are thinking one step
ahead of what already exists.

JVDP:
Yes, you have to. The present is the past, you know. If you shoot for anything
that exists already, by the time you get there somebody will have shifted the
field. And much more so in terms of computer graphics, because there are so
many brilliant people working on it.

GIM:
What else should students keep in mind as they prepare for a career in
gaming?

JVDP:
Find something that you excel at. If it’s doing the concept art, the
palette, the design, that’s super important. If your inclination is low-level
code, we highly value our low-level engineers. You know, it’s a rare breed, the
guy or girl that can actually pull an amazing effect out of a few bytes.

So one, follow your passion, and
two, find a niche—something that is unique, that hasn’t been done, that you can
solve better and faster than anybody else. These are the people that I hire.
People that have this ability to find a niche and explore it. When you’re a
student, a lot of times you think that all these problems have been solved
before. But there really is no problem that has been solved before. Everything
is evolving.

Most of the people that I’ve worked with
through the last ten years had to convince their supervisors or the companies
that something was a worthwhile pursuit. Don’t let your teachers say don’t do
that. A lot of times people are not going to see that there’s a fissure or a
gap that needs solving or can be solved. So if you see something that you can
dive into and solve—do that.