This video shows a personal retargeting experiment of facial motion capture.
The animation is pure motion capture with no keyframe animation added.

The main goal of this project was to experiment with an idea I had for facial motion capture retargeting.
For many reasons, I chose Gollum, the famous character from “Lord of the rings”.
I was looking for a humanoid character with proportions dramatically different from my own face
in order to evaluate the efficiency of my retargeting method.
Using the famous “schizophrenic” sequence in “The Two Towers” gave me the perfect wide range of facial
expressions I was looking for.
Finally, I’m a great admirer of Peter Jackson, Weta and Andy Serkis’s work on “Lord of the Rings”.
I’d like to apologise for my poor acting! I tried hard but I’ve never been able to mimic Andy’s unique deep sharp timbre.

Motion Capture:

My first challenge on this project: I had to improvise a small
mocap studio in my cellar. I bought four cheap Samsung HMX-H300s that
I hacked in order to start recording with a single switch.
These are the cheapest cameras I could find at the time with manual control
over the exposure and focus. It is important to keep a fast shutter speed
to avoid motion blur and a closed aperture for a decent depth of field.
Of course, these kind of exposure settings need a fair amount of light so
I placed two Kino Flo lighting systems one on either side of the camera array.

As you'd expect, this kind of low end consumer camcorder is just OK but not
great from a research point of view.
The main problem was the fact that I couldn’t synchronise them with a proper
black burst reference. This resulted in a slight delay between the cameras
(a maximum of half a frame at 50 fps) diminished the animation accuracy on fast
movements.

The second problem was the rolling shutter. It takes a slight delay
for the camera to store the data from the sensor line by line.
Again, this resulted in a deformation of the image and obviously reduced the
mocap precision on fast movements.

I was a little worried about the synchronisation so I decided to build a precise (1000th/s) stopwatch in order to check and align the recorded videos.
Finally, I used Movimento to generate the mocap data. Movimento is a software that was created
by Realviz, along with Image Modeller and Matchmover, prior to their acquisition by Autodesk.
I spent a lot of time tracking each of the 120 markers from 2,3 or 4 different camera views.
I could’ve used a higher frame rate than 50 fps and a higher resolution than 720p to help
on tracking some difficult markers but I didn’t want to blow my budget.

Motion Retargeting:

My first task was to write a tool in Maya to stabilise the motion capture.
The stabilisation consists of removing the head movement from the motion capture.
Considering the unwanted inertial movement of a device like a helmet with a mounted
camera and the fact that there’s not a single rigid area on a human head, stabilising facial
movement is not an easy task. In my case, I used a hair band to reduce the contact with my
moving scalp.

The retargeting process is the most difficult task in my opinion. There are a lot of different solutions
on the market nowadays but I wanted to experiment with an idea I came up with.
I was looking for a simple visual solution. I developed a way
to “sculpt” the original animation so I could match the new proportions and the flow of
the targeted face.

The raw mocap is imported to Maya and the retargeting data is processed live
after the calibration step.

Modelling & Texturing:

I believe modelling and texturing a realistic face from scratch is one of the most difficult jobs

for a 3D modeller.
This meticulous task requires extreme attention to detail and proportion.
Gollum wasn’t an easy face to model to be honest. I struggled to catch the likeness but it was very interesting to study such an amazing creature design.

For this project, I decided to model the base mesh in 3DS Max and I added the details in Mudbox.
I like modelling with layers in a non destructive workflow and I always found 3DS Max’s stacker a great tool for this reason. Most of the creative
software nowadays (music,compositing,3D sculpting,painting,animation etc) are all based around that simple concept of layers.

I did most of the texturing in Mudbox as well. I was a bit limited on the texture resolution because Mudbox requires a lot of memory on the graphic card.
Don’t get me wrong, Zbrush is a great tool too and it’s probably one of the major revolutions of this century in the 3D industry so far.
My personal preference though is to use Zbrush for concept art rather than in an animation pipeline.

Rigging:

I’ve often been asked what the best technique for facial animation is: skinning or blend shapes?

Well, neither of them are perfect really. The skinning method provides more freedom of movement while blend shapes give better control on extreme
expressions.
A combination of skinning with corrective blend shapes is the best solution if the production can afford it.
Finally, there is no doubt that a layer of procedural animation like a dedicated muscle system (dynamic,physics etc) is essential for realistic animation.
Obviously, it is very difficult to simulate the real stretching, compressions, creases, muscles and bones with only joints and blend shapes.
The prominent cheek bones on my Gollum model is a good example. If you have a close look at my video, you’ll see volume loss around the cheek bones
during the squints.
I could’ve used some extra blend shapes to minimise the problem but a collision system would’ve been more appropriate.

I used a simple rig setup for this project. A total of 120 joints drive a first layer of animation. Then I developed a corrective system consisting of simple
blend shapes and corrective vector displacement. I used Mudbox to sculpt the creases on different expressions and to generate the vector displacement
maps.
Rendering:

I started to use Arnold in 2006 at Mikros, a VFX company in Paris. The integration in Maya wasn’t great at the time but the rendering quality was
already outstanding.
Today, Arnold fulfils all my expectations of modern software : artist friendly, efficient and fast without sacrificing render quality.
Arnold is a very simple but powerful renderer and it’s very well documented on Solid Angle’s website.
I didn’t have time to do a proper compositing with separated layers so what you can see here is the rendering straight out of Arnold.