Relighting Human Locomotion with Flowed Reflectance Fields

University of Southern California Centers for Creative Technologies
National Taiwan University*
University of Southern California School of Cinematic Arts**

Abstract

We present an image-based approach for capturing the appearance of a walking or
running person so they can be rendered realistically under variable viewpoint and
illumination. In our approach, a person walks on a treadmill at a regular rate as
a turntable slowly rotates the person's direction. As this happens, the person is
filmed with a vertical array of high-speed cameras under a time-multiplexed lighting
basis, acquiring a seven-dimensional dataset of the person under variable time,
illumination, and viewing direction in approximately forty seconds. We process
this data into a flowed reflectance field using an optical flow algorithm to
correspond pixels in neighboring camera views and time samples to each other,
and we use image compression to reduce the size of this data. We then use
image-based relighting and a hardware-accelerated combination of view morphing
and light field rendering to render the subject under user-specified viewpoint
and lighting conditions. To composite the person into a scene, we use an alpha
channel derived from back lighting and a retroreflective treadmill surface and
a visual hull process to render the shadows the person would cast onto the ground.
We demonstrate realistic composites of several subjects into real and virtual
environments using our technique.

Multiple instances of a running subject are composited into an image-based
lighting environment with matched viewpoint and illumination. The subject was
captured running in a human-scale light stage apparatus under a time multiplexed
lighting basis to allow for image-based relighting. A novel combination of light
field rendering and view interpolation was used to render the subject from an
arbitrary 3D viewpoint.

About VGL
The ICT Vision & Graphics Laboratory develops new techniques for creating and displaying photorealistic computer graphics of people, objects, and environments.
We specialize in developing image-based methods for acquiring shape, reflectance, and motion from digital photography and video.
The results are computer-generated virtual models which look and behave as realistically as possible, viewable from any viewpoint and in any illumination condition.