Let There be Lightfields

One of the most common questions we get from clients and partners is “how does Uncorporeal’s holohuman technology relate to lightfields?”

“Lightfield” is a term that gets bandied about a lot lately—and don’t be surprised if you hear it even more in the near future. But because of how liberally the term is used, a lot of people don’t have a clear idea of what a lightfield actually is. This blog post sets out to demystify the concept.

Two flat images that our brain interprets as 3-dimensional

While the real world is three-dimensional, what we see in any instant is flat. Like a pair of cameras, our eyes resolve incoming light onto what is essentially a two-dimensional surface; the retina. And so our brain is constantly presented with two, slightly different, flat images of the light that has reached us from the surrounding world. You can think of a camera—or an eye—like an open bottle left out in the rain; it only catches the rain that happens to make it through the opening at the top.

In films and television, a director and cinematographer dictate the position of the cameras and so what is recorded and played back is all the light that happens to make it through the lens, at that specific point in space.

The holy grail of VR is to have a way of filming the real world—and a way of playing it back—that allows the viewer complete freedom to move and focus their eyes, turn their heads and to walk around, unconstrained. So that no matter where the viewer is standing, no matter what direction they’re looking, or where their eyes are focused, what they see is a completely accurate reproduction of the original scene. In order to do that, it’s not enough to know about the light that’s reaching a certain point in space. You need to know about all the light that’s passing through every point in the entire space that you’re recording.

In a nutshell, that’s what a lightfield is; the complete dataset of all light passing through every point in a certain space. In our rain analogy, that’s equivalent to knowing the size and direction of every single raindrop, at all times.

Now let’s think for a second about what would be involved in recording that lightfield dataset. You’d need to measure every single photon that was bouncing around, without diverting any of them. Even with my limited understanding of quantum physics, I can tell you that’s impossible!

The good news is that—for now at least—we don’t really care about light on its way from light sources to the people and objects in our scene. We only care about the light that bounces off those things of interest, that might conceivably end up in the retina of a roaming observer. And furthermore, we can use clever tricks to interpolate what we know about the light that passes through two or more separate points in space.

Uncorporeal’s 36 camera “holohuman” capture array

So, what people mean today, when they talk about lightfield recording, is actually what I refer to as sparse lightfield capture (as opposed to the purely theoretical total lightfield capture described above). A sparse lightfield capture involves placing multiple cameras around a scene or subject, such that the characteristics of the light bouncing off it in different directions can be recorded.

Which is exactly what we do when recording our holohumans (a.k.a. volumetric performance capture). You may be aware that other companies in the field, such as 8i, 4D Views and Microsoft, record performances with similar, sparse camera arrays. What sets our approach at Uncorporeal apart is that we preserve all that lightfield data, right the way through our cloud-based photogrammetry pipeline, into our proprietary U4D codec. And we use that data, in real-time, to accurately reproduce the light that would be reaching the eyes of the viewer, as they move their head and walk around in VR.