Matthew Hirsch1

Sriram Sivaramakrishnan2

Suren Jayasuriya2

Albert Wang2

Alyosha Molnar2

Ramesh Raskar1

Gordon Wetzstein1

1MIT Media Lab

2Cornell University

Prototype angle sensitive pixel camera (left). The data recorded by the camera prototype can be processed to recover a high-resolution 4D light field (center). As seen in the close-ups on the right, parallax is recovered from a single camera image.

Abstract

We propose a flexible light field camera architecture that
is at the convergence of optics, sensor electronics, and applied
mathematics. Through the co-design of a sensor that comprises
tailored, Angle Sensitive Pixels and advanced reconstruction
algorithms, we show that—contrary to light field cameras
today—our system can use the same measurements captured in a
single sensor image to recover either a high-resolution 2D image, a
low-resolution 4D light field using fast, linear processing, or a
high-resolution light field using sparsity-constrained optimization.

Video

Slides

Note: Slideshare does not convert animations properly. I recommend downloading the slides from the slideshare site for the full experience!

Additional Information

Schematic of a single angle sensitive pixel. Two interleaved photodiodes capture a projection of the light field incident on the sensor (left). The angular responses of these diodes are complementary: a conventional 2D image can be synthesized by summing their measurements digitally (right).

Illustration of ASP sensor layout (left) and sampled spatio-angular frequencies (right). The pictured sensor interleaves three different types of ASPs. Together, they sample all frequencies contained in the dashed green box (right). A variety of light field reconstruction algorithms can be applied to these measurements, as described in the text.

Microscopic image of a single 6 × 4 pixel tile of the ASP sensor (left). We also show captured angular point spread functions (PSFs) of each ASP pixel type (right).

Comparison of different reconstruction techniques for the same captured data. We show reconstruction of a 2D image (bottom right), a low-resolution light field via linear reconstruction (bottom left and center), and a high-resolution light field via sparsity-constrained optimization with overcomplete dictionaries (top). Whereas linear reconstruction trades angular for spatial resolution — thereby decreasing image fidelity — nonlinear reconstructions can achieve an image quality that is comparable to a conventional, in-focus 2D image for each of 25 recovered views.

Overview of captured scenes showing mosaics of light fields reconstructed via sparsity-constrained optimization (top), a single view of these light fields (center), and corresponding 2D images (bottom). These scenes exhibit a variety of effects, including occlusion, refraction, specularity, and translucency. The resolution of each of the 25 light field views is similar to that of the conventional 2D images.