What is the possibility/feasibility of creating a display that would output those images, allowing the viewer to focus on whatever piece of the image s/he chooses? Essentially eliminating the main issue with current 3D technology.

1 Answer
1

First of all, this is a really amazing piece of technology, in particular because it has achieved something that optical engineers have dreamed of for a long time1, and done so with underlying techniques that we have had for quite a while. Just to give you some impression of how cool this is, I'm an optical engineer and the first time I heard about this I was certain that it was either a hoax, a massive exaggerated description of something less impressive (like an unsharp filter in photoshop or something), or simply a piece of godawful technology journalism. However, once I found the doctoral dissertation of the guy who developed this technology Dr. Ren Ng), I realized how it works, and really how clever it is.

(note: I'll probably add on to this answer a couple times, because I don't have time to give a good and thorough summary right now. If you want a really thorough description, check out the thesis I linked above. It may be a little over your head if you don't have background in optics though, which is what my summary here should help with.)

Hardware

This technique depends on some very clever data processing techniques, and on the unusual type of camera (Ren Ng calls it "plenoptic" which isn't a term I've heard before) that is used to capture the data.

This camera has an array of very small lenses (a "microlenslet array") at its image plane, where a normal camera would just have the image sensor. The image sensor is positioned slightly behind this.

The microlens array alters the incoming light before it hits the sensor. If you were to look at this raw data as it is captured by the camera, it would look similar to the image you would expect from a normal camera, but it would be composed of thousands of little blobs rather than being a nice continuous image (I'm not talking about pixels, these dots are many pixels across). If you zoom in on this image, you would see that each dot is actually a very small, possibly blurry image of a portion of the scene being photographed.

On its own, this image is ugly and not really good for anything, but because we know exactly how it was altered by the lenslet array, we actually have more information about the light that entered the camera 2. With some clever data processing algorithms, we can retrieve this information and use it to determine the focus condition of each part of the scene being imaged.

Algorithms

(this section will be expand when I have more time)

In the most basic sense, the job of an imaging system is to produce an image where each point records the color and brightness of a corresponding point in the scene being photographed. A plenoptic camera also aims to determine, for each point in the image, how the light from the corresponding point in the scene was focused. This amounts to figuring out not only where that light hits your sensor (which a normal camera measures) but what path it took to get there.

The data processing done after the image is captured is able to reconstruct this information because instead of having only one piece of information about each ray of light -- which pixel on our sensor it hit -- we now have a second piece of information -- which lenslet in the array did that ray pass through.

Once we have figured out the path of each bundle of light that we captured from the scene, we can calculate the image we would have gotten from a normal camera for any focus setting over a large range.

1: There are actually other way to achieve this, but the methods developed by Ren Ng are impressive and novel in that they are reliable and don't require insane amounts of computing power/time. This is what makes his technology marketable to consumers.

2: Actually, we didn't really get extra information, we just traded a little of one type of information for a little of another type. A normal camera would be able to produce one pixel in the output image for each pixel in its sensor, but this camera will produce an output image with about one pixel for each microlens in the array. The details of the algorithm may raise or lower that ratio a little, but the basic idea about trading one type of information for the other will always hold. This is, by the way, on reason that this technology didn't happen sooner -- it wasn't until recently that we could produce high quality lenslet arrays with enough lenses to produce a good picture.

Thanks for the history. And also digging up the link to the dissertation. Really awesome stuff. My understanding of optics jumped many fold. (While still not understanding most of the paper.) And, I guess that the answer to the second part of my question is that we aren't likely to see that kind of display for quite a while.
–
Jacob EggersSep 12 '11 at 17:53