Multi-View 3D Reconstruction

For a human, it is usually an easy task to get an idea of the 3D structure shown in an image. Due to the loss of one dimension in the projection process, the estimation of the true 3D geometry is difficult and a so called ill-posed problem, because usually infinitely many different 3D surfaces may produce the same set of images.

3D reconstruction of barley from 25 input images.

3D Reconstruction from multiple views

The goal of multiview 3D reconstruction is to infer geometrical structure of a scene captured by a collection of images. Usually the camera position and internal parameters are assumed to be known or they can be estimated from the set of images.
By using multiple images, 3D information can be (partially) recovered by solving a pixel-wise correspondence problem. Since automatic correspondence estimation is usually ambiguous and incomplete further knowledge (prior knowledge) about the object is necessary. A typical prior is assume that the object surface is smooth.

Our research is focused on convex variational methods. The 3D reconstruction problem is formulated as an energy minimization problem.
Due to the convexity of this energy, any (local) minimizer corresponds to the global minimum of this energy.

Minimizers of this energy are found with iterative numerical optimization methods which evolve the surface gradually from the initial surface to best one with respect to energy functional.
As a further consequence of the convexity, these methods are independent of the initialization. The initial surface can be of any shape, for example a simple box.

Two example reconstructions are shown below along with some of their corresponding input images.

Spatio-Temporal 3D Reconstruction from multiple videos

Considering a dynamic scene that changes over time, 3D reconstruction can be applied to every time step independently. However, one can achieve temporally more consistent results by using the information from several time frames together, thus computing a spatio-temporal hyper-surface in 4D space.

The following video shows results of our latest publication on 4D reconstruction.