Rendering with camera control using sphere orientation (elevation , azimuth, and yaw) or camera matrix

Results rendered off-screen and returned as MATLAB matrices. It is therefore suitable for console versions
of MATLAB, as well as batch processing of many views / models

Outputs depth map (inverse depth at each pixel) along with rendered views; can be used with the MATLAB 'surf' function.

Outputs an 'unproject' matrix, linking each pixel with the coordinates the 3D
surface point projected onto that pixel; it is therefore ideal for calibration /
pose estimation using a
3D model as reference.

Along with the 'calib' function, can be used to compute pose from 2D-3D correspondences, and pose-adjust models to images (see below)

3D model caching - repeated rendering of the same model automatically use cached
data and do not involve re-loading / reading the CG file

Code Download

calib.1.0.1.zip, a MATLAB wrapper for the OpenCV calibration function,
used with renderer to adjust the pose of the model rendered using renderer. (see readme for install details, FAQ and more).
This is the Linux version, in order to use on Windows, see below for the
Windows, OpenCV binaries.

OpenCV binaries for calib on Windows are also available, in opencv_lib.zip and
opencv_bin.zip. To use these, unpack to two
separate folders and add these folders to the system PATH before running MATLAB.

An unofficial
(and unsupported) OSX port is at
bitbucket.org, with special thanks to
Yoav HaCohen.

Elad Shtivi's code and Windows binaries for the real-time demo, including facial animation (see videos below) is now available from
github.

14th of Apr. 2015
Updated version for the Linux version of renderer;
now supports MATLAB 2014b and includes changes to the documentation (previous
version available below).

8th of Feb. 2015Chao Yao has generously provided
pre-compiled Windows binaries for the renderer function,
along with source files, modified for easy compilation on different Windows platforms
/ MATLAB versions. Code is available in renderer.Windows.1.0.0.zip.
Please see readme.windows.1.0.0.txt for
additional information.

1st of Dec. 2014
Face frontalization project is
now online,
including MATLAB code for synthesizing front-facing views of faces in
unconstrained images. This project utilizes the calib function as well as data
produced by the renderer function (though renderer is not required for
frontalization).

15th of Sept. 2014
- Elad Shtivi's Windows code and binaries for the real-time demo of renderer and calib, including facial animations, is now online from
github. Videos demonstrating some of its capabilities are available on Youtube:

28 Aug. 2014
- Some basic capabilities of the calib+renderer
combo, including the addition of facial animation, are demonstrated by Elad Shtivi in the following video.

13 July 2014
- Updated renderer code with some bug fixes. New READMErenderer.txt with more information,
quick start guide, FAQ and more. Example public domain wrl file now included in
the package.

17 March 2014
- Pre-complied dependent libraries file now also contains the
Coin3D library, required by OpenSceneGraph
to support some 3D formats. This addition is quite large; the previous version
of the pre-compiled libraries file can still be downloaded below, in case Coin3D
is not required.

1 March 2014
- calib version 0.0.2 containes a few fixes and missing files

Example usage

Example taken from
Hassner, ICCV'13:
Rendering a 3D face model to match the
pose of a face appearing in a query photo. Assuming that the code from www.ics.uci.edu/~xzhu/face/
is used for detecting facial features, and a textured 3D model of a face saved as
file 'ref_model.obj'.

Figure 2. Illustrating the example code above. Using a reference
3D model from the USF. DARPA Human-ID 3D Face Database. Detected features on the
query image (red points), are matched to the same detections on the reference image
(yellow points). The 'unproject' matrix refU is then used to provide each 2D point
on the reference image, its corresponding 3D point on the 3D reference model, in
the model coordinate system. From these 2D-3D correspondences, a camera matrix is
obtained and renderer used again to render a pose adjusted reference image and corresponding
depth map.

Copyright and disclaimer

Copyright 2013, Liav Assif and Tal Hassner

The SOFTWARE
("renderer" and \ or "calib") is provided "as is", without any guarantee made as
to its suitability or fitness for any particular use. It may contain bugs, so use
of this tool is at your own risk. We take no responsibility for any damage that
may unintentionally be caused through its use.
The code makes use of the OpenCV and the
OpenSceneGraph libraries. Any use
of this code must respect their respective licenses.