3D Tracking of 3D Rigid Objects with Direct Image Alignment and Local Appearance Based Feature Matching

We demonstrate how by combining the notion of modular eigenspaces with multiple keyframes, an image-alignment based tracking algorithm can be made to accommodate large variations in 3D pose. Since tracking without automatic pose initialization is practically useless, a second main contribution of our work is to demonstrate how local appearance based feature matching can be effectively used to augment the tracking algorithm for pose initialization. The initialization algorithm kicks in automatically whenever the tracking algorithm fails because there is a large and sudden variation in object pose. Our method for 3D pose refinement during object tracking extends the work of Cobzas et al. For tracking, we represent an object model by a set of 3D planar patches that approximate the boundary surface of the object. Associated with each patch is its appearance model that consists of a collection of eigenimage sets. Each eigenimage set is obtained from a keyframe image and the images recorded from the nearby viewpoints. Our work includes an automated procedure for keyframe identification from a training video sequence. With 640x480 images, the tracking algorithm takes roughly 0.3 to 1.0 second/frame on a 3 GHz Pentium-IV machine and pose initialization takes less than 1 second on the same processor.

• Pradit Mittrapiyanuru

The videos below show, respectively, the 3D tracking results on the Board, the Curved-Box and the Single-View-Box objects by using our Direct Image Alignment Based 3D Tracking algorithm.

[MPEG1 - 4.38 MB][MPEG1 -3.6 MB][MPEG1 - 2.43 MB]

The pictures below show, respectively, the keyframes of the object models for the Board, the Curved-Box and the Single-View-Box objects.

The videos below show, respectively, the 3D tracking results on the Board, the Curved-Box and the Single-View-Box objects by solely applying the pose initialization on a frame-to-frame basis.

[MPEG1 - 4.38 MB]

[MPEG1 - 3.60 MB]

[MPEG1 - 2.43 MB]

Extension to Mutliple View Tracking: the videos below shows the result on the Box and the Can objects by using the extenstion of our Direct Image Alignment Based 3D Tracking algorithm to multiple view tracking .