FingerTracker

FingerTracker is a Processing library that does real-time finger-tracking from depth images. It is based on work done by Murphy Stein at NYU. It uses fast marching squares to find the contour of the hand and then estimates finger endpoints by looking for inflections in the curvature of the contour. It can work with depth maps from either OpenNI or libfreenect (it expects the depth maps to be scaled to the 500-2047 range provided by the Kinect drivers).

Using FingerTracker with the Kinect

To use FingerTracker with the Kinect, you need a Processing library that lets you access the depth images from the Kinect. The two main options are Dan Shiffman’s libfreenect wrapper or SimpleOpenNI. Both libraries offer access to the depth and color images from the Kinect. In addition, SimpleOpenNI also offers access to the other advanced features available from the OpenNI Kinect middleware such as suser detection, skeleton tracking, gesture detection, etc.

Finger-tracking vs. multi-touch

FingerTracker’s technique for detecting fingers is not based on anything specific about human anatomy or any sophisticated techniques for detecting digits. Instead, it is based on the shape of a particular region of a depth image. It looks for contours at a particular depth and then selects the areas where the contours taper-down into finger-like points.

In practice, this means that FingerTracker is, in someways, similar to a multi-touch library. It simply tells you a series of interesting points where a person or object has touched a particular plane in space. It does not tell you the identity of any particular finger or the continuity of which point corresponds to which finger over time (though there are some tricks for figuring these things out).

Installation and Usage

To install Isolines, download the library and unzip it into your Processing libraries folder. Restart Processing and FingerTracker should show up under the “File > Examples” menu under “Contributed Libraries”.

Here is a complete usage example with explanation inline as comments:

// import the fingertracker library// and the SimpleOpenNI library for Kinect accessimportfingertracker.*;importSimpleOpenNI.*;// declare FignerTracker and SimpleOpenNI objectsFingerTrackerfingers;SimpleOpenNIkinect;// set a default threshold distance:// 625 corresponds to about 2-3 feet from the Kinectintthreshold=625;voidsetup(){size(640,480);// initialize your SimpleOpenNI object// and set it up to access the depth imagekinect=newSimpleOpenNI(this);kinect.enableDepth();// mirror the depth image so that it is more naturalkinect.setMirror(true);// initialize the FingerTracker object// with the height and width of the Kinect// depth imagefingers=newFingerTracker(this,640,480);// the "melt factor" smooths out the contour// making the finger tracking more robust// especially at short distances// farther away you may want a lower numberfingers.setMeltFactor(100);}voiddraw(){// get new depth data from the kinectkinect.update();// get a depth image and display itPImagedepthImage=kinect.depthImage();image(depthImage,0,0);// update the depth threshold beyond which// we'll look for fingersfingers.setThreshold(threshold);// access the "depth map" from the Kinect// this is an array of ints with the full-resolution// depth data (i.e. 500-2047 instead of 0-255)// pass that data to our FingerTrackerint[]depthMap=kinect.depthMap();fingers.update(depthMap);// iterate over all the contours found// and display each of them with a green linestroke(0,255,0);for(intk=0;k<fingers.getNumContours();k++){fingers.drawContour(k);}// iterate over all the fingers found// and draw them as a red circlenoStroke();fill(255,0,0);for(inti=0;i<fingers.getNumFingers();i++){PVectorposition=fingers.getFinger(i);ellipse(position.x-5,position.y-5,10,10);}// show the threshold on the screenfill(255,0,0);text(threshold,10,20);}// keyPressed event:// pressing the '-' key lowers the threshold by 10// pressing the '+/=' key increases it by 10 voidkeyPressed(){if(key=='-'){threshold-=10;}if(key=='='){threshold+=10;}}