Categories

Kinect for Xbox One Colour Depth Map using Helix Toolkit and WPF

The Helix Toolkit is one of the easiest ways I’ve found to quickly add 3D features to a WPF Application.

KinectV2 (Kinect for XBox One) is a motion capture device designed for gaming. It comes with a free SDK making it super easy to build KinectV2 apps for windows. In just a few lines of code you get access to the various streams captured by the KinectV2 sensor.

In this Article I’ll be demonstrating a simple way of showing a 3D color depth map inside a WPF application using the HelixToolkit along with the Microsoft SDK for KinectV2.

The depth map will be rendered using the X, Y, Z coordinates from the KinectV2 depth camera stream and the coloured using a texture generated from the RGB camera stream.

The image below shows the output from the program capturing a fairy that lives with me and shows up from time to time.

Dependencies

Galasoft.MVVMLight – Used to create ViewModels for our views. All of our interactions with Kinect 2 will be within this class.

HelixToolkit.WPF – Provides 3D model and rendering features. In this sample we’ll be using the HelixViewport3D and PointsVisual3D classes.

Microsoft.Kinect – The Kinect for Windows 2 SDK

In order to run the sample on windows you will need to purchase a Kinect for XBox One sensor and adapter to allow it to be used on Windows. Click the images below to pick them up from Amazon.com

Kinect for Xbox One Sensor

Kinect for Xbox One Adapter

Application Architecture

The following diagram shows the structure of the application.

Class structure for the sample

The MainWindow hosts a HelixViewport3D which will display our PointCloudModel.

The MainWindow’s ViewModel has a reference to the Kinect 2 Sensor and handles events to update our PointCloud model.

The PointCloud model is a subclass of the Helix PointsVisual3D class.

To avoid expensive operations on the UI thread, the PointCloud is updated on a separate thread via a PointCloudWorker, a custom subclass of the BackgroundWorker class.

Depth and Colour frame data is communicated between the PointCloudWorker and MainViewModel via the KinectData class.

The Viewport

The MainWindows hosts a DockPanel. The HelixViewport3D is the lastchild control so will fill the window. I’ve added a View menu bound to some of the viewport controls for convenience. The snippet below shows how easy it is to add a 3D viewport to the window.

Interacting With Kinect

Inside the MainViewModel we are interacting with KinectV2. We initialize Kinect to send us events for Depth and Colour frames. We then handle MultiSourceFrameArrived events and update our KinectScene with data from Kinect.

An important point in the code is that we are using the CopyFrameDataToIntPtr functions to acquire the Kinect frame data for processing. These versions of the procedures outperform the versions that don’t use an IntPtr since they are manipulating memory buffers directly.

Also note that we are using a utility class to map the frame data to the coordinate spaces we need for rendering the depth map.

First the Depth frame is mapped to Camera space so that we can position the points in 3D with respect to the Kinect sensors coordinate space.

Second the Depth frame is mapped to colour space which will give us the location of RGB pixels that correspond to the Depth frame positions, allowing us to colour the depth map appropriately.

At the end of the event handler, we pass the Kinect frame data to our scene to update the point cloud.

The PointCloud Model

The PointCloudModel is a simple class that exposes a ModelVisual3D. It also controls the SampleSize (how many pixels from the depth map we will use) and the point size for the 3D depth map.

The PointCloud class is created and added as a child Visual3D of the model.

Finally the Update() function passed the KinectData to the PointCloud to that it can update itself.

public class PointCloudModel
{
// for performance reasons we will not use every depth map point
public static readonly int SampleSize = 10;
// controls the size of the point cloud points
public static readonly int PointSize = 4;
PointCloud pointCloud;
// expose the point cloud externally
public ModelVisual3D Model { get; set; }
public PointCloudModel()
{
// initialize the point cloud
this.pointCloud = new PointCloud(SampleSize);
this.pointCloud.Size = PointSize;
// add the point cloud to the model
this.Model = new ModelVisual3D();
this.Model.Children.Add(this.pointCloud);
}
public void Update(KinectData data)
{
// update the point cloud using kinect data
this.pointCloud.Update(data);
}
}

The PointCloud class has two main components, the texture, which is a WriteableBitmap, and the PointCloudWorker.

public class PointCloud : PointsVisual3D
{
// The RGB texture taken from the colour stream
public WriteableBitmap texture;
// a background worker to update the point cloud as data arrives
PointCloudWorker worker;
...
}

In the constructor for the class we initialize the texture, setting the model material to be an ImageBrush that uses the texture.

We also add a ScaleTransform3D to the model. This renders the model as the KinectV2 sees it. If we don’t do this the model would be reversed on screen.

Finally we initialize the PointCloudWorker class that does the grunt work.

In the Update method, we ask the PointCloudWorker to update the PointCloud if its not busy. Depending on performance this may mean that we don’t update the cloud for every Kinect frame we recieve.

public void Update(KinectData data)
{
// if the worker is still updating, just bail
// note: this may add some update delay depending on system performance
if (this.worker.IsBusy) return;
// send the data to the worker
var workerArgs = new CloudWorkerArgs
{
Data = data
};
this.worker.RunWorkerAsync(workerArgs);
}

When the PointCloudWorker completes its job, we replace the points in the PointsVisual3D with the new points generated by the Worker, and update the texture with the latest RGB pixels. We also set the TextureCoordinates in the Mesh to those created by the worker.

private void Worker_RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e)
{
// get the result from the worker and update our model
// We will just replace the texturecoordinates and points with the results from the worker
var workerResult = e.Result as CloudWorkerResult;
this.Mesh.TextureCoordinates = new PointCollection(workerResult.TexturePoints);
this.Points = new Point3DCollection(workerResult.PointCloud);
this.texture.WritePixels(
new Int32Rect(0, 0, Kinect2Metrics.RGBFrameWidth, Kinect2Metrics.RGBFrameHeight),
workerResult.Data.ColorPixels,
Kinect2Metrics.ColorStride,
0);
}

The PointCloudWorker

This is where we do the grunt work.

In the constructor we create a set of indexes that are used to sample the depth map and update the pointcloud and texture vertices arrays.

The KinectData is passed as part of the event args to OnDoWork. We use a Parallel.For loop to update the point cloud asynchronously and efficiently.

Note that for the texture vertices we need four points (two triangles). We want each cloud point to take on the colour of the single pixel in the RGB texture, so each vertex points to the same point in the texture.

Finally, we update the point cloud X,Y and Z coordinates, taking care that of -Infinities (in fact if we include -Infinities in the point positions, the model will not render at all).

When we are done we set the resulting point cloud in the DoWorkEventArgs.Result property.

Program Output

Here is a set of images of the back of my office. The first picture is front on, the second is an isometric view. The HelixViewport3D allows you to rotate the view by right-clicking and moving the mouse.

For performance reasons in the code I have sampled every 10 pixels from the depth map. The sampling amount can be adjusted in the PointCloudModel class.

And that’s it!, we have a nice and simple 3D rendering of what Kinect can see.