Things get a little more complicated when it comes to processing a stream of samples from the RealSense camera. It is quite easy to get it working. but much more difficult to get it working right. We show you how in this second article about RealSense In C#.

When you have finished with the loop you need to dispose of the processing pipeline:

sm.Dispose();

This is all fine and it works, but notice that there is a problem. The tight loop implemented as for(;;) uses the main thread even though the program is just waiting for AcquireFrame to return.

If you implement things this way using the UI thread then the UI will freeze and your users will be very unhappy. They will be able to see the results of your work but they will not be able to interact with your program at all.

The obvious solution is to take the AcquireFrame, QuerySample, process sample loop and move it onto a separate thread of its own.

This isn't difficult and it is the approach most of the sample programs take. However the API provides a set of methods that will implement the necessary threading for you - why not use it?

Event Based Streaming

Using event based streaming is very easy. You start off with the same steps.

Create an instance of SenseManager.

Enable the streams you want to use.

After this things are only slightly different.

First you have to supply a callback function that will be automatically called every time a particular state occurs. To connect the callback to the state you need to create a instance of the Handler object. This has properties designed to allow you to specify any of a number of callbacks:

OnConnect - called when the device is connecting or disconnecting

OnModuleSetProfile - called just before the module configuration is set

OnModuleProcessedFrame - called when module data is available

OnNewSample - called when a captured image is available.

If we are streaming raw data from one of the cameras we need to use OnNewSample. When you start to use the more sophisticated processing modules to get gesture or hand data then you need to use OnModuleProcessedFrame.

All you have to do is create a Handler:

PXCMSenseManager.Handler handler = new PXCMSenseManager.Handler();

and set the callback property that you want to use. In the case of our example just the OnNewSample callback which is a standard method with the signature:

pxcmStatus Callback(int mid, PXCMCapture.Sample sample)

The mid parameter is a stream identifier if multiple streams are in use - this can be mostly ignored in simple setups. The sample parameter is a Sample object containing the data from the streams that have been enabled. The Callback returns a status code usually

pxcmStatus.PXCM_STATUS_NO_ERROR

Once you have the Handler object correctly initialized you can set the processing pipeline read to run using an overloaded version of the init method:

sm.Init(handler);

This specifies the Handler object that contains the references to the callbacks that you want the pipeline to call for each state.

Finally you can set the pipeline running using the

StreamFrames(bool);

method. If the Boolean parameter is true the call blocks until the streaming is canceled. That is the callback will be repeatedly called but the thread that started the process will hang until the streaming is canceled by calling the Close function. If the Boolean parameter is false then the call returns immediately and the thread that called StreamFrames continues to process instructions and events. The callback is called repeatedly until the Close function is used.

A Simple Video Example

Now that we know the theory let's put it into practice. In turns out that there are a number of problems that arise because of the use of two threads - one for the UI and one for the data processing.

Start a new WPF project called WpfCallback and add the necessary references and DLL needed to use the Realsense camera - see Getting Started With RealSense In C# if you need detailed instructions.

Next place a button on the form and define its click event handler as:

You can see that when the button is click we get a SenseManager, create a Handler and set it's onNewSample to reference a method we have to write called OnNewSample. Then we initialize the pipeline specifying the handler and set the whole thing going with a non-blocking call to StreamFrames.

Now all we have to do is write the OnNewSample callback and this is where things get a little subtle.

Place an Image control on the form and name it image1 - this is where we will display the video data.

All we have to do is use the Sample object passed to the callback to extract the video data and then convert it into a WriteableBitmap, as described in the previous part of this series.

So, a direct approach to the problem would be to first get the image data:

If you try this you will discover that you get a runtime error when you try to set the Image source to the WriteableBitmap. The problem is that the UI components are not threadsafe and so they detect any attempt to access them from any thread other than the once that created them. In short, you can access a UI component from the UI thread but no other.