Monday, June 7, 2010

About a month ago Richard Campbell asked me if I'd like to give an interview about SLARToolkit on the .Net Rocks talk show. Of course I agreed right away. .Net Rocks! is the largest .Net developer podcast out there and Carl Franklin and Richard Campbell always have great guests. It's a real honor for me to have been on show #564.
We mainly talked about the Silverlight Augmented Reality Toolkit (SLARToolkit), but also about some of my other projects and articles.
You can download the podcast in various formats from here. I hope you like it. Also make sure you subscribe to .Net Rocks!

This blog post will show how to use the webcam, the CaptureImageAsync method and also how to implement and use the VideoSink. But most important I'll cover what the differences between the CaptureImageAsync and VideoSink are and when to use which.

Silverlight Webcam 101

The Silverlight 4 webcam API is pretty easy to use and just a few lines of code are needed to show a webcam video stream on screen.
Silverlight's CaptureSource class provides the webcam stream that is used as the source of a VideoBrush, which in turn fills a Rectangle with the video feed from the webcam. It's also possible to use any other Shape with a Fill property.
The CaptureDeviceConfiguration class can be used to retrieve a list of all installed video and audio devices on the system. Most of the time it's sufficient to use the GetDefaultVideoCaptureDevice to get the default device. The user can specify the default video and audio devices with the Silverlight configuration; he or she only has to press the right mouse button over the Silverlight application, click "Silverlight" in the context menu and select the "Webcam / Mic" tab to set them.

Silverlight Webcam / Mic configuration dialog.

The following C# code initializes the webcam (captureSource) in the Loaded event of the page and fills a rectangle (Viewport) with a VideoBrush:

This will open a webcam permission dialog asking the user for the device access. This setting is consent and Silverlight can remember if the user has previously allowed that certain application.

Webcam permission dialog

Capturing The Webcam
There are two different approaches to capture the webcam in Silverlight. The CaptureSource's CaptureImageAsync method and CaptureImageCompleted event provide a snapshot on demand and can be considered as a pull-based technology. A custom VideoSink implementation on the other hand constantly gets the raw stream from the webcam and can be considered as a push-based approach.

Push: VideoSink Webcam Capture
The other capturing approach constantly pushes every frame from the webcam into a VideoSink. The abstract VideoSink class has four methods that have to be implemented in an own subclass in order to use it.

The basic set-up of a custom VideoSink looks like this:

// MyVideoSink is derived from Silverlight's VideoSink
public class MyVideoSink : VideoSink
{
VideoFormat vidFormat;
// Could be used to initialize a container for the webcam stream data
protected override void OnCaptureStarted() { }
// Could be used to dispose a container for the webcam stream data
// or to write a header of a video file format
protected override void OnCaptureStopped() { }
// Is called when the VideoFormat was changed
protected override void OnFormatChange(VideoFormat videoFormat)
{
this.vidFormat = videoFormat;
}
// Is called every time the webcam provides a complete frame (Push)
protected override void OnSample(long sampleTime, long frameDuration,
byte[] sampleData)
{
// Process the webcam snapshot
// sampleData contains the raw byte stream
// according to the videoFormat from OnFormatChange
Process(sampleData, this.vidFormat);
}
}

The following C# code initializes MyVideoSink with the webcam. It should be added after the captureSource initialization code above:

The VideoSink's OnCaptureStarted and OnFormatChange are raised after the captureSource.Start() method was called. The OnSample method is constantly called as long as the webcam is activated. The actual interval OnSample will be called is defined in VideoFormat.FramesPerSecond which is provided through the OnFormatChange method. The OnCaptureStopped is raised after the captureSource.Stop() method was called.

Push vs. Pull
Obviously the two approaches have different characteristics.

More information like frame number and duration (accurate sample times)

Slightly faster than CaptureImageAsync if every frame is needed

Push: Constant sampling

Please note that the CaptureImageAsync method can also be called periodically. Thereby it's possible to get a snapshot in a defined interval which might be faster than using a VideoSink that fires every 30 or even 60 frames per second (fps).

The following C# code calls CaptureImageAsync every 100 milliseconds, which means every 10 fps a snapshot is taken:

Conclusion
Both approaches are helpful for different scenarios. The pull-based CaptureImageAsync method is useful for taking single snapshots, whereas a push-based custom VideoSink can be used for capturing complete sequences and encoding the webcam stream.

GetPixel and GetPixeli methods to get the color as Color struct at a specified x, y coordinate.

FromResource method to load an image from the application's resource only by passing the relative path without the need of the full Pack URI syntax. Example: "Data/flower2.png" instead of "MyAssemblyName;component/Data/flower2.png".

Live
As usual I also wrote a new sample application that shows the Fill* methods in action. The sample starts with a real-time demo that animates the Cardinal spline's tension of the FillCurveClosed method, plus some random animated filled ellipses. The sample also contains a static page showing some of the possible filled shapes.