Wednesday, December 30, 2009

Another great and successful year comes to an end and I hope that the next year will get even better. We all don't get younger and we change over the years, but the last Silverlight demo I made for this year allows you take some fake aged memories with your webcam. The Silverlight 4 webcam and microphone support is always fun and even more when it's combined with a pixel shader. This demo uses my Old Movie Pixel Shader to create a neat real time webcam effect that simulates scratches, noise and other effects you might have seen in old movies. I also brushed things up a bit by framing the video output with a filmstrip and using some nice new icons from eggheadcafe.

Live
The Silverlight 4 runtime must be installed to run the demo. It's available for Windows and Mac.

The Webcam capturing could be started and stopped with the camera Button. If you press it for the first time you need to give your permission for the capturing. This application uses the default Silverlight capture device. You can specify the video and audio devices that are used by default with the Silverlight Configuration. Just press the right mouse button over the application, click "Silverlight" in the context menu and select the new "Webcam / Mic" tab to set them.
Press the disk Button to take a framed snapshot and save it to a JPEG file on your harddisk.
The amount of noise can be changed using the Slider and the movie ToggleButton allows you to disable the Old Movie Shader.

How it works
It's basically the Silverlight 4 CaptureSource class and my Old Movie Pixel Shader attached to the video output. I covered the Silverlight webcam and shader usage in this blog post and the JPEG encoding in this one. The Old Movie Shader was presented here.

Final words
Thanks to my kids and especially to my wife for all her patience in 2009. I love her!
I also like to thank all the people and new friends I got to know this year and especially the Silverlight community. I hope you liked my blog posts and enjoyed the Silverlight demos I've made. I have more to come next year.
I wish you all a great new year, all the best in 2010 and peace!

Thursday, December 17, 2009

In the Silverlight 4 Augmented Reality Proof Of Concept blog post I proved that it's possible to implement Augmented Reality applications with Silverlight 4 and the built-in webcam API. The proof of concept used an image showing a Silverlight logo that was attached to the tracked marker. With this blog post I'm back in this field, but this time with real 3D Augmented Reality in Silverlight!
I use the open source Augmented Reality library NyARToolkitCS again and implemented the necessary interfaces to make it work with Silverlight. The NyARToolkitCS library is a completely managed version of the well known ARToolkit. I also use the open source managed game engine Balder for the rendering of 3D models that augment the reality. Balder is mainly developed by the Silverlight MVP Einar Ingebrigtsen. He is working hard on his 3D game engine and provided quick fixes for some issues I encountered while using Balder and I also had a nice chat with him, so big shout out to Einar.

Live
A webcam and at least the Silverlight 4 runtime must be installed to run the sample. To view the application you need to install the Silverlight 4 runtime. It's available for Windows and Mac. You can also watch a video below.
If you want to try it yourself you need do download the SLAR and / or L marker, print them and hold them in front of the camera. The marker(s) should be printed non-scaled at the original size (80 x 80 mm) and centered for a small white border. Also make sure the camera is set up properly and the scene is illuminated well without hard shadows. See the SLARToolkit Markers documentation for more details.

Simply press the "Start Fun" Button, hold the printed marker in front of the camera and move it around. Use the ComboBox to select a different 3D model. The "Flip x-axis" Checkbox could be used to flip the video (the webcam output is mirror-reversed by default).
If you click the "Start Fun" Button for the first time you need to give your permission for the capturing. This application uses the default Silverlight capture device. You can specify the video and audio devices that are used by default with the Silverlight Configuration. Just press the right mouse button over the application, click "Silverlight" in the context menu and select the "Webcam / Mic" tab to set them.
The Car model was made by Psionic, the Tank model by Ken Beyer. The Teapot is from the Balder samples and I guess it was made by Einar Ingebrigtsen. The rest was made by myself.

Video
This time I used Expression Encoder's Screen Capture feature to record a short video of the demo in action. Please keep in mind that the screen recording software eats up a lot of resources while recording and that the actual frame rate is way better.

How it works
Every time the CompositionTarget.Rendering event is fired, a webcam snapshot is taken asynchronously with the CaptureSource.AsyncCaptureImage method. This snapshot is converted in the right format and passed to the NyARToolkitCS library that outputs a transformation matrix. This matrix is used to transform the 3D model. The Balder viewport has a transparent background and the scene with the 3D objects is laid over the webcam video output. Another layer is an image that uses a WriteableBitmap and the WriteableBitmapEx library to highlight the tracked marker region with a red quad.
Sounds pretty easy, but actually it was not that easy and straight forward. One of the hardest parts was the initialization and to find the right parameters.

Results
I'm quite satisfied with the results and if you consider that all the heavy computation is done by the CPU, the performance is pretty good as well. The demo runs smoothly at 50-60 fps on a dual core machine utilizing its whole power and it should also scale up on more cores.

To be continued
If you expected source code this time I have to disappoint you. The code is not at that level where it should be, but I wanted to push this demo out. I'm not sure if I find enough time this year to work on the code and I don't want that someone tries to use it in the current state. But trust me, I will provide open source functionality to make Augmented Reality in Silverlight in the near future and I normally keep my promises, so stay tuned for more happy days...

Monday, December 7, 2009

I finally kept my promise and put the WriteableBitmap extensions up on CodePlex. This step was long overdue, but when the Silverlight 4 beta was released I couldn't resist and had to play with the new webcam API first (though I'm not finished yet).

The WriteableBitmapEx library is a collection of extension methods for Silverlight's WriteableBitmap. The WriteableBitmap class that was added in Silverlight 3, allows the direct manipulation of a bitmap and could be used to generate fast procedural images by drawing directly to a bitmap. The WriteableBitmap API is very minimalistic and there's only the raw Pixels array for such operations. The WriteableBitmapEx library tries to compensate that with extensions methods that are easy to use like built in methods. The library extends the WriteableBitmap class with elementary and fast (2D drawing) functionality, supporting common shapes like point, line, ellipse, polyline, quad, rectangle, triangle. Conversion methods and functions to combine WriteableBitmaps (Blitting) are part of it too.

Nikola Mihaylov (Nokola) made some optimizations on the DrawLine and DrawRectangle methods and posted them on his own blog or in the comments of my posts. So a release as open source project was really long overdue.

Additional work
I recreated the whole Visual Studio 2008 solutions, changed the namespaces, headers, added comments and cleaned up the code base and samples. I also grouped the extension methods into different CS files with a partial class. It is now possible to include just a few methods by using the specific source CS files directly or all extension methods through the built library assembly.

License
The WriteableBitmapEx library is released as open source under the Microsoft Public License (Ms-PL) license. The Ms-PL license allows the use of the library without affecting the license of the user, thus making it also easily usable for commercial projects. On the other hand the weak copyleft characteristic ensures that changes on the source have to be committed back and that the library or parts of it always be free and never become closed source.

The WriteableBitmapEx CodePlex project site lives here. You might also want to see the Issue Tracker for a list of features that will be added in the future or to add your own. If you have any comments, questions, suggestions or want to see your name in the Credits list, don't hesitate and write a comment, use the Issue Tracker on the CodePlex site or contact me via any other media.
Have fun and let me know it if you do some cool stuff with it.

Thursday, December 3, 2009

In this short blog post I want to present a demo I've actually made last week. Unfortunately I had no time to work on it since then. This demo is a proof of concept for doing Augmented Reality with Silverlight 4 and the built-in webcam API.
I use the open source Augmented Reality library NyARToolkitCS and implemented the necessary interfaces to make it work with Silverlight. The NyARToolkitCS library is a completely managed version of the well known ARToolkit.

Video
I've recorded a short video with my iPhone 3GS. The video was recorded at night and it's no secret that the iPhone camera is not the best, but I think it's good enough to see how the marker is tracked and the Silverlight logo moves with it.

To be continued
The first results are quite good and the demo runs smoothly at 60-70 fps on a dual core machine utilizing both cores. I will continue my work on this project and provide more details, samples and of course source code.
Stay tuned...

Monday, November 30, 2009

In the comments of my Silverlight 4 EdgeCam Shots blog post "marcb" asked me how to convert the WriteableBitmap to a byte array to save the snapshot in a database.
I thought the answer might be also useful for others. Furthermore I will provide ready to use code for JPEG encoding and decoding of the WriteableBitmap.

JPEG encoding and decoding
If you want to store many images or transport them over a network the needed storage size could quickly become a big problem. For example an image with the size 512 x 512 needs 1 Megabyte storage space and a 1024 x 768 image even 3 MB. A solution could be image compression using JPEG encoding and decoding. To accomplish this I've used the open source FJCore JPEG library which is distributed under the MIT License and works nicely with Silverlight.

Keep in mind that the standard JPEG format doesn't support alpha values (transparency) and that the compression is lossy. So don't encode and decode images subsequently with JPEG.
It is also possible to use the built-in Silverlight class BitmapSource and its SetSource method to decode an JPEG stream.

Monday, November 23, 2009

In my last blog post I have covered the new Silverlight 4 Webcam API and provided a demo that used my edge detection pixel shader to create a nice real time webcam effect. In this post I make an extended version available which can save webcam snapshots as JPEG files and I also discuss some limitations of the webcam API's built-in CaptureSource.AsyncCaptureImage snapshot method. Furthermore I will give some ideas on how to build a Silverlight 4 video chat / conference application on top of the provided JPEG capturing and encoding code.

Live
To view the application you need to install the Silverlight 4 runtime. It's available for Windows and Mac.

The Webcam capturing could be started and stopped with the "Start Capture" Button. If you press it for the first time you need to give your permission for the capturing. This application uses the default Silverlight capture device. You can specify the video and audio devices that are used by default with the Silverlight Configuration. Just press the right mouse button over the application, click "Silverlight" in the context menu and select the new "Webcam / Mic" tab to set them.
Press the "Save Snapshot" Button to take a snapshot and save it to a JPEG file on your harddisk.
The threshold of the edge detection can be changed using the Slider and the "Bypass" Checkbox allows you to disable the shader.

How it works
The base Silverlight 4 webcam usage code was covered in my last blog post.

The code is pretty obvious: A SaveFileDialog is shown and if the user enters a file name and hits OK, a stream to the file will be opened and passed to the SaveSnapshot method. There's only one think to keep in mind when using the SaveFileDialog.ShowDialog() method, it can only be called from user-initiated code like an event handler, otherwise a SecurityException is thrown.

The Rectangle's surrounding Grid "ViewportHost" is rendered into a WriteableBitmap and the WriteableBitmap's Pixels are copied into another buffer with a different format. The rendered image is then written as a JPEG encoded stream using the open source FJCore library which is distributed under the MIT License. I've found some code at Stackoverflow on how to use the library in combination with the WriteableBitmap, but I modified / shortened it.
See my post on how to Convert, Encode And Decode Silverlight WriteableBitmap Data if you are interested in more WriteableBitmap conversion and JPEG encoding code.

Why not CaptureSource.AsyncCaptureImage?
You might be wondering why I haven't used the Silverlight 4 webcam API's built-in AsyncCaptureImage snapshot method of the CaptureSource class.

The AsyncCaptureImage method grabs a frame directly from the device and therefore any effects like the edge detection pixel shader won't be visible in the rendered image.

It only works if the CaptureSource was started, so the user can't save the visible image when the capturing was stopped. See the MSDN for details.

Please keep in mind that the first beta version of Silverlight 4 was used for this blog post and that some things will be changed in subsequent releases.

Where to go from here
The code in the SaveSnapshot method could be optimized and also multithreaded. The rendering should run in it's own thread. The encoding in a separate thread too and the network transport also. This approach would utilize modern quad core CPUs. After that it might be a good starting point for a video chat / conferencing application that continuously renders JPEGs and transports them between Silverlight clients. This technique is similar to the M-JPEG video format that also uses separately compressed JPEGs.
To make this idea usable for a Silverlight video chat, only the rendering, the JPEG encoder and the transfer method need to be fast enough for real time streaming.
Please leave a comment what you think about it.

The Webcam capturing could be started and stopped with the Button in the middle. If you press it for the first time you need to give your permission for the capturing. This application uses the default Silverlight capture device. You can specify the video and audio devices that are used by default with the Silverlight Configuration. Just press the right mouse button over the application, click "Silverlight" in the context menu and select the new "Webcam / Mic" tab to set them.
The threshold of the edge detection can be changed using the Slider and the "Bypass" Checkbox allows you to disable the shader.

Video
I've recorded a short video with my iPhone 3GS that shows the demo running on my Samsung NC 10 Atom Netbook. The built-in webcam is not very good and the video was recorded at night, but I think it's good enough to see how the effect looks.

How it works
The new Silverlight 4 webcam and microphone API is pretty easy to use:

The edge detection pixel shader is applied to the "Viewport" Rectangle in the XAML. For a more intuitive behavior I've also attached a negative ScaleTransform to flip the x axis of the Rectangle. Otherwise the webcam output would be mirror-reversed by default.

Thursday, November 5, 2009

Last week I've released the second part of my WriteableBitmap extensions methods. I've added DrawLine() methods and presented a sample application that showed that the WriteableBitmap line-drawing methods are 20-30 times faster than the UIElement Line class.

A stable and fast line-drawing algorithm is the basis for most shapes like triangles, rectangles or polylines in general. For this blog post I've extended the WriteableBitmap with some specialized methods for various shapes including a fast ellipse rasterization algorithm.

Live

The application includes various scenarios where different shapes are drawn. By default a little demo is shown that I call "Breathing Flower". Basically different sized circles rotating around a center ring are generated. The animation is done using the best trigonometric functions in the world: sine and cosine.
The scenario "Static: WriteableBitmap Draw* Shapes" presents all shape extensions currently available. From left to right: Points - SetPixel(), Line - DrawLine(), Triangle - DrawTriangle(), Quad - DrawQuad(), Rectangle - DrawRectangle(), Polyline - DrawPolyline(), closed Polyline - DrawPolyline(), Ellipse - DrawEllipse(), Circle - DrawEllipseCentered().
The other two scenes randomly draw all shapes or only ellipses and allow controlling the work load by setting the number of shapes. The Silverlight frame rate counter at the upper left side shows the current FPS in the left-most column.

How it works
Most of the new extension methods use the DrawLine() function to build up a shape. Only the DrawRectangle() method implements a simplified line drawing using some for loops which is faster than calling the DrawLine() method four times. The DrawEllipse() function implements a generalized form of the Midpoint circle algorithm. I've used "A Fast Bresenham Type Algorithm For Drawing Ellipses" from this paper by John Kennedy.
The extension methods are pretty fast and if you need to draw a lot of shapes and you don't need anti-aliasing, Brushes or other advanced UIELement properties, the WriteableBitmap and the Draw*() extensions methods are the right choice. If you don't like the sharp edges, you can apply the Silverlight 3 Blur effect to the image:

The DrawPolyline() method uses an array of x- and y-coordinate pairs and the array is interpreted as (x1, y1, x2, y2, ..., xn, yn). If a closed polyline should be drawn, the first point must also be added at the end of the array.
The DrawTriangle() and DrawQuad() methods needs all shape points as x- and y-coordinates. The DrawRectangle() function plots a rectangle out of the points that represent the minimum and maximum of the shape. The DrawEllipse() method interprets the parameters the same way, but the DrawEllipseCentered() function takes the center of the ellipse and the radii as arguments.
All methods are available for the Color structure or an integer value as color.

Source code
You can download the Silverlight application's source code including the complete and documented ready-to-use WriteableBitmapExtensions file from here. Check out my Codeplex project WriteableBitmapEx for an up to date version of the extension methods.

To be continued...
For the next part of this series I'm planning to add fill extensions methods to the WriteableBitmap like FillRectangle(), FillEllipse(), etc.

Update 11-06-2009
Nokola optimized the DrawLine() function a bit and made it 15-30% faster than the standard DDA implementation. I've replaced the DrawLine() method in the extensions with Nokola's optimized version, fixed some bugs and updated the source code. The original DDA implementation is now called DrawLineDDA().
Thanks Nikola!

Update 11-11-2009
Nokola optimized the DrawRectangle() function and I've updated the implementation of it. I've also added a faster Clear() method without parameters that fills every pixel with a transparent color. This was also proposed by Nokola.
Thanks again Nikola!

Wednesday, November 4, 2009

MetalScroll is an alternative for RockScroll, a Visual Studio add-in which replaces the editor scrollbar with a graphic representation of the code. Compared to the original, this version has a number of improvements:

The widget at the top of the scrollbar which splits the editor into two panes is still usable when the add-in is active.

You must hold down ALT when double-clicking a word to highlight all its occurrences in the file

Lines containing highlighted words are marked with 5x5 color blocks on the right edge of the scrollbar, to make them easier to find (similar to breakpoints and bookmarks).

I have used RockScroll for a while, but encountered some bad Visual Studio crashs if I changed from code view to the Windows Forms Designer and as a consequence I had to uninstall it. Later I asked Scott Hanselman (@shanselman), who released RockScroll on his website, if he could contact the RockScroll author Rocky Downs and ask him for the source code, so I could fix it and might release it as open source, but Scott Hanselman never got an answer from the RockScroll author. You can imagine how pleasantly surprised I was after I had read the MetalScroll news from Johan Andersson (@repi) today.

My favorite feature is not the smart Scrollbar nor the cool splitter, it's the word occurrence highlighting, a feature that I missed since I worked with Eclipse years ago. MetalScroll works with Visual Studio 2005 and 2008 and it's free and even open source! So don't hesitate, download and install this great Visual Studio add-in now. You won't be disappointed.

Thursday, October 29, 2009

The WriteableBitmap class is a nice feature that was added in Silverlight 3. It could be used to generate fast procedural images by drawing directly to a bitmap. The WriteableBitmap API is very minimalistic and there's only the raw Pixels array for such operations. This Property stores a 32 bit integer as color value for each pixel of the WriteableBitmap.

A couple of months ago I've written a handful of SetPixel methods that made it easier to use the WriteableBitmap, but they only provided a better interface for the Pixels Property and had no real functionality. This time I've written a more algorithmic method which performs a rasterization of a line: The famous DrawLine(). As the name implies is it used to draw a line between two points in the bitmap. To say it in advance: This is the way for drawing huge amounts of lines in Silverlight. I've used the elegant C# 3.0 extension methods again to add this functionality to the existing WriteableBitmap class.

Live

The application includes various scenarios where line-drawing is performed and it is also possible to set the number of lines to control the work load. The Silverlight frame rate counter at the upper left side shows the current FPS in the left-most column. You can find some details about the other parameters in this excellent blog post by András Velvárt(@vbandi).
For comparison I've also added a randomized line drawing scenario using the UIElement Line.

How it works
I've implemented two common line-drawing algorithms: The well-known Bresenham algorithm that uses only cheap integer arithmetic and a Digital Differential Analyzer (DDA) which is based upon floating point arithmetic. I don't want to explain the algorithmic details that were already explained several times in the literature or on the web. The linked Wikipedia articles are a good starting point if you are interested.

The default DrawLine() method uses a DDA algorithm and is available for the Color structure and an integer value as line color. Furthermore it needs the x and y coordinate of the start point (x1, y1) and the end point (x2, y2) of the line.
I have also added a Clear() method that fills the whole WriteableBitmap with a Color.

Results
The WriteableBitmap line-drawing approach is more than 20-30 times faster as UIElement Line. So if you need to draw many lines and don't need anti-aliasing or other UIELement properties, the DrawLine() extensions methods are the right choice.
One interesting fact: The Bresenham algorithm that uses only simple integer operations is not the fastest line-drawing algorithm anymore. It was one of the earliest computer graphics algorithms invented in the 1960s. Since then the hardware has changed dramatically and a simple floating point DDA technique gives almost the same results on modern hardware.
Please keep in mind that these results may differ depending on the used hardware.

Source code
You can download the sample application as a Visual Studio 2008 solution including the complete and documented ready-to-use WriteableBitmapExtensions class from here. Check out my Codeplex project WriteableBitmapEx for an up to date version of the extension methods.
Have fun and let me know it if you do some cool stuff with it.

To be continued...
I'm planning to write more shape extensions for the WriteableBitmap like DrawRectangle(), DrawEllipse(), DrawPolyline(), etc. I will publish them in follow-up blog posts.

Monday, October 26, 2009

The Microsoft Live team announced in a blog post that the Silverlight Streaming service is being discontinued. The Silverlight Streaming service offered free hosting for Silverlight applications and videos.

"Microsoft Silverlight Streaming by Windows Live Beta is being discontinued and will eventually be taken down.A new Windows® Azure(TM)-based hosting and delivery service will be launched by the end of 2009, though this is not a direct replacement for Silverlight Streaming and will have costs associated with its use."

Further they write:

"Don't panic! All your current content is safe and you will receive sufficient notice for you to make an informed decision on where to host your Silverlight content and applications. However, in the interim, we would like you to be aware of the following:

Effective immediately, no new sign-ups are permitted for the Silverlight Streaming service. In addition, the Silverlight Streaming publishing plug-in for Expression Encoder will no longer be available for download.
The new Windows Azure functionality will not be a direct replacement for the Silverlight Streaming service and will be a paid subscription service."

The blog post also gives some instructions on how to retrieve the hosted Silverlight content. One can't really say that the offered (Azure) "migration process" is comfortable and frictionless. Actually, as far as I know there is no real Azure migration service at the moment, one can only access the Silverlight Streaming file system and copy the content.

What now?
In my opinion this is not a very nice move of Microsoft for bloggers like me. I have used Silverlight Streaming to provide live examples for my Silverlight posts. I mainly blog about Silverlight development and provide content and source code for free, therefore I help Microsoft to spread their technology - for free. It's a shame that a free streaming service at least for personal / educational purposes like mine isn't offered anymore. I might see things too naive and got the intention of Silverlight Streaming wrong, but from my point of view it's definitely not nice.
For now I have uploaded the samples and Videos to my public Dropbox folder and stream them from there. Dropbox uses Amazon's S3 storage system to store files.
Timothy Parez (@delegatevoid) pointed me on this alternative and it actually works fine. Thanks Timothy!
I hope this blog post from the Silverlight Streaming team does not reflect the final decisions for Microsoft's Silverlight hosting. At least I can hope...

Saturday, October 24, 2009

A couple of days ago William Moore (@codenenterp) asked me via Twitter if I know a pixel shader that makes odd pixels black. I didn't know were such a shader is available, but I knew that it should be not hard to write one in HLSL. So I fired up the Shazzam tool and wrote two shaders.

I have started with a pixel shader that kills the color in every odd row of the texture and makes every second scanline transparent. If a black rectangle is overlaid with such a shaded image, each odd scanline will be seen as black, thus resulting in the desired black scanline effect.

An alternating line effect could be realized if an image instead of a black rectangle is overlaid.

After that I've extended the pixel shader to kill every odd pixel to produce the checkerboard effect William Moore wanted.

This is how a Silverlight Button could look like if the shader is applied to it:

How it works
The pixel shaders are pretty simple and small. As always, I've commented the HLSL code, but if you have any further questions, feel free to leave a comment.
Here is the shader that kills every odd scanline:

As you can see the only difference is that the kill pixel shader uses the sum of the texture coordinate's x and y component to determine if the pixel is even or odd.
For the effect of the uppermost blog post image (mandrill), I've used the product of x and y instead of the sum.
The size of the killed pixel group could be changed with the TextureSize parameter. The following example uses an eight of the original picture size as TextureSize.

Source code
I have also written a small Silverlight application that shows an image with the pixel shader attached. You can download the Visual Studio 2008 solution including the pixel shaders from here.

Thursday, August 20, 2009

In my latest WriteableBitmap Performance post I mentioned that the Silverlight 3 pixel shaders run really fast, although they are executed on the CPU and not on the GPU and that nice real-time effects can be done with it. A Silverlight 3 pixel shader could also be applied to any UIElement, thus the MediaElement too. The Silverlight MediaElement is mainly used to play music or video and with an attached shader many cool effects could be realized. As you might know I like nice effects and due to this passion I have coded another pixel shader for Silverlight. This time I have implemented a shader that simulates scratches, noise and other effects you might have seen in old movies.

Live

The intensity of the scratches and the noise is controlled with the Sliders. The shader could be disabled with the "Bypass" checkbox and it's also possible to load a WMV clip with the "Load" Button.William Moore made some short video clips with After Effects for me and I have uploaded them to my Dropbox. Just copy a URL to the TextBox and press "Load" to try them with the shader:

The 1st effect adds some random scratches, which are moving slowly each frame.
Unfortunately the Shader Model 2 doesn't support any random function and the noise() intrinsic could not be used for real-time effects. That's a problem if you want to make pseudorandom-based shaders, but one key element of restricted shader development is pre calculation and texture lookup. The randomness, which is used in the shader, comes from a WriteableBitmap, that is filled with random color in the initialization step of the application. The shader in turn samples random values from that texture.

The 2nd effect adds a bit random noise to the image. If the texture coordinates with the frame counter would only be used, a static, moving pattern would be seen and it wouldn't look noisy at all. So actually a new random noise texture is needed every frame to achieve the desired effect. Generating a random texture with a WriteableBitmap in every frame would be too expensive. Instead of this, a little trick is used and only four random numbers are generated every frame and passed to the shader as parameters. These random seeds are then used as texture coordinates for a random lookup in the random colored texture. Therefore each pixel in every frame samples a different random value. Although it is not highly random, the values are sufficient enough to produce changing noise.

The 3rd step converts the pixel to grayscale and colors it afterwards with a desaturated Sepia tone.

The 4th and last effect computes the distance to the center and multiplies it with the color. This generates the vignette effect as a fadeout to black towards the edge. A random light flickering of an old projector is also simulated.

Update 12-29-2009
I refactored the code behind shader class and separated it completly from the calling code (MainPage.xaml.cs). Now the shader class generates a default noise texture and random coordinates itself.
The linked source code was updated.

Update 12-31-2009
See this blog post if you want to try the shader in real time with your own webcam.

Wednesday, August 5, 2009

Two weeks ago I wrote the Silverlight 3 WriteableBitmap Performance blog post and got some good response on it. One of it came from the author of Quakelight, Julien Frelat. He contacted and asked me if I had tested his PNG wrapper technique, which is used in the Silverlight port of the famous game Quake. I thought Quakelight uses the WriteableBitmap and not a custom Stream hack, so I haven’t considered to check it for the Speedtest. Actually Quakelight uses the Silverlight 3 WriteableBitmap, but the source code includes a custom Stream for Silverlight 2 too.
For better performance the Quakelight PngWrapper and BitmapData classes use a 8 bit color palette instead of full 32 bit ARGB colors. This fact makes it not directly comparable to the other competitors, which all support 32 bit ARGB colors, but for certain problems 256 colors could be sufficient. That’s why I wrote this follow-up and integrated Quakelight’s Silverlight 2 Stream implementation.
The Speedtest generates an interference image and writes each pixel to a buffer, which is then used as the source of an Image. This effect could also be realized with a pixel shader and as I was working on the Speedtest v2, I implemented the effect as a pixel shader too.
Make sure to check the Update section at the bottom of this post.

The application measures the time, which every implementation needs to draw the "Maximum Frames" and calculates the mean frames per seconds (fps). The third text column shows the relative performance compared to the WriteableBitmap.
If the tests complete very fast, you should increase the "Maximum Frames" to get better results.

How it works
The image is still 512 x 512 pixels in size and the mean frames per second are measured, but I changed the CalculateColor(int x, int y) effect a bit and removed the lookup table for the sine movement of the circles. The effect looks almost the same, but the code is much better to understand. At least I hope so.

The results differ a bit from the first article, which is caused by the other algorithm that is used here. There are other color distributions and thus results in a slightly different drawing. The Silverlight rendering has a great impact on the performance and unfortunately Silverlight doesn't ship with the .Net Stopwatch class and the only way to get suitable data, is the measuring of larger code blocks, including the drawing, with the imprecise DateTime struct.
The Quakelight BitmapData is almost as fast as the Silverlight 3 WriteableBitmap, but at the cost of the 256 color limitation. Although the pixel shader is not executed on the GPU, it still runs ultra-fast compared to the other competitors.
If you want to implement a procedural image generation technique I recommend to try a pixel shader, but keep in mind that Silverlight 3 only supports the limited Shader Model 2. I noticed that the Silverlight pixel shaders seem to use SEE and are automatically executed in parallel if they run on a multi-core processor. The framerate on a dual-core machine was twice as high as on a single-core CPU. This parallel software shader implementaion in Silverlight is actually the only right way to implement them and nothing special. Shaders are designed for parallel execution on the GPU.

I have been looking into Silverlight Graphics performance myself and was interested in the new SL3 features. So I read your blog entry for WriteableBitmap performance and was also interested in the new MediaSource managed codec abilities, which looked like another way to get pixels to the screen, but also audio as well which could be interesting for games etc.
So I took your source for the test sample and added two MediaSource tests, one using single threaded frame generation, the second using a background thread and a double buffer based on work from Pete Brown at on his Commodore 64 emulator in Silverlight.
I also reworked the UI with checkboxes to turn on/off tests as well as adding 3 new types of tests beyond the Circle interference to test performance of the rendering method vs the pixel generation. These include a random noise, simple scrolling line, and a no op. I also refactored some to make it easier to add new test types. I did not implement shader based versions of my tests, so if you run the shader test on anything but circle interference its basically a no op for now J.
I thought it might be an interesting addition to the sample as yet another way to generate dynamic bitmaps, the mediasource performs very well, although still behind WriteableBitmap.
Note for the non-double buffered version of MediaSource there is a padTime that can be tuned, it is currently set to 10ms and can be lowered to improve frame rate, but below a certain time it will get too short based on how faster your computer is and cause the media player to think it is losing frames and start skipping badly. I haven’t figured out a good way to compensate for this automatically yet, it has to do with how long the video render takes in addition to the pixel generation. The double buffer does not have this issue as it is a fixed frame rate set by frameTime.
I have attached the sample to this email, let me know what you think, and thanks for the great blog posts.

Thursday, July 30, 2009

One great addition to Silverlight 3 are pixel shaders. The Blur and the Drop Shadow shaders are bundled with Silverlight 3, but it's also possible to attach custom shaders to any UI Element. Unfortunately shaders are not executed on the GPU by Silverlight, but the Software implementation is pretty fast.
There are many cool effects, which are not already part of the WPF Pixel Shader Effects Library, one could implement. So only the sky is the limit - and the Shader Model 2.0 instruction count limit of course.

Silverlight pixel shaders could be written in HLSL and compiled with the DirectX shader compiler fxc. The produced binary file is then loaded with the Silverlight 3 PixelShader class. Quite easy huh? I must admit that I'm not a complete newbie to shader development. I even wrote some shaders with Shader Model 1.1 in the assembler shading language a few years ago, but haven't done it a while.
For Silverlight 3 I've implemented an edge detection post processing effect. It's a parametric pixel shader, which performs a common image processing technique called convolution.

Live

The initial image "Lenna" is a famous test picture for image processing algorithms. Daniel Collin (@daniel_collin) pointed me on the interesting story behind that picture of Lena Söderberg. Although Lena is a pretty lady, you should try another image too.
You can control the threshold of the edge detection with the Slider and disable the shader with the "Bypass" CheckBox. Select one of three preset convolution operators from the ComboBox: Scharr, Prewitt and the well-known Sobel operator. It's also possible to change the first column of the Gx convolution kernel to try your own operator. Actually two 3x3 convolution kernels are used by the algorithm. One for the horizontal (Gx) and another one for the vertical (Gy) direction. The Gy is just a 90° rotation of Gx and the last column of Gx is the inverse of the first column. The middle column is zero. So instead of 18 parameters (3x3x2) only 3 parameters need to be passed to the shader.

How it works
I used the Shazzam Tool for the shader development. It's a nice tool to write and test shaders for WPF. It also generates a corresponding class for the pixel shader, which is then used in XAML. Most of the controls take advantage of Silverlight's 3 great (Element) Data binding mechanism.
There's really nothing more to write about the Silverlight application, all the magic happens in the pixel shader:

The colors of the current pixel's neighbors are sampled from the image, which are then multiplied with the corresponding kernel value. The results are summed up, the threshold is applied and the new color is returned. I optimized the operation a bit more by calculating some static variables, using the squared threshold and leaving the calculation for the zero kernel column / row out.

Source code
The Visual Studio 2008 solution including the pixel shader is available from here.

Update 12-30-2009
See this blog post if you want to try the shader in real time with your own webcam.

Friday, July 24, 2009

Having a fast dynamic bitmap generation API at hand is essential for procedural image generation and a lot of computer games. Therefore many Silverlight developers were disappointed that WPFs WriteableBitmap wasn't available before Silverlight 3. Fortunately there was the BitmapImage.SetSource method, which uses a Stream as parameter for the bitmap source and could be used to fill an Image. I think Joe Stegman was the first who wrote a custom PNG Genertor Stream, which used this Stream mechanism and made it possible to generate procedural images with Silverlight 2. Other implementations followed.

Now that we have Silverlight 3 and the WriteableBitmap class, all these custom PNG Stream implementations became obsolete. There are still some developers, who complain about the performance of the WriteableBitmap. I was curious how the custom PNG Stream implementations compete with the WriteableBitmap and how big the speed difference really is. That's why I wrote a small Silverlight 3 application, which measures the frames per second of the custom PNG Stream implementations and the Silverlight 3 WriteableBitmap.

The application measures the time, which every implementation needs to draw the "Maximum Frames" and calculates the mean frames per seconds (fps). The third text column shows the relative performance compared to the WriteableBitmap.
If the tests complete very fast, you should increase the "Maximum Frames" to get right results.

How it works
The Image has the size 512 x 512. One after another every drawing method is executed and the time is measured. The method CalculateColor(int x, int y) computes the color for every pixel. For that I implemented a nice old school demoscene effect, which produces an interference image. I was inspired by the brilliant Amiga demo State of the Art from 1992.

The circles center position is animated with a sine function. For better performance I use a pre calculated lookup table (LUT) here. The rings are built using the clamped Euclidian distance to the center. Actually every Math.Sqrt() produces one colored circle, which is cut into rings by the shifting and clamping. Of course this could also be done with some loops and sine / cosine or other techniques. The square roots are calculated on the fly and not stored in a LUT. Otherwise the calculation would be too fast and not representing a real use case.
The rest of the implementation is quite simple and there's not much to explain. If you are interested in the details, please look at the source code or write a comment.

Results

The WriteableBitmap is obviously the fastest implementation. Actually I haven't expected anything else, but I hoped it would be a bit faster. Nevertheless, the Silverlight 3 WriteableBitmap is almost twice as fast as the SlDynamicBitmap library and Balder's RawPngBufferStream.
Please consider, although I use relative values, you might encounter some slightly different test results. Depending on the used hardware each implementation could perform better or worse.

Source code
The Visual Studio 2008 solution of the Speedtest application is available for download from here.

Update 08-06-2009
I've written a follow-up to this article and included the Quakelight PNG implementation and a custom pixel shader.