Monday, November 30, 2009

Convert, Encode And Decode Silverlight WriteableBitmap Data

In the comments of my Silverlight 4 EdgeCam Shots blog post "marcb" asked me how to convert the WriteableBitmap to a byte array to save the snapshot in a database.
I thought the answer might be also useful for others. Furthermore I will provide ready to use code for JPEG encoding and decoding of the WriteableBitmap.

JPEG encoding and decoding
If you want to store many images or transport them over a network the needed storage size could quickly become a big problem. For example an image with the size 512 x 512 needs 1 Megabyte storage space and a 1024 x 768 image even 3 MB. A solution could be image compression using JPEG encoding and decoding. To accomplish this I've used the open source FJCore JPEG library which is distributed under the MIT License and works nicely with Silverlight.

Keep in mind that the standard JPEG format doesn't support alpha values (transparency) and that the compression is lossy. So don't encode and decode images subsequently with JPEG.
It is also possible to use the built-in Silverlight class BitmapSource and its SetSource method to decode an JPEG stream.

Thanks for the suggestion Nikola. I recently added a fastpath to the Blitting function using Buffer.BlockCopy(), which is 3x faster even for copying of each scanline. Somehow I forgot to alter the Convert methods. I'll change it. Thanks again for the info.

BTW, I left a comment on your blog post about the TGA save, but you haven't replied yet.

Yes, I saw the comment just now. I really have to figure out a way to manage comments better - my site hoster's email service doesn't work as expected with the .NETblogengine and also having some spamming problems.. :) Nice job on the blitting :)

THanks for the article. Very interesting indeed. I am interested in a screen sharing application with SL and I wonder if your code can be used for this purpose. Any comment is highly appreciated. Thanks

it could work using MJPEG compression, but the biggest problem is the latency and the sync. See the latest discussion here:http://forums.silverlight.net/forums/t/145729.aspx

And you need to make an elevated trust app in order to use COM for screen capturing.Riu made a nice OOB chat application including screen capturing, without compression. Check out the blog post and the video:http://pontonetpt.com/blogs/ruimarinho/archive/2010/02/16/streaming-webcam-amp-desktop-em-silverlight.aspx

Maybe I just don't see it, but how could I use the FromByteArray() Function without knowing the ImageSize?

I would do something like this:- Open a File with the OpenFileDialog- Get the ByteArray from the Stream- Use FromByteArray to Fill the WritableBitmap- Resize it- And save it to the DB with ToByteArray()

But to use the FromByteArray Function I have to create a WritableBitmap with a Size which I don't know...???

In addition I use the FJCore Library to convert the WritableBitmap to a Jepg using your function from above before saving it into the DB so I don't have to remember the Width and Height... here is the Code...

//Remember Original FileName _dokument.ThumbnailOriginalFileName = file.Name; try { //Works only in FullTrust Mode, but we try it... (If not in full Trust Mode, we remember only the Name of the File) _dokument.ThumbnailOriginalFileName = file.FullName; } catch { }

//Convert it into a jpg with FJCore var destStream = new MemoryStream(); thumbImage.EncodeJpeg(destStream);

//Return a byte[] - to be saved in DB (so you don't have to remember with and height) destStream.Position = 0; var buffer = new byte[destStream.Length]; destStream.Read(buffer, 0, buffer.Length); return buffer;}

Can you give an example of using IValueConverter interface to implement two way binding in an Image control. I tried this for a long time but with now success. It will be a lot of help if you can do this. Thanks for this great article.

I guess you want to bind a WriteableBitmap to an Image control's Source property.

The problem is, in XAML the Source value of the image control is interpreted as URI."You can set this property in XAML, but in this case you are setting the property as a URI. The XAML behavior relies on underlying type conversion that processes the string as a URI, and calls the BitmapImage(Uri) constructor. This in turn potentially requests a stream from that URI and returns the image source object."- taken from http://msdn.microsoft.com/en-us/library/system.windows.controls.image.source(v=VS.95).aspx

So the Image control is always calling the BitmapImage(Uri) constructor internally.

I don't like it either, but it's easier done in code-behind using this snippet:

Why would creating a WriteableBitmap on the UI thread using a memory stream give a Catastrophic failure? I get an Exception E_UNEXPECTED when calling WriteableBitmap.SetSource using a new MemoryStream created from a byte[]. I'm trying to compress the sample from the VideoSink using FJCore but it throws when I set the source:

I’ve even tried creating the bitmap from a stream that used a static byte array:

byte[] testData = new byte[] { 0X42, 0X4D, ... 0XFF, 0X00 };

and it still throws. What’s weird, creating the bitmap using the static array from the button’s event handler works fine. But creating it using the handler from the Dispatcher (even though they execute on the same thread) will cause the exception.

I'd recommend to use this Dispatcher:Application.Current.RootVisual.Dispatcher

I guess this won't change much. But why do you want to use a memory stream? You get the pixels as interleaved ARGB byte array and want to fill a WriteableBitmap that uses an ARGB int as pixel.You can do something like this:

you're just converting it to an ARGB byte array which is then saved. This raw data is not a common file format for windows. Try the WriteableBitmapEx WriteTga() method instead or use the JPEG encoding I provided above.

I'm using a WriteableBitmap to create thumbnails of user selected images; the problem is the application is an upgraded Silverlight 2 app and it makes a lot of use of BitmapImages and would take too long to fully convert.

The only would be to convert the WriteableBitmap data to a supported format like JPEG and use BitmapImage.SetSource with a MemoryStream.

I would strongly recommend to change the code so it uses the base class of the WriteableBitmap, BitmapImage, the BitmapSource. See: http://msdn.microsoft.com/en-us/library/system.windows.media.imaging.bitmapsource(v=VS.95).aspx

Hi Rene, many thanks for the details; I have changed the code to use the base BitmapSource class and it allows me swap between the WriteableBitmap and BitmapImage classes when displaying local and remote images.

Reflection.Emit is not supported by the .Net Compact Framework, which is the underlying runtime on Windows Phone. You should use LoadJpeg and SaveJpeg to decode / encode a WriteableBitmap load / save as JPEG stream.http://msdn.microsoft.com/en-us/library/ff707350(v=VS.92).aspx

Good catch David. Yes, the Decode is reversed. I must admit that I always use the built-in BitmapImage class and its SetSource method for JPEG decoding.

Another thing about the IValueConverter. Don't do this subsequently with the same image. JPEG is a lossy compression and will result in image artifacts if the encode is performed too often. You should rather save the original stream if it is already encoded.

The ToByteArray method seems to be exactly what I need for my app. Unfortunately when I save the byte[] to a file, it's not creating a valid image. (Similarly if I upload the byte[] to a SharePoint document library, the image is invalid as well.)

I've validated that the WriteableBitmap is valid. The new file does consist of 153,600 bytes, which equals the height x width x 4, which I'm guessing is right. Any suggestions on what else I may be able to do to debug? I'm assuming the methods simply works "as is" and doesn't require and changes.

Hello, Rene. I am trying to create bitmap images in background. But when I am doing so - I have thread access error. Thats why I want to read my local images as stream and then to convert them to pixels array of type Int32. Can You tell me how can I do that without using BitmapImage and without using WriteableBitmapImage?Thanks.

I tried to implement the conversion of the WriteableBitmap to an ARGB byte array that you posted but I can't seem to find the Pixels property. (int[] p = bmp.Pixels;)Can you tell me what's the problem?

Sure you can, you just have to implement the BMP spec which should be rather easy. I guess there's an implementation available out there you can re-use. The WriteableBitmapEx also has a WriteTga extension method you can use. Tga is similar o bmp but supports the alpha channel.

Using EncodeJpeg method is it possible to calculate beforehand the second argument "quality" in JpegEncoder constructor? I mean if I have a max weight for final jpeg image and if while encoding it with Encode method of JpegEncoder I get the weight bigger than required I need to resize the image again and again lowing down the quality parameter. So are there ways to figure out the quality parameter?

I'm not aware there's something like this built-in. You can probably make your own heuristics, but in the end the best result is just computing the size in the background. See how pro photo editing software handles it.

About Me

René Schulte is an internationally recognized, independent expert and was honored for his community work with the Microsoft MVP Award and as Nokia Developer Champion. He blogs about many topics of Mobile, .NET, and Windows development in general and writes articles. Sometimes he can also be found at various events as a speaker like //build and the Microsoft and Nokia Windows Phone launch events and he takes part as a judge for Mobile Development Contests (Microsoft Imagine Cup, AppCampus, Core77).
René Schulte is a Windows Phone developer since day one and develops highly acclaimed apps such as Pictures Lab, Funny Faces, Face Lens, Helium Voice, Cloud Recorder and more. He combines the advanced algorithms of computer graphics, image processing and computer vision with a great user experience. René Schulte also created and maintains several popular open source libraries for WPF, Silverlight, Windows Phone and WinRT, such as WriteableBitmapEx or the Augmented Reality library SLARToolKit.
René Schulte also works for IdentityMine, an outstanding interactive design, software development, and user experience (UX) agency.