Thursday, December 17, 2009

World# - Real Time 3D Augmented Reality with Silverlight

Please read the Update at the end of this post.

In the Silverlight 4 Augmented Reality Proof Of Concept blog post I proved that it's possible to implement Augmented Reality applications with Silverlight 4 and the built-in webcam API. The proof of concept used an image showing a Silverlight logo that was attached to the tracked marker. With this blog post I'm back in this field, but this time with real 3D Augmented Reality in Silverlight!
I use the open source Augmented Reality library NyARToolkitCS again and implemented the necessary interfaces to make it work with Silverlight. The NyARToolkitCS library is a completely managed version of the well known ARToolkit. I also use the open source managed game engine Balder for the rendering of 3D models that augment the reality. Balder is mainly developed by the Silverlight MVP Einar Ingebrigtsen. He is working hard on his 3D game engine and provided quick fixes for some issues I encountered while using Balder and I also had a nice chat with him, so big shout out to Einar.

Live
A webcam and at least the Silverlight 4 runtime must be installed to run the sample. To view the application you need to install the Silverlight 4 runtime. It's available for Windows and Mac. You can also watch a video below.
If you want to try it yourself you need do download the SLAR and / or L marker, print them and hold them in front of the camera. The marker(s) should be printed non-scaled at the original size (80 x 80 mm) and centered for a small white border. Also make sure the camera is set up properly and the scene is illuminated well without hard shadows. See the SLARToolkit Markers documentation for more details.

Simply press the "Start Fun" Button, hold the printed marker in front of the camera and move it around. Use the ComboBox to select a different 3D model. The "Flip x-axis" Checkbox could be used to flip the video (the webcam output is mirror-reversed by default).
If you click the "Start Fun" Button for the first time you need to give your permission for the capturing. This application uses the default Silverlight capture device. You can specify the video and audio devices that are used by default with the Silverlight Configuration. Just press the right mouse button over the application, click "Silverlight" in the context menu and select the "Webcam / Mic" tab to set them.
The Car model was made by Psionic, the Tank model by Ken Beyer. The Teapot is from the Balder samples and I guess it was made by Einar Ingebrigtsen. The rest was made by myself.

Video
This time I used Expression Encoder's Screen Capture feature to record a short video of the demo in action. Please keep in mind that the screen recording software eats up a lot of resources while recording and that the actual frame rate is way better.

How it works
Every time the CompositionTarget.Rendering event is fired, a webcam snapshot is taken asynchronously with the CaptureSource.AsyncCaptureImage method. This snapshot is converted in the right format and passed to the NyARToolkitCS library that outputs a transformation matrix. This matrix is used to transform the 3D model. The Balder viewport has a transparent background and the scene with the 3D objects is laid over the webcam video output. Another layer is an image that uses a WriteableBitmap and the WriteableBitmapEx library to highlight the tracked marker region with a red quad.
Sounds pretty easy, but actually it was not that easy and straight forward. One of the hardest parts was the initialization and to find the right parameters.

Results
I'm quite satisfied with the results and if you consider that all the heavy computation is done by the CPU, the performance is pretty good as well. The demo runs smoothly at 50-60 fps on a dual core machine utilizing its whole power and it should also scale up on more cores.

To be continued
If you expected source code this time I have to disappoint you. The code is not at that level where it should be, but I wanted to push this demo out. I'm not sure if I find enough time this year to work on the code and I don't want that someone tries to use it in the current state. But trust me, I will provide open source functionality to make Augmented Reality in Silverlight in the near future and I normally keep my promises, so stay tuned for more happy days...

my name is Julien Marty and I am a student in my first year of higher education at the ENSIIE (Ecole Nationale Supérieure d'Informatique pour l'Industrie et l'Entreprise). First, I want to congratulate you for your project about Augmented Reality with Silverlight 4. I am very interested in Augmented Reality that is why I wanted to contact you. Actually, I have developped the same project as yours but in C# language and it was for windows application. I would like to do it with Silverlight 4 but i have not succeeded in it. I use NYARToolkitCS.

Ok.For waiting your release can you send me by email the librairie of NYARToolkitCS adapting to Balder please and if possible your source code. I want to continue my research.my email is : juliensmarties@hotmail.comthanks for your help

Awesome stuff! I'm sure you're like me and don't have a lot of "spare time", but you know what would be killer: to put your marker on your hat or glasses and alter the display based on how far away you were and how off center you were and support something like what you see on this video...http://www.macrumors.com/2009/12/17/apple-working-on-3d-hyper-reality-displays/Should be possible right?

Thanks Tim. That idea is awesome and it should be possible. I think I will look into it after I released the code for other stuff.BTW, the projects from Johnny Lee are always impressive. He is now working in the Project Natal team. :)

Maybe someday the Natal team will make some of those fabulous libraries they're developing for human movement recognition available to Silverlight. ...assuming it's just .NET code and not hardware or dependent on XNA.

@luisThanks. Silverlight is currently not supported by Windows Mobile 6.5 and this sample was made with the first beta of Silverlight 4. But there are some rumors that one of the next WinMo versions will support Silverlight.

@Julien:I would release the open source project sooner rather than later if I could, but there are some licensing issues that need to be solved first. The project is actually almost done, but there's this paperwork. I will release the project right after I've an agreement with the ARToolkit license holders.

OK. for waiting can you say me what did you do ?Are you modify the directX function by balder function in the NYARToolkit library ?Can you give me a way has follow that I can continue my researches please.Thanks,Julien

Hi! Congratz for the application!Well, I have to build sorta app like that and I don't know how to start... about the interfaces you said you had to implement for making NyARToolkitCS able to work with Silverlight, could you talk more about this??

Hello, i'm trying to redo the same thing, but i don't understand everything about matrix, transformation, projection, etc..

Is your mesh always in 0,0,0 and only the camera is moving ? Or the camera is fixed and the mesh is moving ?

I think it's the first solution, and i'm tryng to calculate camera position and orientation in space, with DetectionResults[0].Transformation and MarkerDetector.Projection, but i don't know how to take tranformation matrixs and calculate coordinates in space.

i'm very impatient to understand how you did this.(scuse me for my poor english)

Hello, you can provide at least a snippet of code that explains how to use 3D models in SLARToolkit. I'm trying, but it only works with images, text and other controls that are inside the "canvas", it must be because SLARToolkit uses the projection of elements of Silverlight to peform the transformations. When I try to use the elements of Balder, such as meshes, it can't peform the transformations.

Please, if you can help me, I would greatly appreciate it. I need it because I'm doing my thesis of my degree in computer science, and I'm running short on time.

Please read the project page in the Samples section. The important part is highlighted (bold):"The samples are part of the source code repository, but not included in the binary release."http://slartoolkit.codeplex.com/

So just download the code and you'll find all the samples, including the Balder sample, in the downloaded package.http://slartoolkit.codeplex.com/SourceControl/list/changesets#

Thanks for the wonderful work, i have downloaded the source code for silverlight 3d which with the car tank teapot and cube.. and try to add another 3d in .ase format when i debug there is no error popped up but then when the website open with the silverlight icon once it reached 100% that page goes blank white.. may i know whats the reason or any solution... hope to hear soon.. your fast feedback will be very helpful thanks alot

Please make sure you have the Exceptions enabled. You might miss something. VS Menu -> Debug -> Exceptions -> check 'Thrown' at CLR Exceptions.It's also possible your model is too big and therefore you can only see one side. Just scale it down.