sabato 1 dicembre 2012

Saltarelle vs JSIL -- Raytracer Demo

Differently from Saltarelle, JSIL operates at intermediate language (IL) level. After the C# program is compiled by Visual Studio, JSIL examines the MSIL instruction codes contained in the dll files and translates them into javascript.

This allows a greater compatibility and portability of .NET applications, but probably introduces an unnecessary layer of abstraction between the original C# code and the javascript output but the additional code needed to simulate all the .NET library has a negative impact on the execution of the compiled code.

This is shown by the following test.

I've converted the Raytracer Demo featured on the JSIL website to the Saltarelle compiler, with the aim to compare the execution speed of the two versions.

Adapting the code to Saltarelle was easy but not straightforward. Saltarelle was missing some needed System classes that I had to re-implement, namely Bitmap, Color, StopWatch, Console and Random. I tried to keep them as similar as possible to the JSIL version so to not affect the result of the test. I also had to write the import code for mersenne.js, a small library the demo uses for generating random numbers (probably because javascript's Math.random() lacks the random seed feature).

Result of the test

I ran the test on a four core Athlon AMD processor with Windows 7 and Chrome browser. The result is summarized in the table:

Saltarelle

JSIL

Script code size (.js)

108 KB

15 MB

Average time/pixel

0.006 ms

0.093 ms

So, on average, Saltarelle is 15 times faster than JSIL. Impressive!

The output script code is also very small-sized with no loading time, while the JSIL version has to download 15 MB of stuff before it can start.

3 commenti:

Cool to see someone do this comparison - I didn't realize Saltarelle was able to compile unmodified application code in this fashion (or did you have to modify the actual code? It looks equivalent to me).

A couple complaints though;

first of all while your measurements for execution time seem more or less right, your conclusions about performance are totally wrong. I don't think you spent any time looking in the profiler or even reading the output.

There's no 'unnecessary layer of abstraction' anywhere in the JSIL output that results in any performance degradation; you can easily confirm this by looking at the generated JS from Saltarelle and JSIL for the functions that show up as hotspots in the profiles, they are equivalent. For example:

Obviously there are differences in variable names, Saltarelle includes comments, and a couple minor semantic differences (JSIL produces slightly more opaque, but faster code for the foreach construct) but there is no unnecessary abstraction in either function. Both compilers are preserving meaning and not adding any unnecessary overhead to this operation.

The reason the JSIL version is slower is simply due to modern JS runtimes still being unable to effectively inline simple functions and generate efficient in-memory data structures in a variety of cases. Saltarelle's output is much easier to inline since it does not actually implement the full semantics of .NET. The *actual code* generated by both compilers to do things like loops and computations is effectively equivalent, and you are measuring the performance hit(s) caused by JavaScript runtimes' failure to inline it in all cases.

It's definitely true that the JSIL output is pretty big, but more or less for the same reason: You compiled a real .NET application, so you got all the junk that comes along with it: .NET types, etc. The fact that you had to implement Console and Random yourself kind of goes to illustrate that this is probably the wrong way to compare these two tools: Saltarelle is perfect if your goal is to build something with a C#-like language, but the wrong choice if you want to port an existing application. Likewise, I would not recommend JSIL to someone who's aiming to build an app from scratch for the web - the burden of real .NET would just get in your way.

Yes the code is almost unmodified, apart from missing types I had to change the pixel writer routine to make it work with Canvas directly.

I agree, now that I reread it, the "unnecessary layer of abstraction" is misleading, it leads to think that there is a sort of IL emulator which is not the case.

I started to think of JS runtime issue too, recently there was a new Saltarelle release which changed how .NET methods are simulated and that lead to a significant drop in performance in the Raytracer, not completely justified by the change introduced. (For curiosity, it's because .NET methods are no longer in the prototype, eg mystring.Length() is now compiled as something like ss.StringLength(mystr);)

BTW, your work with JSIL is amazing, when I have to impress people with C# running in the browser I usually show them "Escape Goat"!

It's interesting to hear that such a simple change on Saltarelle's end caused a performance hit; I'd encourage you (if not Saltarelle's developer) to file bugs with the developers of V8 and SpiderMonkey in scenarios where it seems like a change shouldn't cause a performance hit. At the very least, sometimes they can provide guidance on how to coax their runtimes into doing the right thing; in other cases it can result in fixes.

Thanks again for taking the time to do this test and write it up. I appreciate it :)