I started back in December when I first noticed that Adobe’s Flash Player seriously has some performance issues. It always was by far the slowest of all technologies.
Today, I retried some of my self-written benchmarks using the new Flash Player 10.1 RC4 and I was absolutely blown away. The new version is so fast, it’s absolutely incredible.
I am not done yet with my thesis until mid-July, so I won’t publish to much about it here but I thought it might be interesting to show just one benchmark-result here.

I don’t want to go to much into detail regarding the test implementations since these might change til July, so if you want to know more about the exact details on my benchmark, you gotta wait til I am done with my thesis. So long, think of this as a “preview” ;-) since some things might still change. The final test/benchmark will be revealed in 2 months when it’s 100% finished. I just didn’t want to post anything that’s not done yet. As already said, this is just a little teaser for the final benchmark.

Test setup
All tests were run on a Macbook Pro with an Intel Core 2 Duo at 2.53 GHz and 8 GB of RAM.
Tests, which require a plugin were running using Safari.
Additionally, I also ran some JavaScript-based tests on Firefox, Google Chrome and Safari.
The code base for all tests is basically the same, except for differences regarding Syntax, of course. This makes all test results comparable.

Results (Click to enlarge):
All values are in milliseconds [ms] => Less is better

In numbers:

Flash Player 10.1: 528 ms

JavaScript (Safari 4.0.5): 1015 ms

JavaScript (Google Chrome 5.0.375.29: 1039 ms

Silverlight 4.0.50401.0: 1300 ms

JavaFX 1.3 (JRE 1.6.0_17): 1392 ms

Firefox 3.6.3: 1449 ms

JavaFX 1.2 (JRE 1.6.0_17): 1635 ms

Flash Player 10.0: 3201 ms

Adobe, what the hell did you do???

Note: Please be aware, that I am basically testing FP 10.1 vs FP 10.0 here. This has nothing to do with the latest RC4 version of FP 10.1 since I haven’t done any tests with older release candidates yet.

Update II: According to a blog entry of Tinic Uro, an engineer at Adobe Systems, the reason why Flash Player 10.1 works so well on Mac OS is Apple’s Core Animation Framework (Thanks to Matthew for the link!)
What I really liked about this blog post is the following:

“You might have noticed that Core Animation is a Cocoa API. Yes, Flash Player 10.1 is a true Cocoa app now (with a Carbon fallback to support Firefox and Opera which are not Cocoa yet).”

Yes, it’s true, at least regarding the graphics performance.
I’m working on a JavaFX application with vectorgraphics which are built out of hundreds of shapes. With JavaFX 1.2 there was an extreme delay with mouseover-events.
With JavaFX 1.3 it runs smooth as butter.

Basically, Adobe finally got access to an Mac API from Apple that helped really speed things up. Apparently because of this Flash is now faster on a Mac than it is on Windows when it comes to graphics rendering. There has been some speed increases on Windows but it seems to be nothing this dramatic.

Adobe also worked a lot to decrease CPU and memory in order to get Flash Player working smoothly on mobile devices and the desktop version of Flash got a lot of these optimizations as well.

@Fredo: As mentioned in the post, this is just some info I wanted to share when I had a first look into the new Flash Player 10.1.
Don’t think of this as a final result, since things might still change.
The final test/benchmark will be revealed in 2 months when it’s 100% finished.
I just didn’t want to post anything that’s not done yet.
As already said, this is just a little teaser for the final benchmark.

@Matthew Hey, thanks for that link. Looks interesting.
The funny thing: I did some tests with FP 10.1 on Windows (7) as well and it seems like we also get a nice performance boost there as well, so it doesn’t seem to be a Mac-only issue.
Seems like Adobe finally did a good job on the Flash Player.
Maybe it’s to late though (due to Flash’s bad reputation).

@Matthew: That comment is completely incorrect and a blatant lie by your regular Flash fanboy.

Core Animation has been available for YEARS now and Apple has been encouraging everyone in the development program (including Adobe) to move to the new model. Unfortunately Adobe kept building on the old Carbon API with their own rasterization, just like they lagged moving to Cocoa until CS5. So your comment that “they finally got access to a Mac API” is a pure fanboy lie trying to change the issue from Adobe not implementing things fast enough by making it look like Apple was hiding an API… even though everybody else has been using it for years.

It is true that Apple has opened access to the hardware video APIs (they were not opened up until substantial work could go into making sure that Adobe’s code would not crash the system with bad GPU/decoder usage) and so when talking about video performance that is completely true. But anything to do with rendering has been available for a really long time now.

While most of your rebuttal was true, the last point you made, in which you claim that the reason the hardware video APIs were unavailable was because Apple was worried about Adobe code crashing the system, is completely untrue. Apple had not allowed anyone access to the hardware video APIs on Mac OS. This merely included Adobe, and was not directly aimed at them. Interesting that you would call somebody out as a fanboy and then go into fanboy mode yourself.

I’m not sure what I would be a fanboy of since I’m a flash developer at a startup that was aiming to make iPhone applications, maybe fill me in? I hope you don’t mean Apple as I left there quite awhile ago and use Windows 7 as my platform. I’m just sick of all these flash evangelists acting like Adobe is a knight in shining armour and spreading blatant mistruths.

Also you can believe whatever you would like about video hardware APIs, but the major reason was stability issues with Adobe Flash. The API wasn’t exposed to avoid kernel panic crashes as the Flash player was already the #1 cause of crashes on the OSX platform. The API has been reworked with a lot of extra protection to make sure that Adobe could not take down the kernel and would only kill the process.

Unfortunately the APIs were not exposed to anyone to use which has hindered video performance on the platform, but they were unable to do so knowing that Adobe would immediately use it.

Uhhh… can we debate these issue without resorting to calling people fanboys?

If you have some other information this information that Adobe was presenting is untrue, I would be interested in hearing it. Currently, it’s hard not to take Adobe’s side on these things, since Adobe openly blogs and tells their side of the story, while Apple works in secrecy. Apple fans make guesses on their reasoning, but there’s no direct word from Apple about these things and Adobe’s side seem to pan out.

Such as if you read that link I provided, Core Animation might have been available for years on a Mac, but only very recently available to plugins via Safari. Adobe’s story seems to bear out since originally, since when they posted it, this increase in speed only worked on nightly builds of WebKit and is still not available in any of the other Mac-based browsers outside of Safari. Hopefully Firefox and Chrome on a Mac will soon also pass along the Core Animation API to plugins so they can see a speed increase.

Also Mike Downey who works on the Microsoft Silverlight team, but previous worked for Adobe recently tweeted: http://twitter.com/mdowney/statuses/13081946335
“I will say that I know for a fact that Flash runs much faster on Windows because MS gives Adobe the hooks they need – Apple does not.”

As for hardware acceleration on the Mac, while I haven’t examined the API myself, shouldn’t such an API just pass the video data and return quickly the RBG data? Why should such an video API like that need extra protection? It seems that Adobe were more easily able to get this from Microsoft and work directly with the video card companies (NVIDIA, Intel, AMD) and implement this quickly meanwhile they had wait and put more pressure on Apple to get a similar API from them. Perhaps I’m wrong, but this is what it seems like to me.

There must be something wrong with your tests I’m quite sure that Silverlight is the fastest runtime of all the compared.

I’m currently preparing a Flash vs. Silverlight presentation for a conference in Copenhagen together with one of the best Flash developers in DK (I’m the Silverlight guy). We have been researching a bit on the performance differences between the two technologies both on what others have written on the web and made some simple performance tests our self. As for pure number crunching performance, for example calculating primes, we are seeing that Silverlight 4 is about 5-6 times faster than Flash 10.1 both on Windows and Mac. With graphics performance the performance is more equal but SL is still the faster of the two.
Try for example to take a look at the prime calc example: http://www.itwriting.com/primetest/index.html or the at the bubble animation test here: http://bubblemark.com/

Why not publish your code before finalizing your thesis so that others can review your benchmarks and make sure that you at getting prober results?

The benchmark which produced these test results is just 1/10 of a series of tests, which also includes a test for the calculation of prime numbers. Regarding this special case, I can agree with you that Silverlight seems to be faster than Flash 10.1 on Mac OS X (about 2-3x, according to my benchmark).

One thing, Silverlight seems to be particularly slow at is String-concatenation. Try it for yourself and build a very simple test and concatenate a lot of strings together using the (+) operator. You’ll see that Flash 10.1 is a lot faster here.
This fact has a pretty huge impact on the test results published in this post.

I know that Bubblemark benchmark you posted here but I doubt that its very representative since all it does is moving bitmaps around. Actually, I don’t understand what this thing is supposed to test.

Regarding your idea of publishing my source, the answer is clearly: No :-)
As I already said, I won’t publish anything that isn’t done yet.

My point is that core virtual runtime of Silverlight performs far better than the Flash runtime. The Silverlight runtime (the Core CLR) is almost identical to one used by .NET (same source according to MS) which is one of the fastest runtimes around. Though I don’t know any exact details I believe that there simply are some core architectural differences in the languages/byte code/memory handling, etc… caused by the origin of each runtime that makes it more or less impossible (without a major rearchitect that will break backward compatibility) for Flash to reach the performance level for SL.

Now this SL performance advantages should show up in all benchmarks that tests “raw number crunching” performance and if not then there is a good change that the Silverlight code is not written in the most optimal way.

Other benchmarks that tests specific areas of the SDKs could show different results i.e. it might be that Flash have a faster XML parser than Silverlight (never tested it so I don’t know) simply because the implementation is done in a different way and that even mitigate the underling runtime advantages of Silverlight.

Likewise graphics performance will often rely on how much GPU acceleration each platform has for the area tested – video decoding, vector graphics, shaders, etc… In the bubble benchmark example both Flash and SL uses cached bitmaps. Before SL 3, SL didn’t have support for cached bitmaps and therefore Flash did beat SL by a large margin but after SL 3 SL wins.

As for the String concat test.

In Silverlight (as .NET and Java) stings are immutable meaning that once created they cannot be modified so what happens when you use + operator is that a new string is allocated and the content of the two source strings is copied to the new string.

This make using the + operator in Silverlight (same for .NET/Java) significantly slower that using the class build for string manipulation – StringBuilder.
So here I share some code :
C# string concat using + Operator

After the 100 million concat the string will take up about 400 MB of memory so since the + operator will copy the content of the source strings it will need to copy 400 MB of memory where as the Append usually only needs to copy the 10 char being appended (the StringBuffer might however need to expand the underlying buffer).

In the examples above the StringBuilder concat is about 20000 to 30000 faster than the + operator concat. On my machine the string concat (2.6 Ghz) example took 0.5 sec to finish. I never run the + operator test with 100 million iterations because that would have taken hours.

With 100000 iterations the + operator concat completed in 3.2 min and the StringBuilder in 6 msec.
So try to change your Silverlight string concat bechmark into using the StringBuilder class and I’m sure that you will see SL being faster than Flash here as well.

If you have other areas where you see SL perform slower than Flash please send post them and I will try to see if I can optimize it.

I agree with you if you say that SL core runtime is faster than Flash, especially if it comes up to mathmatical calculations which is mostly the cause for its superior performance advantage compared to the Flash runtime.

Regarding the string-concatenation issue: In Flash/AS3 Strings are also constants, just like in C#. If SL core runtime is supposed to be faster than Flash, the concatenation test using the (+) operator should be faster, shouldn’t it?

Regarding more tests:
– Basically all various operations on strings are not very efficient in SL, for example: Searching in Strings with the []-operator.
– Printing a lot of single characters to the screen (one by one), e.g. to a TextBlock component, is also very slow.
– Most operations on arrays are also inferior to their Flash pendants (Accessing single elements in arrays, pushing, searching etc…)
– 2D Graphic performance is horrible. I am not talking about playing around with bitmaps. Try to create a lot of native graphic objects, like e.g. rectangles and move them around. It’s no fun.
– 3D performance is even worse. I used Papervision for Flash and Kit3D for Silverlight. I know, both aren’t 100% comparable since they are different frameworks but I still thought it’s worth a try because both do not use hardware acceleration. Result: SL was waay slower than Flash.

Don’t get me wrong: There are many cases where SL is far superior to Flash especially if it comes up to algorithms dealing with numeric values (e.g. MD5 hashing, Prime number generation and -factorization, JPEG compressing etc.), but in some cases it isn’t.

One thing I should mention: I am not only testing the core runtime performance. In my opinion, the efficiency of the supplied API implementation must also be considered. A fast runtime is useless if the API is shitty.

I read your article and your comments with interest. With all due respect for me it looks like you lack experience of the .NET technology. The technical background for the .NET string concatenation like Martin explained is some fundamental you have to know, if you want to make performance tests with strings.

And for the rendering in Silverlight there are also issues you should know. Well, you can easily write .NET code in Silverlight that works just fine, but it can be code that performs badly. It’s not unusual for someone which is not that experienced with a technology to write such code, but it’s important if you want to make a performance comparision like you want to do.

I am looking forward to the publication of your final results and your source code.

After reading Martin’s comments about the StringBuffer-classes, I included those into my benchmarks. The string-concatenation related tests now exist in 3 variants:
– Using the (+) operator
– Using the str.concat(str1) methods
– Using a StringBuffer

So far, it seems like the StringBuffer gives a huge boost to Silverlight, as well as JavaFX applications. On Flash and JavaScript, almost no changes could be observed. Using the (+)-operator here lets them perform almost as good as the Silverlight- and JavaFX-versions utilizing StringBuffer. It’s interesting to see how these runtimes perform very well (regarding the string-concatenation operation) even without a StringBuffer. Isn’t that actually a minus for Silverlight and JavaFX if they rely on such classes if Flash and JavaScript work well without them? :-)

Regarding your comments about my knowledge on the .NET framework and its rendering engine, you’re partially right. I’m not THE Silverlight-guru claiming to know everything about this technology. What I basically did was, building a reference design in Flash and then port it to other runtimes. If you have any suggestions about things I should know, which are absolutely vital for this issue, you’re welcome to post here :-)

maybe it’s actual a minus for Silverlight and JavaFX, because someone who is new to these frameworks hardly knows such issues.

Programming languages like AS3 and JavaScript may be easier to handle, but on the other hand you lost a bit of control over the things, that are happening in the background. For example the StringBuilder in Silverlight gives you the possibility to define the char capcity and therefore influence the memory usage of your string. Does AS3 gives you such a influence to their strings? I don’t know really, but I doubt AS3 will do.

So there are advantages and disadvantages in my opionion.

Regarding the Silverlight rendering engine I am also not much experienced. I made a few fun projects with graphics and animations in Silverlight. I was taken to a couple of performance problems while I was developing. First I would suggest to you to take a look to the Silverlight performance tips from Microsoft:

Furthermore I can gave you some tips based on my experience with Silverlight:

* Use always canvas controls as LayoutRoot and for your parent controls that contains your elements you want to animate/render.
* Avoid animating properties like width or height of controls. Instead use ScaleTransform.
* Avoid animation elements/controls via Canvas.SetLeft or Canvas.SetTop. Use TranslateTransform.
* Avoid to render elements yourself via CompositionTarget.Rendering. Instead let Silverlight do the rendering.
* Avoid performance consuming calculations in the UI thread. This may reduce your framerate.
* PixelShader effects like the blur effect will need much performance, because they are rendered via software.
* Use Hardware-Acceleration to boost the performance via caching. But keep in mind that in some special cases the caching might decrease the framerate, so do not cache elements blindly. An example for this might be particle effects. Use EnableCacheVisualization to dertimine whether elements are cached or not.
* Use EnableRedrawRegions to see which elements are rendered all the time, although it’s not needed. For example try to avoid changing transformations parameters of elements, although no or minor changes of the values are made.

These are a few things I ecountered. There are propably more things to know, but for this someone has to go deeper into the matter :-) But I hope the things above will help you a bit.

Unfortunately, I am almost done with my thesis and thus cannot do much changes any more.

Regarding the additional options, the StringBuffer classes offer, you’re right. AS3 does not offer this (as far as I remember). There is no StringBuffer class anyway – You have to build it on your own using Arrays or Vectors.

I cannot back up your claim that languages like AS3 or JS are easier to handle. I personally, think that especially JavaScript is a true nightmare regarding syntax and debugging issues compared to compiler-aided languages like C# or Java.

Thanks for your tips regarding the rendering engine but I fear that most of these (blur effects, resize transformations etc.) do not apply for my benchmark. My test simply creates a shitload of small rectangles on a canvas and moves them around. Not more and not less.
Further: I want to avoid any caching-techniques because otherwise the results between the different RIA runtimes are not 100% comparable (imho).

I have a hard time seeing how it’s invalid to use the String class directly for the Silverlight examples. Actionscript generates new String objects when you use the + operator, and so that seems like the closest way to match Silverlight up with Flash for that sort of test.

If Martin wants to use the “best” technique for each platform as a means of benchmarking, rather than using apples to apples, then I would suggest he should write a copyPixels() example for his Bubblemark test that creates balls which extend BitmapData.

Because the result of the string concatenation with the plus operator is identical for Flash and Silverlight does not mean that they are technically the most performant ways of concatenating strings. So you will make performance tests cause of syntactical equality. That’s completly weird.

It’s like making a race between an old beetle and an Ferrari and say to the Ferrari driver he is only allowed to drive in the first gear. And at the end you are wondering that the beetle has won.

Restricting the Ferarri driver like this is what you do if you use the plus operator for string concatenation in Silverlight. You will get the worst results for performance. If someone want to make serious performance comparisons he has to use the most performant techniques for each category and not that one that are syntactical the most equal.

I can’t agree with that. It’s absolutely ok to compare your Ferrari and Beetle running only in the first gear. So that means that you want to objectively examine which of both performs better in this case. If the Beetle is better, then that’s the results of the test. Period.

I have the feeling that we share different ideas of types of performance tests. It depends on what you plan to measure. Is it the “overall” performance or specific parts of the whole picture?

Regarding this string-concatenation issue: I think I already mentioned multiple times that there is no StringBuffer class in Flash. Thus, I basically did two tests:

1) Comparison between Flash and Silverlight using the (+)-operator

2) Comparison between Flash and Silverlight using the StringBuffer class and a self-written implementation in AS3 for Flash.