Self-proclaimed “RIA Cowboy” James Ward recently updated his RIA data loading benchmarks, this time adding Lazlo into the mix as well as gzip compression on the text streams. The tests are pretty straightforward and are meant to measure performance and latency when loading rows of data via various transport mechanisms:

The goal of this application is to demonstrate various methods of loading data in RIAs and see how those methods impact performance, bandwidth, and client memory usage. These benchmarks aim to be honest and unbiased. Thus they are open source and anyone can submit patches to have new tests added or to have the existing tests improved. Yet, these tests should always fairly represent real world uses of these technologies.

I ran a test using 5000 rows and got the following results:

The results are staggering showing Adobe Flex’s AMF3 transport protocol completely outperforming both standard XHR and Lazlo with even faster results when using Flex DataServices with data paging. Give the application a run here.

Update:

I re-ran the tests based on feedback from reader comments and James Ward and here are the updated results now returning 1,000 rows. Click on the image to get the full image display:

While the BM surely yields some general information, I cannot think of a lot of applications that would need to transfer 5000 table rows in a data request. For normal data processing it is rather senseless. I find this very strange…

Like Frank said, the Flex paging isn’t really fair. When it only loads in 100 records at a time, of course it’s fast.

Also, there’s no surprise that the data bandwidth of AMF is smaller…it’s a binary format.

I’m also confused as to why the server exec time is any different between Dojo and JSON…it returns JSON in both instances. And, of course the parsing time is high for JSON because the code then iterates through the dataset to build an HTML table using string concatenation.

What this does demonstrate is that there’s plenty of things to consider when building a web application from server exec time, to bandwidth, to rendering.

I wonder how could JSON use so much bandwidth (usually smaller than XML) and why we should care about client render time (usually a couple of milliseconds to evaluate a big JSON string) … should anyone show me sources, server language and practices used for these benchs?

It seems to be another “AS3 rules!” post … using a proprietary plugin/player and winning for few milliseconds but I’m sure … I’m wrong …

The fact that the JSON data generation takes longer than the XML generation (and by a wide margin) makes me think this test is deliberately broken. I’ve seen client side native XML parsing be faster than JSON parsing, but XML generation is never faster than JSON generation. At worst JSON generation should take the same time as XML, since JSON and XML are really very similar representations.

There are many factors to consider in RIA performance benchmarking. My Census app is designed to show seven metrics: server exec time, transfer time, parse time, render time, bandwidth usage, client side memory usage, and client side sort time. These metrics may or may not be useful for what you are building but they do usually reveal the performance benefits of having a binary protocol for data transfer and having a JIT enabled client-side VM. In some applications that matters and in some it does not. You can read more about my thoughts on this benchmark in the original blog post:http://www.jamesward.org/wordpress/2007/04/30/ajax-and-flex-data-loading-benchmarks/

@Andrea – The code is all on SourceForge:http://sourceforge.net/projects/flexapps
Check out the census project from cvs.
Also JSON is usually only slightly smaller than XML because all you really save on is the close tag.

@Stephen – JSON is slower because I’m using a Java Library to do that piece. If I were not using a library and just generating the strings then you are right that it would be faster than XML. I am not trying to deceive anyone but instead trying to have a benchmark that models the real world uses of these technologies. The code is open source. Please take a look at the code and if there I changes you think I should make let me know.

@Rey – Now that I look at your screenshot it appears that you may have run the Ajax JSON test with more rows than the other tests since the bar is a bit bigger in height. That might also explain why that test is showing different results than what I usually get.

@James: I’m not sure if that was the case but I’ll rerun the tests and repost a new pic just to be sure.

Comment by Rey Bango — August 28, 2007

This benchmark should be taken with a grain of salt the size of a volleyball.

If I understand correctly, the author knowingly lowered the transfer time and bandwidth results for paged Flex 50 (fifty) fold. He writes “I ran a test using 5000 rows” whereas the results for paged Flex are given for a transfer of a 100 rows.

In comments above, he writes “JSON is usually only slightly smaller than XML because all you really save on is the close tag.”, which in itself is only a half-truth at best. What I’d like to know, though, is why does the “slightly smaller” JSON transfer take twice the bandwidth of XML?

I’m sorry, but this benchmark seems to be paid for by Adobe.

Comment by Michal Tatarynowicz — August 28, 2007

@Michal – Only one Flex test uses data paging. All the others do in fact transfer all 5000 rows. Also I have already noted that it is possible that Rey may have loaded more rows for the JSON test. He is looking into that. Feel free to run the tests on your own and see what results you get.

The JSON results seem extremely skewed. How is it possible that a JSON has more bandwidth than XML? Also, once the JSON is parsed into a native object on the client it is much faster than querying an XML DOM over and over.

@Charles – For many applications Ajax JSON is enough. Many others need the power of a JIT enabled VM and a binary serialization protocol. I created the Census benchmark to help people understand when Ajax JSON is enough and when they need something more powerful.

There’s a chart that makes it look like flex uses a tiny fraction of the bandwidth used by other techniques. But when you read more of the details you find that the flex test just doesn’t load all the data.

What–other than an attempt to deceive–would cause someone to make a chart comparing flex’s performance on partial data against other technologies’ performance on a full data set?

Comment by Rufus — August 28, 2007

@Rufus – The chart labels that bar as “Flex Paged”. I am sure that Rey, who created the screenshot, had no intention of deceiving anyone. Also I believe that anyone who uses the benchmark app understands that the data for that test is paged in. Is there something more I should do to make if clear that the data for that test gets paged in?

@Rufus: I took the screenshot posted. I simply went through each test listed and ran it. That’s it. In fact, I re-ran the tests and reposted the results as an update because James pointed out a possible discrepancy with the Ajax JSON results. I don’t believe James is trying pull a fast one on anyone but is simply trying to show some benchmarks that he’s generated. If you’re concerned about the tests, you should contact James and ask for the code so that you can evaluate it and come to an opinion. I’m sure that if you come to different results, James would be more than happy to compare notes.

Comment by Rey Bango — August 28, 2007

I’m sorry for being so harsh on James, and I know that a _professional_ would probably know how to read the results. The problem is that _my boss_ (a professional, for sure, but in a different field) could take look at them, and ask me to redo our apps using Flex, which would be unnecessary and potentially damaging (not to mention expensive and time-consuming).

Thank you for updating the tests. I still think that paged Flex bandwidth test (which _does_ transfer less data) result should be separated from other results and marked for obviousness, or removed from the test whatsoever — it’s comparing apples and oranges. Think about it, wouldn’t you raise an objection if I posted results showing that Ajax/Json used 50 times less bandwidth than Flex?

Comment by Michal Tatarynowicz — August 28, 2007

@Rey – The code is all open source on Sourceforge: http://sf.net/projects/flexapps and feel free to email me if you have any questions / comments: james jamesward org

@Michal – Paging is not something specific to Flex. It’s a technique that is widely used for large data sets. Flex (as well as Open Rico) support transparent paging in datagrids. Other frameworks utilize a user driven paging paradigm. Either way you are right that paged data is a different paradigm than non-paged data and I should make this more clear in the results. Thanks for the feedback. This will be at the top of my list for the next version of this benchmark app.

A question about AJAX-JSON: Is there a performance difference if you dynamically create a script tag which loads a generated js file in which all objects are serialized to JavaScript objects (like JSON but without having to eval() the XHR response)?

@Frank: No prob. Sorry for the initial confusion. I guess I clicked too quickly. :)

Comment by Rey Bango — August 29, 2007

> Other frameworks utilize a user driven paging paradigm.
>This benchmark was created specifically to show how RIAs deal with large data sets. [from blog comments]

I would be very interested to see a simple paging vs paging test! I think paging is the obvious solution for any dataset large enough that the size creates a performance issue, whether the paging is coded from the front- or back- end. I imagine enough developers would be interested that paging code would be easy to find.
On the other hand, if Flex is crazy fast with data, I’d like to see how it handles situations where you don’twant paging, and you want5000 rows. (i.e. graphing large datasets)