/* HACK: refactor in next version */

Monthly Archives: August 2009

Note: before reading this article please see my views on comparing ORM tools. In short: this comparison is not suitable for comparing NHibernate and Entity Framework, due to the complexity of the tools vs the simplicity of this test; treat this post accordingly.

The reason I’m revising previous results is because some members of the NHibernate community looked into the code and discovered that the comparison wasn’t totally fair. While the test cases with Entity Framework used batch operations, the NHibernate implementation had this feature turned off. Tuna Toksoz was kind enough to re-write the critical parts of the code and apply settings enabling batching, you can download the revised source code here.

Note that I’ve made the source open for two reasons. First, I’d like the tests to be reproducible. And more importantly: I have experience with the two frameworks however I’m not an expert at any of them: therefore treat the results with caution. Anyone is free to analyze the source and if there are some issues in the code resulting in major performance drops, please let me know.

However improving performance does not stop here. Fabio Maulo posted a great article on why the original code was far from optimal and improved runtime results even more.

Conclusion

As I’ve stated before I don’t beleive there is a good way in comparing ORM frameworks and even if there were, this little test certanly is not it.

As the contribution of the NHibernate community members showed the more complex and configurable the tool, the more experience you’ll need to make the best of this. If you’re working on a performance critical product, you’d better know you’re tool well and use it accordingly.

But which one should I use?

From the current results it seems that there is no major performance difference between two tools and even if there is it might be due to misuse. So my advice is to use which ever tool suits you more based on its features.

At first glance Entity Framework seems to have a shorter learning curve as well as some nice integrated VS tools to make development easier. On the other hand there is a constantly active and helping NHibernate community always ready to answer questions and NHibernate seems to be much more customizable at a higher level. Also see this thread on EF vs NH features and learning as well to help decide.

The decision is up to you which one you find more suitable for your needs. However since both are proven and mature solutions you won’t be making a big mistake going with which either.

Posting of the previous performance comparison results (see here) has caused a much greater response I ever expected as well as some confusion about the goal of this test that I would like to respond to.

How real (world) is this?

Most importantly a performance test is useful if it can be used for predicting performance of a real world application. Now the performance test in I did was a really primitive, batch based CRUD test. Unless you are developing an application that does batch actions such as in this test, the results can’t really be used for predicting performance of real world applications. We’re talking about two mature frameworks with lots of additional services – e.g. caching and transaction handling – that are not evaluated in this test even though they can have a huge impact on performance.

Be a little sceptic

I would suggest treating results with caution and checking on the source. There are a lot of people expert on a single ORM framework, some are experts on two but it mostly stops there. Unless stated that the author(s) is/are experts on the topic I would approach these results in a sceptic way.

As for my own case: I do have experience with the tested frameworks however am far from being an expert with any of them – so treat my results with caution. I’ve published the so anyone can analyze it and if there are some issues in the code resulting in major performance drop, hopefully someone will let me know – just as some members of the NHibernate community did after publisihing the first results.

There is no “best”

I believe there is no valid way to decide which framework is ultimately better. There is an ongoing debate about ranking ORM tools and how well a test like this is good for deciding on which framework is better. I think it is not. Just to mention a few reasons:

Most real world applications have a far more complex working than this test simulates. What’s more, commonly used additional framework features – e.g. caching, transaction handling – are not tested at all even though these features can have a great impact on real world performance

Performance may not be everything. Even if these performance tests would reflect performance in a real world environment (as they don’t), ORM is not used based on performance. When using an ORM tool you make a compromise to have worse performance than using native SQL and gain additional features. It’s usually a tradeoff: the more features you get, the bigger the overhead will be.

There is one case when this kind of test can be useful though. If you find a great performance difference between two frameworks (e.g 50-1000 times), it’s a good indicator that there’s either some issue with the test itself or the framework. If you’re sure that the test is written correctly this comparison could lead to the analysis of why the framework is unacceptably better than the other. If you can find the valid answer, then good. If you can’t, now that’s when things get interesting and the test was worth doing.

The best valid way of comparing an ORM tool in my opinion is based on the project you need it for. Write together your needs, weigh them and evaluate the frameworks accordingly. ORM tools are comparable just as cars are. You can’t decide which one’s better unless you know what to use it for. An F1 car may be great for a speed race but would utterly fail in a desert challenge.

As part of my thesis I measured the performance of some .NET ORM frameworks including NHibernate and Entity Framework. Measuring was done by implementing two simple applications using the same table structure and doing the same operations on the same data.

Stating that Sense/Net 6.0 is an alternative for Vignette v8 does sound really bold though. Vignette is (well, was, until Open Text acquired it) a company on NASDAQ with over 600 employees with an annual revenue well over $200M while Sense/Net is a company of 45 and an annual revenue of about $2M. Vignette probably spends 10x more money on R&D than the total annual income of Sense/Net. What are the chances that Sense/Net can make a software even comparable to Vignette? Practically nothing.

The reason however that I made this statement is because I couldn’t prove the other way round – that Vignette is far more advanced than Sense/Net. The easiest way would have been to place the two products next to each other and evaluate them (actually this is what some vendor neutral companies like CMS Watch or J.Boye do in the forms of reports). As a simple developer/blogger though I didn’t have that option. An evaluation version of Vignette is not available for me – or even my company: we are just too small for a such a giant. So that left me to compare with the sources Vignette provided: the video and their webpage. And based on this pretty much the same functionality can be achieved with our open source software. If you are a developer you can even install the portal featured in the video and try our yourself.

I know that the comparison is not fair at all and that Vignette has a huge number of features that Sense/Net does not. I would love to get more information on Vignette to have a better comparison so if there is any publicly available data/demo other than on the Vignette v8 page, please let me know.