The benchmark

Same disclaimer as last time : This is not a real benchmark ! so I’ll copy paste the paragraph I wrote last time :

In this test I’m just concerned about the usability of an application server for a developer. The idea is to download it, install it, start it and take some measurements : size of download, ease of installation, size of disk, startup time, size of RAM... That’s all. No deployment of an application, no fancy twists to gain performance… Because some of these application servers are resource consuming, I’m doing all my tests on a Windows XP SP3 virtual machine (running on Virtual Box 4.1). It is a fresh install of Windows XP sp3 with 1Gb of RAM and not too many software installed. So when I boot there are 27 processes running and using 230 Mb of RAM (leaving 770 Mb free). Virtualizing can be slower, so keep in mind that startup time can be a bit faster that what I’m giving you here. I use JDK 1.6.0_27 (when it’s not bundled with the server). No optimization at all is made (I haven’t twisted the JVM, or antyhing application server parameter, everything comes out of the box).

To calculate the startup time, I don’t do any fancy rocket science either. I just start the server a few times, check how long it takes and use the best startup time. Also remember that some servers do not load any container at startup, making them very fast to start. That’s why I first trigger the web admin console (when there’s one) so I’m sure at least one web application is deployed (I did a print screen of the admin console when there’s one). Then I just stop and start the server a few times and get the best startup time (by reading the logs). To calculate the memory footprint, I just use the Windows task manager and check the size of the java.exe process.

GlassFish 3.1.1

GlassFish 3 is the open source reference implementation for Java EE 6. The 3.1 release has added clustering capabilities. Being the reference implementation, GlassFish 3.x implements both Java EE 6 and the Web Profile 1.0, that’s why you have two different bundles you can download (one of 80.6 Mb and the other 45.8 Mb).

Resin 4.0.23

Resin 4 was one of the first web server (such as Jetty or Tomcat) to move to the Web Profile 1.0. For me it’s the success story that confirmed that profiles in Java EE were needed. It’s really difficult to go from a Web Server to a full Java EE 6 application server (remember that there’s still Entity CMPs in EE 6) but much easier to implement the Web Profile. Resin is very CDI centric and based on the Caucho implementation called CanDI.

JBoss 7.0.2-Final

It looks like JBoss 7 is, finally, a nice millésime. For those of you who waited so long for JBoss 5 to get certified and then didn’t see any JBoss 6, JBoss 7 is what you need with a fantastic startup time compared to the older versions (in my previous benchmardJBoss 5 was the slowest app servers, slower than WebSphere 7). Another novelty is an admin console (sexier than the good old JMX Console).

TomEE 1.0.0-beta-1

Like Resin, Apache TomEEis the perfect success story for the Web Profile. TomEE is no more than Tomcat + OpenWebBeans + OpenEJB + OpenJPA + MyFaces + other bits. It really shows that Java EE is a jigsaw puzzle where you can take open standards, bundle them together and become a certified Web Profile application server.

(*) Looks like to be able to log on to the admin console, you need to change the %TOMCAT_HOME%\conf\tomcat-users.xml configuration files and add a manager-script role to a user. But I couldn’t make it work.

Siwpas Enterprise 2.0.0-GA

Siwpas (Simple Web Profile Application Server) is quite new compared to the good old application servers. But like most of them, it embeds well known open source stacks, most from Apache (Tomcat, OpenEJB, OpenWebBeans, MyFaces, OpenJPA, Apache BeanValidation). It’s open source but you’ll still need a 30 days trial license to startup the enterprise version of the server.

Websphere 8.0.0.1

Websphere 8 is the Java EE 6 application server of IBM. Like the previous versions, you need a big bandwith to download one gigabyte of modules that get installed in nearly 2 hours. I noticed that there is a developer edition but once installed it looks like the 2.2 Gb of the full server (I might got lost in the maze of the IBM website). There is also a Community Edition and it looks like it’s based on Geronimo (and the executable to install it is also 116 Mb). IBM, please, make it simple.

Thanks to @sebsto I found a gem : it looks like IBM is working on a version WAS 8.5 Alpha that would look like the other app servers (light and easy to install). I’ll stay tuned on this one.

Jeus 7

A bit unknown but in version 7.0 (so it shows this application server has been around for long time), Jeus 7from TMaxSoft was a bit difficult to startup (create an account on the web site, download, install, run a wizard and read the README.txt to make it started). Then I discovered that there is no Jeus 7 for Windows so I downloaded the Linux version and installed it on Ubuntu (so the results of the microbenchmark are not very accurate compare to the other made on Windows).

Interstage

Where the hell is Tomcat ?

Tomcat 7 is not a Java EE 6 application server nor does it implement the Web Profile 1.0. It is just a Servlet 3.0 container, so it’s shouldn’t appear in this blog post. But, everybody likes Tomcat, everybody feels Tomcat is way much faster than any server on earth (even faster than Neutrinos or maybe not). So I had to put it in this benchmark just to be a source of comparaison. And, indeed, Tomcat is the fastest application server of the list (3 times faster than GlassFish or 4 times faster than JBoss). But it only does Servlet 3 while the others implement between 12 (Web Profile) or 28 (Java EE 6) specifications.

Summary

As a summary I will show you two graphs with startup time and resource consumption so it’s easier to compare.

Startup time

When you are a developer, startup up time matters (it doesn’t when you are in production). Thanks to tools like JRebel or the hot deployment of our IDEs, we tend to restart our app servers less and less. But, sometimes it happens (PermGens, classpath mess up, memory leaks, refreshing app server cache…). I didn’t put WAS 8 nor Jeus 7 in the graph below because they would just break the scale (42 and 23 seconds). It’s interesting to notice that 4 app servers (Resin, GlassFish, JBoss and TomEE) start in less that 5 seconds (Jonas & Geronimo are still in Milestone). Resin is 3 times slower than Tomcat, but again, it comes with the Web Profile.

Size once installed on the drive & size of process in RAM

The size of the application installed on your disk is not very accurate : some app servers come with documentation, samples… others no. But still, that gives you an overview of how big an app server can be (again, I couldn’t put WAS in this diagram as it will break the scale with its 2.2 Gb). The memory consumption is more interesting. Without any JVM or server twists, when you startup, they nearly all take less that 100Mb of RAM(see how close TomEE, GlassFish, JBoss and Siwpas are).

Conclusion

First of all, as you can see there is no Weblogic. Unfortunatelly Weblogic is the last big player to be certified Java EE 6. I don’t know the roadmap, hopefully I’ll soon be able to update this blog entry with the latest version of Weblogic. Second, this is not a real benchmark : I didn’t deploy any application nor I did some stress tests. One task I would like to do is to deploy a Java EE 6 application on all these servers and see how portable the code is. But that’s for another blog.

What did I want to show in this blog ? Well, that application servers have changed. Except for WAS, Weblogic or Jeus, all the others are fast at startup and take few resources (memory).

And why did I want to show this ? Because I’m fed up to still hear in 2011 : “Tomcat is light, Java EE app servers are not”. As you can see, in terms of startup and memory, the gap between Tomcat and Resin, GlassFish or JBoss is really small.

How is it going to affect your development life ? Do you remember when 7/8 years ago we had to mock our database accesses because we didn’t want to install a fat oracle database on our laptop ? These days are over now because we use in-memory databases such as H2 or Derby. Same thing is happening with application servers : soon app servers will be running in memory and we won’t be mocking anything. We will fire a test case, app servers and databases in memory, and that’s it. And with tools like Arquilian, it looks like this future is already happening.

Good post, great work! I think startup times are very important in development. There has been a lot of improvement over the past years. I wish it could have been a priority in the WL8 days :).

I’ve run similar measurements about a year ago ( http://dow.ngra.de/2010/12/14/jee-oss-container-startup-times-apples-vs-oranges/ ). There also we thought that next step is to actually deploy an application and measure the startup time then. Deployment will trigger a longer startup time because more services are used and will give a better picture. I wish there was a good app that could be used on all app servers (or maybe even on servlet containers) not just app server specific admin consoles.

The other problem with including tomcat in the comparison is that it is not running any applications, where the appservers are all (except Jeus 7) running (and have accessed) a fairly complex web application (their console). The startup and memory times for tomcat are not really comparable in that sense either.

I had the same problem with TomEE: could not create a user with admin console access. I figured out that there are some incosistencies between tomee documentation and example XML’s. A correct entry in tomcat-users.xml goes like:

Two important things here:

1. the “username” keyword (instead of “name” which can be found in examples)
2. The “manager-gui” role

Anyway, I found out that TomEE cannot run a simple servlet without web.xml, which is incorrect… Sad thing which makes me doubt in all the certification processes like web profile cert given to TomEE…

Bare with us on TomEE. The TCK uses JSR-88 for deployment and it’s become clear that this is just different enough from Tomcat’s drop-and-wait webapps dir approach to cause issues. We’re hacking up a second JSR-88 impl for the drop-and-wait approach and that will help us work out these issues. All should be well before TomEE goes final.

This is sort of useless data. I really don’t care how big the executable is, or how much RAM it takes or how fast it is to startup or how “easy” or “difficult” the installer is. Once a server is up, it stays up for a LONG time. Often times, when a server stars up it runs a bunch of initialization code, warms-up caches, loads data from persistent storage, initializes state, etc… That stuff take WAY longer than the server actually starting up and the data you hold in RAM dwarfs the actual base server cost. So, I really don’t understand the point of this exercise, it provides no really relevant data for a real-world scenario.

What is relevant is the performance characteristics of the various servers in terms of load, caching, messaging, DB management, etc… I want to know how does Glassfish do vs. Geronimo when we throw a couple hundred requests at it per second with a complex distribution in terms of various services, DB access, logging, event passing, etc…

What you’re asking matters, don’t get me wrong, but there are already benchmarks for that. SPEC for instance, you’ve probably heard about them.

Those benchmarks are needed when you are a (very) large enterprise that has a multi-million line application, serving an established base of demanding customers and that needs to deal with a lot of traffic and gives out SLA guarantees, QoS, etc.

For small shops, indy developers, people new to the platform trying out things, and I dare even say developers in general, the stuff Antonio tests here is of the *utter most importance*!

In development, not even with JRebel, my server DOES NOT stay up for a LONG time. I continuously restart and redeploy. The size of the executable matters A LOT when you have different workspaces in say Eclipse, and assign each workspace an individual server to keep things simple. If all that’s needed to “install” a new server is to unpack a 25MB tarball (TomEE), then the ease with which I can create new workspaces for some radical experiments is unparalleled (and yes, developers need to experiment a lot).

WebSphere here is atrocious. As a simple developer trying out things, I don’t care about uptimes, or requests per second. All I need is a Java EE runtime, so I can deploy my sample code and see how it behaves. Having to download gigabytes of data and sit through installs that take hours, is just undoable. From an operations point of view, where you installed WebSphere 6 in 2004 and let it sit there untouched for 10+ years, it might indeed not matter, but for an agile and dynamic developer it’s a total disaster. And I’m not even talking about the IBM JDK requirement and having to run both Eclipse and WebSphere with that JDK (which thus means I can’t run on OS X).

Exactly your idea, that difficult installs and gigabytes of enterprise cr*p don’t matter, is EXACTLY what has been driving so many developers to alternative platforms. If you can’t grab the mindshare of new developers, then eventually your platform will just die out. It’s as simple as that.

Luckily, not everyone thinks as you, since GlassFish, JBoss AS, TomEE, etc have been addressing exactly this requirement. Even former heavyweight WebLogic is almost there (it still requires an install instead of just an unpack, but they’re trying). And surprisingly, even IBM is getting the message as Liberty is exactly the kind of lightweight setup developers grave for. Unfortunately Liberty doesn’t support all Java EE APIs yet, but if it eventually does (even if only the web profile) IBM might have a winner there again.