Part 2: OpenJ9 versus HotSpot

Intro

Yesterday I compared different JDK versions and OpenJ9 versus HotSpot on memory and speed. The memory part of the test was realistic if you ask me, an actual working Spring Boot application that served REST objects.

Today I decided to test OpenJ9 and HotSpot a bit more using an actual benchmark: SPECjvm2008.

SPECjvm2008

SPEC (Standard Performance Evaluation Corporation) has a couple of well defined benchmarks and tests, including an old JVM benchmark called SPECjvm2008. This is an elaborate benchmark testing things like compression, compiling, XML parsing and much more. I decided to download this and give it a spin versus OpenJ9 and HotSpot. This should be a much fairer comparison.

Initially I encountered some issues, some of the tests didn’t work against Java 8 and the tests wouldn’t even start against Java 9+. But eventually I got it working by excluding a couple of benchmarks with the following parameters:

Results

After waiting a long time for the benchmark to finish, I’ve got the following results:

The graph is measured in ops/m, higher is better. Results may vary of course depending on hardware.

In most cases HotSpot is faster than OpenJ9, and in two cases HotSpot is much faster, crypto and derby. It appears this is a case where HotSpot is doing something special that J9 isn’t doing (yet?). This might be important to know if you’re working on applications that do a lot of cryptology, for example high performance secured endpoints.

One place where OpenJ9 came out on top is XML validation. Parsing/validation is also an important part in most modern applications, so this could be a case where J9 makes up some lost ground in actual production code.

Conclusion

Is there a real conclusion from this? I don’t think so.

The real lesson here is: Experiment, measure and you’ll know.
Never decided anything based on some online benchmark.

If there is anything else you’d love me to test, send me a tweet: royvanrijn