decodiﬁed

Micro-Benchmarking Scala Code - The Easy Way

While evaluating, which of the numerous available Scala JSON implementations to use for spray once again I found myself in need of some tool simplifying the writing of proper JVM benchmarks.Manually writing benchmarks for the JVM that actually measure what you intend to measure is much harder than it initially appears. There are quite a few rules to keep in mind, even making your hand-written benchmark comply with just half of them requires a lot of attention.

As ever so often other people have already had the same problem and decided to do something about it. In this case a few guys at Google came up an open-source framework for JVM micro-benchmarks called Caliper. Caliper takes care of many of the hard and complex parts of a good micro-benchmark, like properly “warming up” the JVM before actually taking measurements or determining how often to run your code in order to achieve a good trade-off between measuring accuracy and total runtime.Of course there are still a few things you have to watch out for, even when using Caliper. However, the framework can be of great help when you’d like to decide between two alternative implementations of the same algorithm based on performance, determine whether a particular “optimization” is really worth the effort or find out how a certain JVM setting affects the runtime performance of some ingenious code of yours.

In order to make Caliper easily accessible to Scala developers I whipped up a small SBT template project that you can use as the basis for your own benchmarking endeavours. As a simple example the project already contains a small benchmark testing the performance of ”foreaching” over a Scala Array against a simple while loop as well as a @specialized custom for loop replacement implementation.This is what the Caliper output looks like when run from within SBT on my machine: