Other Projects

Blog

August 13, 2012

Zinc and Incremental Compilation

August 13, 2012

sbt is arguably the best tool for building Scala projects, with two of the key features being its incremental compiler and its interactive shell. An incremental compiler will only compile what needs to be recompiled after changes. A warm compiler can compile your code twice as fast, or faster. If you’re committed to using a build tool other than sbt, for whatever reason, then you’ve generally been out of luck for fast incremental compilation… until now.

We’ve separated out sbt’s incremental compiler and made it more widely available through a build compiler called Zinc. Zinc has already been integrated with the Scala Maven Plugin and is currently being integrated with a new build tool developed at Twitter called Pants.

Zinc

Zinc is a stand-alone version of sbt’s incremental compiler. It comes with a command-line interface and built-in Nailgun integration. Zinc can be used as an alternative to scalac, adding incremental compilation, and as an alternative to fsc, serving as a build daemon.

Running with Nailgun provides Zinc as a server, communicating commands via a client, keeping cached compilers in a warm running JVM and avoiding startup and load times. This gives the same benefits and performance as sbt’s interactive shell.

Scala Maven Plugin

Compiling with Zinc has been added to version 3.1 of the Scala Maven Plugin, bringing incremental compilation for both Scala and Java to Maven. We’ve also added support for talking to a Zinc server, which can improve compilation speeds dramatically.

Example

As a realistic example I’ve taken the first few modules of Akka 2.0 and set them up as a Maven project. The modules are akka-actor, akka-testkit, and akka-actor-tests, with both Scala and Java sources, and akka-testkit has both main and test sources. I’m running this example on a MacBook Air.

Let’s first look at a regular clean compile of all the code, without incremental compilation:

This takes a little longer because the incremental compiler analyzes the compilation so that it knows how the sources relate to each other. For this project it adds around 20% overhead for a full compile.

Now that we’ve compiled everything, with the incremental analysis ready, let’s try changing just one of the tests in akka-actor-tests, in this case ActorRefSpec.scala, and recompile everything:

So in this case it actually takes longer than a clean compile, which is useful to know. The logging shows the files being compiled, starting with the initial invalidation of Actor.scala and then subsequent compiles triggered from this change.

Nailing it

When compiling from sbt’s interactive shell, the JVM is warm and the classloader containing the Scala compiler is reused. This reduces a lot of startup and load time. Zinc supports similar warmth through running as a Nailgun server, and the Scala Maven Plugin supports running as a Zinc client. This can be enabled alongside the incremental recompilation configuration option by setting useZincServer to true.

Let’s look at the same example compiles as above, a clean compile, a change at one of the leaves of the source tree, and a fundamental change, but using a Zinc server. First, we need to start Zinc:

Nice! This is almost a third of the time compared to the full clean compile with a cold incremental compiler. The warm incremental compiler, after a handful of compiles, knocks 60% off the time of the baseline clean compile.

Using the Zinc server gives us sbt-like compilation from Maven, and a warm incremental compiler can compile code twice as fast, or even faster. Incremental compilation can also be used with the continuous compile (scala:cc) goal in the Scala Maven Plugin. For more information about using incremental compilation in Maven see the documentation for Scala Maven Plugin.

Pants

Pants is a build tool developed at Twitter and is modeled after Google’s build system. It’s intended to support selective compilation within large, heterogeneous codebases with an explicitly-modeled DAG structure. Pants is used internally at Twitter and at foursquare, and is also used to build the twitter/commons open-source project.

Until recently, Pants invoked the scala compiler directly. If any file in a DAG node changed, the entire node and anything depending on it had to be rebuilt. Benjy Weinberger from foursquare is currently working on integrating Zinc with Pants, so that it too can enjoy the benefits of incremental Scala compilation:

The tricky part of this integration is the fact that Pants contains
its own node-level invalidation logic. Pants also has options to both
‘flatten’ and ‘unflatten’ the build.

The advantage of the non-flat mode is that the compiler only has to
bite off small chunks at a time. In small codebases this is slower
than building everything in a single pass, because you incur the
compiler startup overhead multiple times (although Zinc’s cached
compilers can help with this). However in large codebases it can be
impractical to build everything in a single pass. Building a couple of
thousand Scala source files may require 8GB or more of RAM and can
take 20-30 minutes. Breaking the codebase down into a modular DAG, and
applying Zinc’s incremental compilation to each node, supports
effective builds and fast rebuilds even in a large codebase.

The combination of Pants and Zinc will allow companies like Twitter
and foursquare to work productively in a single shared, large-scale
codebase, even if any individual developer only builds a small subset
of the codebase.

Summary

We’ve separated out sbt’s incremental compiler and made it available both stand-alone and ready for integration with other build tools. Using Zinc as a build daemon, similar to sbt’s interactive shell, can dramatically improve build times. Zinc has already been integrated with the Scala Maven Plugin, and we’re expecting incremental compilation to find its way into other build tools for Scala. Zinc is also being integrated with Pants, a build tool for large-scale codebases.

Have a Question?

From the blog

The Typesafe crew is thrilled to share that Scala Days SF was just fantastic. A big thanks to all who attended. We were wowed by our awesome keynoters, speakers and volunteer staff, and it was great to feel the excitement and energy at the beautiful Fort Mason.

After an inspiring Scala Days (the next one is in Amsterdam), it's great to be able to shine some light on technologies dedicated to improving the workday of Scala developers. We recently talked about eight hot technologies that perhaps you didn’t know were built in Scala, and in the spirit of that we’re happy to highlight Takipi, a company that's making life for commercial Scala apps better. Branching out from Java, Takipi now helps Scala developers understand when and why their code breaks in production. For more details, we asked Josh Dreyfuss, who recently joined the Takipi team, to take us through it all. -Oliver White, Typesafe, Inc.