Continuous Integration For Scala Projects

Feb 28, 2015

So what is continuous integration, at the very high level it’s about hey is
the code in a good state, no matter by how many people code is being updated,
how often code is being changed or of how many building blocks project consists of.
When you start working on a feature you’re gonna branch of mainline and start
introducing changes. At some point when you find things are broken now you have
to spend time to see if it’s because of your changes or project was already in
a bad state. And when you find out that it was already in a bad state you
would want to start tracing. A who introduced the breaking changes and B when.
So to avoid the head each and frustration what you really want to know is
if project is in a good state before you even start working on it.
One option would be to compile the code before you start changing it.
But if code compiles is it enough? Oh you you also wanna know if unit tests are
passing. But is this enough? Check for compiler warnings? Style violations?
Code coverage? What if you run code coverage and it shows that it’s 85%,
does it mean everything is good, what if code coverage was 95% last week and
code without proper code coverage got committed and code coverage dropped
(do you remember what code coverage was last week, can you even find
out without going back in commits and running coverage reports on those commits.) are you sure
you want to introduce your changes on top of those changes?
Let’s make it even more complex, what if the project consists of many other
projects and the project you’re working on brings all those dependencies
tougher. Maybe you should checkout all those dependency projects
and see if those are in a good state also.
Let’s make it even more complex, what if the project isn’t just set of
projects that get bundled into one big project.
What if the whole code base consists of many micro services that talk to each other
via some form of PRC. OK maybe you can manage to do all this to know if code is
in a good state before you introduce your changes what about the point B I
brought earlier about when code broke, because longer it was in that states
longer it’s gonna take to trace the breaking change and longer to fix.
Can you go and check every single project every time someone makes a commit
to make sure that commit is good? Well actually you can, you just need a
build server. So all those points I was bringing was about whatever your
integration is it should be continuous and it’s continuous if it runs after
somebody changes something (or maybe before is a better option?☺ I’ll get
to this later.)

This should cover what the continuous part is but what about integration,
what is it actually? I would define it as pipeline of checks that run one
after another and if one fails the whole thing fails if all pass then
code is good. Now let’s cover what checks you want to do. You’re the best
person to know what checks you need and what you wanna track. I will
cover the most common ones, at least the ones that are relevant for
Scala code base. Also I would cover how to do it in Jenkins since
Jenkins is my preferred build server but it should me mostly similar
other the other ones.

Compilation

First one that everybody will agree is compilation, does the code compile?
Compilation is one you get for free since Scala has a compiler, it’s is
type safe at the end of the day right. Let’s make sbt compile,
sbt test:compile and sbt it:compile (if you have last two of course)
our first item in this pipeline. If code doesn’t compile no point to do any other
checks, if it does let’s continue.

Since we compiled the code and we got the output we can look for compiler
warnings, so can Jenkins.

Code quality

We know what compiler warnings are but we can do more checks to see if there
are other issues with code also. Scapegoat does exactly that, it’s a scalac
plugin that does static code analysis and runs when you compile your code.
There is an sbt plugin for it and I already talked about it in my previous post
about sbt plugins so make sure to check that post out if you haven’t already.

Since we’re on static code analysis subject let’s talk about style violations
also. Do you care to see everybody following the same style, using right
conventions for method names, class names, variables, etc?
There is Scalastyle which is like Checkstyle for java that does that and much
more than that. And guess what it actually outputs Checkstyle xml report and
Jerkins has a plugin for Checkstyle already. We’ll run this via
sbt scalastyle task and get the report file generated. Once we have it we
can tell Jenkins to pick it up.

Open tasks

Another thing that would be helpful is to know if there are open tasks in
project and if there are what are those. Jenkins has a way to detect
tags like TODO, FIXME and others. Jenkins has a feature where it can scan the
code to find those, just configure it.

Unit tests

Once our code compiles, we’re happy with styles, etc we want to make sure that
unit tests are passing. sbt test can output junit xml report and once we
have it Jenkins already has an integration for junit so we would be done there
just tell Jenkins there those xml files are. Same goes for other flavors
of test, like it:test.

Code coverage

When unit testes are passing it’s still not enough we want to measure what
the code coverage is and of course find code paths that are never being hit
from a unit tests. There are couple options in Scala world for code coverage,
I’m using scoverage. Nice thing about it is that it generates cobertura
coverage report and as you can guess Jenkins already has integration with
cobertura.

At this point we got enough checks in already. Build on it’s own is boolean,
it either passes or it fails. For things like compilation and unit testing it’s
simple if one of those are failing fail the build. For there other ones, when we
measure it, we have numbers like number of compiler warnings, number of check
style violations, percentage of code coverage. In Jenkins we can configure those
so if a particular number goes above value X we fail the build. X can also be
zero depending how aggressive you wanna be there. Violations plugin can be used
there.

Documentation

Another thing we want to get is auto generated documentation, java docs to
be more specific (well in this instance scala docs).
sbt doc (or some other task that depends on doc task, publish for
example) will generate scala docs and Jenkins can pick it up and display
the link on the dashboard or we can host it as a static website somewhere.

Final Sbt configuration

Using the sbt plugin we can configure this. And this series of sbt task would be
enough: clean coverage test it:test scalastyle publish.
This will compile main, test and integration test code-bases,
run Scapegoat as part of compilation (assuming you have it configured for
you project), run tests and integration tests, publish the artifacts
(will cover publishing later in the post). And doc task run as part of publish
since it’s a dependency of it but it never hurts to put it there explicitly.

Notifications

So build pipeline is there, now every time there is a commit in source control
build will trigger and it will either fail or pass. Either way we get instant
feedback which was one of the main problems we wanted to solve. You can get notifications
when job stars, fails or succeeds. I generally don’t care to be notified when a
job succeeds unless it’s deploy job or something (which I won’t be covering in
this post.) but I absolutely care to know when a build job is failing.
You can do pretty much all types of notifications: Email, Slack, Hipchat, IRC,
IM whatever you’re using, Jenkins has plugins for all of those. I personally
like to send an email out when a job fails and put a message in a Slack room
when a job starts or finishes.

And include Slack notifications at post build step at the bottom of job
configuration.

Output of the build

We define build to be a pipeline of checks that the code goes through
every time there is a commit. We torture every single commit to go through this
pipeline and we say if build passes commit is good. So what is the output of the
build? Also we only deploy good changes, right? The output of a good build would
be the artifacts. We worked really hard to make sure that code is good by running
series of checks on it and we better keep those artifacts for deployment. In scala
artifacts would be jars or wars, depending on the project. There are few options
on where to store the artifacts depending how you’re planing on using them. If
it’s a project that’s gonna be referenced by other projects, artifact would be a
jar archive and storing it in an artifact server like Nexus would be a good
choice. If it’s a service or a web site you’re planing to deploy you can put it
on s3, upload to elastic beanstalk if it’s what you’re using or create a docker
container, put that jar or war in it and push it to a docker registry. There
are nice Sbt or Jenkins plugins for all those. For Nexus I use an sbt plugin
and when you do sbt publish it uploads the artifact to my private nexus
server.

Versioning

And since every good commit produces an artifact (or set or artifacts) we should
talk about versions. We already specify the version in sbt file and when
sbt package is run it creates the artifacts with specified version.
One thing that would be good if we could link a specific version of an artifact
to source code. One good way to do this is to create a tag on commit when
we build the artifact. Which isn’t hard to do, we can get the version by
running sbt version, storing it in an environment variable
(VERSION for example) and use Git publisher functionality of Jenkins to create
a git tag with that version.

Pull requests

Remeber I said it would be better to know if a commit is good before it even
gets to mainline. With the setup mentioned so far when a commit gets to master
build will trigger and if it’s bad job will fail. To not allow bad code
to even get to master at first place we need to run integration on it before merging it.
We never merge a branch when you do code review and identify that it needs
more work, right? But what you can’t tell from just code reviewing is if it
compiles, unit tests are passing, what is code coverage after the change, etc.
And do we even want to code review a change if build will fail for it?
What we need to do is to have a magical way that when we go to review a pull
request somehow it has a flag on it saying build is passing for it or no.
There is a Jenkins plugin for building pull requests also, it will run the
build, and comment github pull request.

To get this working all you need to do is to create another job that instead of
building master will build pull requests. For this job there are few things
we don’t actually need. We don’t care do generate scala docs for this type
of builds, version or publish the artifacts, so make sure to exclude those steps
from pull request builder jobs.

Wrapping up

Let’s see what we got so far. Every time there is a commit build will start
immediately and we’ll get instant feedback whether build passes or fails. If
build passes we know that all checks we defined are good for the commit.
We create tags for each version in git, we publish artifacts for deployment or
referencing from other projects.
Other than those we get lot of nice features on Jenkins dashboard.

We can see the status of the build.

See the change log with commit messages, committers and links to github.

We can browser the workspace.

Start a build.

Navigate to github page of the project.

See detailed reports for checkstyle or other static analysis warnings.

Navigate to scala docs.

See code full coverage reports with percentages, which lines are covered which ones aren’t, classes, packages.

We can see the build history and dig dipper in a particular build.

Get height level view or the project with number of violations and graphs that
shows whether over time number of violations, open tasks, number of unit tests
increases or decrease’s. Of course we want it to see number of violations and
open tasks decreasing and number of unit tests increasing. Just a good way to
measure how quality changes with every commit.