Maven, Continuous Integration and Rotating Knives

On the Bamboo development team we recently spent some time investigating how a wrong artifact ended up in one of our dogfooding servers. Apart from the awesomeness of dogfooding, it highlighted the perils of maven and its implications on continuous integration (CI).

The mystery:

A WAR deployed to our dogfooding server contained the wrong version of a library

The contents of the library installed differed from the corresponding artifact in our internal maven repository, despite having the same groupId, artifactId and version (GAV)

We traced the rogue artifact back to a Bamboo build, which never deployed to our maven repository!

So how did it happen?

Maven does not check remote repositories for stable versions of artifacts already in its local repository

A stable version of an artifact was being mvn install ed. Changes were still made to this line of development even though it had a stable version

Maven repositories on a build agent are not cleaned out between builds

Problem

When developing locally, we often use mvn clean install to build the project we’re working on. The install phase of the default maven lifecycle will place the freshly-built artifacts of the project being built into your local repository (~/.m2). This can be a problem in CI because the same build agents are used to build different plans, and these plans may then pick up locally-installed artifacts from a previous build.

To illustrate the problem, suppose we have project A with a dependency on project B

Project A and B have separate plans which build and deploy artifacts, e.g. say PROJACI does mvn clean install, PROJADEPLOY does mvn clean deploy, and so on with PROJBCI, PROJBDEPLOY.

Suppose that we only had one build agent, and the sequence of events ran like this:

PROJBDEPLOY -> PROJBCI (as a result of a new commit) -> PROJACI -> PROJADEPLOY

Using install, PROJACI would pick up the version of project B that was built in PROJBCI, as opposed to whatever is in the canonical remote maven repository (PROJBDEPLOY). This can lead to strange and mysterious problems where the canonical maven repository says one thing, but your build that depends on a SNAPSHOT artifact says another.

Thankfully, an internally developed Bamboo plugin removes SNAPSHOT artifacts in the local repository to maintain consistent disk usage across our build agents, but it also has the useful side effect of preventing such problems since we will now always resolve SNAPSHOTs from the remote maven repository.

Unfortunately, if you are constantly installing a stable version, and not a SNAPSHOT you will still run into this. When using a SNAPSHOT, maven will attempt to retrieve the latest version based on timestamp. Maven assumes that non-snapshot versions are immutable, so if a stable version of an artifact exists in its local repository, maven will not go to the canonical remote repository to check for a newer version and retrieve it.

So suppose we have project A depending on project B v 1.0, an unfortunate sequence of events could be.

CI: Build project B v1.0 on agent 1 (installs projectB-1.0)

Locally: Someone makes a fix to the branch and pushes

CI: Build project B v1.0 on agent 2 (installs projectB-1.0)

CI: Build project A (which depends on projectB-1.0) on agent 1.

Project A will now be built against the first artifact of project B, which will not contain the fix. This was what happened in the rogue library artifact that went into our dogfooding WAR.

It’s worth noting that this would be less of a problem if you are constantly installing a stable version from a tag with no new commits. It’s only if you are making commits to a stable line of development, i.e. what projectB v1.0 is constantly changes, where things get hairy. Maven assumes that you’re using SNAPSHOTs for this purpose.

What you can do about it

Some options are:

Don’t use stable versions for active branches of development (will not prevent SNAPSHOT problems)

Use verify instead of install for CI

Ensure you clean up. Codehaus’ build-helper plugin has a useful remove-project goal that removes locally-installed artifacts for a project

Clean up the maven repositories between each build

The first option is a no brainer, and you should not be doing it anyway. Maven considers stable, deployed artifacts to be immutable, which is why if it has a local copy it doesn’t bother checking for a newer one. This is also why allowing re-deployment of stable artifacts in a maven repository is frowned upon, as pre-redeployment artifacts in consumers’ local repositories are not updated. However, it will not stop similar problems with SNAPSHOTs.

If you use verify instead of install for CI, this prevents locally built artifacts from entering the local repository. Like the previous option, it relies on everyone to play nice. All it takes is a single rogue build to waste a day trying to figure out what went wrong. This same problem affects the third option. It is sometimes hard relying on everyone using the infrastructure to be a good citizen .

The last option is the most hermetically secure. It also requires the least change from teams using the build infrastructure as there is no shared resource to pollute. It gives you the highest level of build isolation, at the cost of latency while the local repository gets populated each build. While this may be an appealing option for smaller projects, it quickly becomes infeasible when you depend on WARs or decent sized binaries that have to be downloaded. If you are fortunate enough to have a low-latency link between your maven repository and your build agents however, it might still be worth considering.

Ultimately what steps you take depends on specific organisational factors; how is your build infrastructure set up, who handles the actual creation and/or maintenance of builds, how big a stick you have and so on. If anything, I hope this has helped highlight some of the potential problems you may run into when setting up CI with maven builds, and given you some ideas on how to deal with them.

Using package is a bad workaround IMHO.
With Maven2 this can cause inconsistent builds if your build needs an artifact with a non standard classifier. This is due to a maven2 bug that will cause the artifact to be downloaded from the remote repo rather than realizing it should be got from the reactor. and if no one has deployed a version of this g:a:v (think snapshot) then the build can just fail – which makes the CI slightly less useful than it should be.

using install is the correct way – however you should be using a local repository that is wiped clean before each build.
Using a clean local repo also has several other benefits – such as ensuring that the build is fully reproducible and that external artifacts have not disappeared (from your local maven repository)

By James on December 20, 2012

I’m hesitant to call a workaround for a maven bug the correct way 😉 I believe the issue in question is http://jira.codehaus.org/browse/MNG-3506 , which has indeed caused us pain. We have other workarounds in place for this, though we want to eventually move to a newer version of maven where this is fixed.

In an ideal world, we would start from a clean repository as mentioned above, though the cleanliness it provides has to be balanced against speed of artifact resolution during the build.