Breaking build is not a crime

For years I’ve been taught that breaking continuous integration build is something that should be avoided under all circumstances. Let me first quote few classics. Uncle Bob in The Clean Coder says:

The team must simply keep the build working at all times. If the build fails, it should be a “stop the presses” event and the team should meet to quickly resolve the issue.

and later in that section:

I have every developer run the continuous-build script before they commit.

Final quote:

They (CI tests) should never fail. If they fail, then the whole team should stop what they are doing and focus on getting the broken tests to pass again. A broken build […] should be viewed as an emergency[…]

In another wonderful book Continuous Delivery by Jez Humble and David Farley the authors go way further. They present 7-point plan that we should follow on every commit (!):

3. Run the build script and tests on your development machine to make sure that everything still works correctly on your computer

[…]

5. Wait for your CI tool to run the build with your changes.

[…]

7. If the build passes, rejoice and move on to your next task.

[…]

If the commit succeeds, the developers are then, and only then, free to move on to their next task.

They suggest the following best practices:

Never Go Home on a Broken Build

[…] When the build breaks on check-in, try to fix it for ten minutes.

To summarize what literature says about continuous integration:

always run all the tests before committing to make sure everything is green

look closely at your CI builds to make sure they pass, don’t proceed with further tasks

if you break the build, you must treat this as an emergency and fix it ASAP

you have very little time to fix, otherwise revert your changes

It’s not uncommon to have some hardware alarms triggering when build goes red. I heard of teams where build-breaking developer had to wear some silly hat or donate a dollar for charity (nothing wrong with that alone). Breaking build is considered a sin, assassination on team’s productivity, carelessness and laziness. CI server becomes this dreadful, scary machine that developers are afraid of. Bamboo even gives each developer points based on the total number of broken vs. fixed builds.

I fully understand this point of view and behaviour, but this doesn’t mean I agree. I feel this workflow is just plain wrong. I am aware that the whole team is working on the same HEAD/trunk in version control so breaking it is possibly a show-stopper for all of them. But I’m against treating CI/source control as some scarce resource that is so mission-critical.

Continuous integration server and VCS should be your personal team-mate, doing work for me. No one is really paying me for staring for, say, 5 minutes at my IDE before each commit to make sure all tests still pass. If that’s the case, I’m suppose to watch CI build blindly for next 5 minutes. If I broke the build, they expect me to drop everything and just jump in, trying to fix build within 10 minutes, as Continuous Delivery suggests. All this in emergency, stressful manner. Why?

Back in the days of Java 1.4 we were taught that concurrent programming using wait() and notify() is hard. But instead of giving up concurrency we came up with better and easier to use abstractions and libraries. At the same time we were reluctant to rename classes as this also renamed .java files, operation not well supported in CVS. But instead of keeping old names forever we migrated to superior SVN. Now because of technical limitations of continuous integration servers and VCS we should treat CI server as a very expensive über-assertion that should never fail. Technology deficiencies seem to affect our productivity and workflow.

I want to break the build whenever I want! When I’m done and my new tests pass, I just want to commit/push my new stuff and let CI server perform full testing. I hope everything flies but if not, I don’t want to feel guilty. I don’t want to stay late or apologize my team-mates. I will fix these unexpected problems when I can. It’s not production, it’s just my experimental new feature failure that no one cares.

This naturally leads to an idea of feature branches. The concept is simple: you develop your feature on a separate branch, CI server might even build all your changes and when you feel you are ready, you simply merge your changes back to mainline. The problem with feature branches is that it’s no longer continuous integration. After days of development your feature might be green and ready alone, but merging it back might be extremely problematic. Also other team members might benefit from your changes, even if they are not complete (but already green). All these observations led me to the following requirements:

I want to push my changes as often as I need

CI server should build my changes in isolation so that if they break, no one sees them or cares

if my changes are good, I want them to be automatically and immediately visible to others

I also want to see changes made by others as soon as possible

Fortunately modern CI servers (I’m using Bamboo as a reference) and version control systems (git here, Mercurial should work exactly the same) are capable of supporting the workflow I’ve dreamed of. The main requirement is that I want to push my changes as fast as possible without running all the tests and breaking master. The first step is to create a separate branch and commit to that branch. We should never commit to master. When I think I’m ready with my new feature I simply push that branch and move on. No running tests locally, no nervous monitoring of CI server. Just push and approach new challenges. This may lead to great savings in time if your test suite takes few minutes to run.

First things first, here is how you set up Bamboo. Under Plan Configuration and Branches select the following highlighted options:

Automatically manage branches will discover and build all new branches automatically. Branch Merging Enabled allows Bamboo to automatically merge new branches with master one way or another. In the Gatekeeper configuration we tick Push on option. Here is how it works: I make several commits to my feature branch. You can push them immediately or after some time:

Notice that my feature branch is placed on top of master. master branch is still green and my, possibly breaking, changes are isolated. Here comes the magic. I configured Bamboo to discover new branches and build them automatically:
What’s so special about that? Bamboo tells me that my changes are fine so I am free to integrate them into the mainline (master branch). Am I? No, Bamboo did it already! It built my changes, found them to be green and automatically merged them into `master so that others can see them. Merging was simple, it’s just fast-forward:

OK, was it really that interesting? After all we would get the same result by simply pushing directly to master… Well, but what happens if we are pushing breaking changes to unmodified mainline? This is the terrifying moment in most of the teams. I just pushed breaking changes and everyone is yelling at me. Fix. Fast. Y U NO RUN TEST? But not in this approach:

Here is where all this pain starts to pay off: feature branch might be broken, but master is untouched. No merging occurred. Only my very own, private branch is broken. Other developers are unaffected. If this build was green, my experimental changes would have been automatically merged to master and pushed. But I made a mistake and they remain hidden. No one cares, my team mates still see stable master. I can go for lunch or fix it tomorrow. No stress, no peer-pressure. When I get it right, Bamboo will automatically apply my fix.

This was all very easy as Bamboo could use fast-forwarding instead of ordinary merge. But what if we try pushing good changes to modified remote origin/master? Suppose we are working on our feature but in the meantime other developer pushed some changes to bugfix branch which happened to be correct so Bamboo decided to merge them immediately:

As you can see our feature branch was not yet pushed to main repository while bugfix branch is already integrated. How will Bamboo deal with feature branch being pushed? The behaviour is a bit more complex, but still manageable: Bamboo first checks out master (including bugfix branch already merged) and tries to merge changes from feature branch. If merge was successful (no conflicts), ordinary build is performed. If build is successful, merge results are pushed to master:

Notice the “[bamboo] Automated branch merge” commit made implicitly by Bamboo. As you can see this commit merges all my feature branch changes into master branch. This approach works but has several drawbacks:

after a while your master branch history might consist of barely Bamboo generated commits. I’d rather see ordinary commits there

For the reasons above it’s better to merge my feature branch first locally with master and push that. In this scenario you are almost guaranteed that remote merge on Bamboo will never fail (only fast forward), it’s predictable and you work on the latest master state. And BTW wondering what would happen if automatic merging on Bamboo fails?

Summary

This approach for working with version control brings best of both worlds: feature branches and continuous integration. Because each developer is working on a separate branch (or even repository!), broken commit won’t ever make it to master branch/mainline. On the other hand automatic merging will make sure our feature branch is always up-to-date and we won’t run into issues when trying to merge days worth of work. Moreover good commits are immediately visible to others while bad ones remain hidden.

Join Us

With 1,240,600 monthly unique visitors and over 500 authors we are placed among the top Java related sites around. Constantly being on the lookout for partners; we encourage you to join us. So If you have a blog with unique and interesting content then you should check out our JCG partners program. You can also be a guest writer for Java Code Geeks and hone your writing skills!

Disclaimer

All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners. Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries. Examples Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.