Branches considered harmful

“Don’t use branches”. Three words that are guaranteed to trigger reactions when I utter them during my continuous deployment talks. Three words that make for interesting discussions. This post contains some of the arguments I use during those discussions.

Continuous deployment and the master branch

A lot of teams are used to having some combination of feature, release and development branches, possibly using tools such as git-flow. At any given moment, depending on the size of the team, tens to hundreds of branches may be active in a single repository.

Full disclosure: I’m a continuous deployment advocate. And I feel that if your goal is to achieve true continuous deployment (or even continuous delivery), that’s a model that’s not going to work. Or at least, it’ll be a barrier to progress. I’ll explain why having only a master branch is the way to go.

The cons of using (feature) branches

Simply put, branches delay integration and break continuous integration. This can lead to many issues: merge conflicts, incompatible code and functionality, build failures, etc. Until everything has been merged to the master branch, it’s impossible to know how the combination of all the changes that a team is working on fit together. This issue is further compounded when it takes a while for branches to be merged.

For a nice description of the pitfalls of using (feature) branches when developing a large scale, complicated product (SQL Server), please read this post by Remus Rusanu.

… In a code bounce, someone tries to check in a huge batch. First they have integration conflicts, which require talking to various people on the team to know how to resolve them properly. Of course, while they are resolving, new changes are being checked in. So new conflicts appear. This cycle repeats for a while, until the team either catches up to all the conflicts or just asks the rest of the team for a general check-in freeze. Then the fun part begins. Getting a large batch through the continuous integration server, incremental deploy system, and real-time monitoring system almost never works on the first try. Thus the large batch gets reverted. While the problems are being fixed, more changes are being checked in. Unless we freeze the work of the whole team, this can go on for days. But if we do engage in a general check-in freeze, then we’re driving up the batch size of everyone else – which will lead to future episodes of code bouncing. In my experience, just one or two episodes are enough to cure anyone of their desire to work in large batches.

But, I hear you say, what if we always keep our feature branches small and short-lived, doesn’t that mitigate these issues? Unfortunately, no:

Not all features can be completed in a day or two;

Product may not want a certain feature live yet, or not in combination with other features;

Even a branch that lives for a day, or two, may contain a large number of commits.

Not to mention develop and release branches, which by definition are long-lived.

Or, put in other words (Dan Bodart):

Feature Branching is a poor man’s modular architecture, instead of building systems with the ability to easy swap in and out features at runtime/deploytime they couple themselves to the source control providing this mechanism through manual merging.

Feature toggles, switches, flags

Enter feature toggles. Essentially if statements, they provide that ability, an elegant way of turning certain features on or off, without having to change and redeploy the code. An application can determine the setting of a feature toggle using cookies, based on IP ranges, configuration settings, any number of sources. There are many libraries that implement (some form of) feature toggles and even companies offering them as a service.

Feature toggles allow us to make full use of continuous integration and deployment, spreading out a feature over multiple small commits. Once a feature is complete, the toggle can be switched on. Feature toggles enable continuous experimentation, iteration and improvement. Be conscious of their number though.

Pull requests and code reviews

A lot of teams use pull requests to do code reviews. One developer writes some code, submits a pull request and another developer reviews that code. Simultaneously, automated checks can run tests and verify whether the pull request can be merged with the main line. Useful functionality, but pull requests are essentially branches, so those are out too!

Code reviews are a very useful tool to maintain standards and quality. But where do they fit in the workflow if we can’t use pull requests and all our commits go the master branch (which is automatically deployed)?

The answer: pair programming!

Pair programming enables continuous, real-time code review. Mix experienced team members with less experienced team members and you get transfer of knowledge as well. Make sure pair partners regularly switch roles (driver, navigator).

Exceptions

Any good rule has a few exceptions. With master-only development, I can think of the following:

Work in progress
You & your pair programming partner started work on a new task right before the end of the day and the work is not yet ready to commit. For example, you have a red test but haven’t finished the implementing code yet. It’s probably wise to push that work on a “wip” branch so that the work, should you or your partner call in sick the next day, or your machine crashes, can be continued without delay. The work should of course be pushed to master as soon as possible, and then the branch should be deleted.

Throw-away prototypingIf you’re trying out a new library or tool and other people in your team need to see that work, then a prototype branch is acceptable. After the test is done, the branch should be removed. Incremental/evolutionary prototyping should of course happen on the master branch.

Closing arguments

I hope this post clarifies my views on (feature) branching, especially when viewed in the context of continuous deployment. Naturally, I’m interested in your feedback, so let me know what you think about using just the master branch!

Michiel Rook

Michiel Rook is a Java/PHP/Scala contractor and consultant from the Netherlands. He loves coaching teams to develop better software and implement continuous deployment. He is a co-founder of Make.io and a member of the Dutch Web Alliance. He is an Army reservist and enjoys music, cars, sports and movies.

4 thoughts on “Branches considered harmful”

I got again some new insights in why branching is evil 😉 Something I didn’t thought of until now is that beyond merge conflicts, more importantly lots of parallel branches can lead to incompatibilities in features which is even worse. Of course, if you have enough code coverage, this should be discovered by your tests. But then again, this is rework. And you still hope that your tests cover everything.

Regarding the code reviews. I too thought that if you do pair programming, code reviews are not needed anymore.
That was before I had any experience with teams performing pair programming.
Recently, I had finally the chance to work in a team that did full time pair programming (and really I would never want to go back to no pair programming). Although this team performed what Jez Humble calls pre-commit code reviews through pair programming, it still did code reviews at the end of a story. And it had value.
The code review was first performed by one other person, than evolved to being done by one other pair, and in the end it was done in mob style to limit the ping pong via the comments of the code reviewing tool. The team was optimising for short cycle times.
In my opinion code reviews are more about detecting bad naming, missing test cases and more importantly sharing knowledge. Design problems shouldn’t be detected during code reviews (then it is too late) and also not bugs (you have your tests for that).

How can you perform codes review in a Trunk Based Development context ? Well I see 2 options:
* either via very short lived branches and the pull-request model
* or post-commit on master. And yes, this means that potentially bad code quality goes into production. I see no problem as long as it does not introduce a bug. Normally bugs should be detected by your tests. If not, if your cycle time is very short their is still no problem as you can fix it very rapidly. And what about the bad code quality ? Well because of the code review, you can still fix it immediately.

For the exceptions: I understand them totally. But still.
If you work in very small increments, there is no need to push changes to the central repo at the end of the day. Should your machine crash, or you are sick, it is only a very small increment that is lost 😉 Anyone should be able to redo that.
Experiments should be throw away code. The point of experiments (spikes in agile terms) is to gain knowledge. Knowledge that you will use later to produce production code. So you still don’t want to keep that and thus you don’t need a branch 😉

One more thing I forgot to say regarding the Code Review:
As you I do think that it should be possible to eliminate the post-commit code review step and only rely on Pair Programming. Unfortunately I haven’t had any experience with teams working like that, which makes me wonder this can only work with very mature and disciplined teams. I am guessing here.

The way you’re describing it, code reviews become a two-stage rocket. The first stage is inline during the development process and is handled by pair programming. The second stage would be post-development and can either be handled by 1) pull requests or short-lived branches or 2) post-commit, perhaps as a mob-style code review or code reading. I’ve had experience with both variants. To me, the pull request model only works when there’s sufficient incentive to quickly handle requests and not let them linger too long (this would delay integration and increase the chances of a conflict). With a sufficiently mature / disciplined team, the occasional code or design reading could be beneficial to improve.

As for the exceptions, I totally agree. If you work in small increments, not much is lost. And indeed, experiments / spikes should be thrown away, a (temporary) branch can however be useful to show or share the experiment with other team members.