November 21, 2011

Branching and Merging Strategies

Earlier this year, I assisted a company in designing a simple source code branching and merging plan. But of course, I have learned some things since that time. I just returned from the Microsoft ALM Conference, and I want to offer some additional insight into branching and merging. (updated 11/25/2011 after further reflection to strengthen defense of the Basic Plan recommendation)

Be aware that the complete plan we developed was several pages with details on when to merge and why. Anyway, the guidance recommends that the trunk or Main branch be reserved for stabilizing the product. This means “no junk in the trunk.” Or more explicitly, “You should not check in changes directly to the trunk, but into a development branch that is later merged into the trunk after suitable testing.”

The team asked me “Why do we need the development branch? Can’t we just check in changes to the Main branch?” After all, that’s what they had been doing forever. The addition of a Release branch seemed like a good enough innovation to them. But they felt the Development branch added needless complexity.

Before I continue, let’s be clear what scenario I am addressing here: a team with one future development effort. There are a LOT of reasons why people want multiple dev branches: one per developer, one per feature. A compelling case is when we are developing multiple future releases that may be incompatible. It’s a painful thing to do but it happens. Anyway, we are not talking about any of those scenarios. They are cases when we know we want multiple dev branches. In this post, we are just talking about a simple case where a team needs one development branch. We are discussing whether to do that development in an isolated Development branch or the Main branch.

At the time our team was reviewing options, I did not feel strongly one way or the other. So I asked this question point-blank on the ALM blog and was given the response that the separation was what was recommended by the rangers. So I passed that on, but was left feeling unsatisfied with the answer because I did could not describe a scenario in which checking into MAIN directly really caused havoc. I had never seen that. Oh sure, I had seen cases where the build was broken for a short time. But that was rare, and it always got fixed quickly.

At the recent Microsoft ALM Summit 2011, I had a chance to put this question to some very talented programmers during a panel on Source Control Strategies, moderated by Keith Pleas. The panel included opinionated veteran developers Martin Woodward, Arlo Belshee, Jim Newkirk, and William Bartholomew, explaining why their strategies were “the best way”!

Their consensus opinion was that it was actually preferable to check in to Main. This put them into a spirited disagreement with Bill Heys, ALM Ranger, speaking for many other Rangers in defense of the recommendation to have a development branch to isolate changes from Main.

I spoke to Jim Newkirk afterwards to try to understand the core of the disagreement. He admitted that he was making some assumptions about the team doing the work. In other words, if you were doing TDD, and working in a highly collaborative environment, following all best agile practices, not taking on more than you can chew in one sprint, following best software engineering practices, and had automated unit tests, system tests, etc…, then anything you checked in should be safe.

In contrast with that modern approach, I have worked on teams where developers not only did not practice TDD, they wrote no unit tests at all. Collaboration was done after the software was delivered to the test team, not up front, refactoring was rare, and regression was common. So given that context, the additional friction caused by working in a Development branch is welcome if it gains us isolation to ensure that the software is tested and in good working order at the end of the sprint, before it is merged into the Main branch.

But this also explains why a team led by Newkirk or Woodward would find the additional step wasteful and inefficient.

One of those variable I mentioned was the tendency of some developers to bite off more than they can chew in a sprint. If they are working on a long feature, and they check in intermediate code that leaves the system broken, then this goes against our desire to have potentially shipable code at the end of a sprint. Again, this style of development is avoidable in many cases. In fact, if I know I must do a really scary refactoring or change, I may proactively create a feature branch just for that work. But sometimes you don’t know in advance that you will make a mess. And some architectures are more prone to this problem than others. Whenever I find I have created a mess like this, I simply undo all pending changes and start over, with a new refactoring strategy, informed by my recent pain. But if your team has a history of making messes like this, then a development branch gives you a chance to isolate and repair the break prior to the end of sprint delivery. But still there is no magic here. If you don’t fix the mess, you must either cherry-pick the feature sets to merge (NOT RECOMMENDED) or merge nothing.

Another variable that influences whether to have development branches is the requirement to review the code before accepting it. If you have a development branch, you can grant merge rights to only senior developers and architects, and expect them to review the code before merging it.

A counter argument I heard at the conference was that to use a branch to isolate and filter software that is poorly produced is to avoid the underlying issues. If you have a team that is not writing unit tests and not collaborating with other team members before they code, then adding a separate branch and additional time to clean up the mess before each deployment is a sad solution.

So now, based on the opinions from all these sources, I have concluded that whether to use a single development branch, or whether to develop in the Main branch, depends on the context. At least I now have some objective factors to use in evaluating the risks. To recap, some variables to investigate include

historic facts of team performance like defect counts and regression.

regulatory requirements for gated review and approvals

risk of failure (not all application errors lead to bankruptcy or death) – The higher the risk, the more value in isolation.

If you think of any other items to add to the list, please comment. If a team have shown they can work safely without a Dev branch, and regulations or risk do not compell it, then why not work in the most efficient way?

However, as a last defense of the dev branch, I will say that having to merge a single dev branch into a quiet main branch should be fairly trivial. Remember that a merge tool like TFS 2010, will not alert you to every change. Only to items changed in both places. If you have developed only in Dev, it will have no trouble merging your changes silently. Thus, even a thousand source control changes may get silently merged in less than a minute. So why not use the dev branch as insurance against a new developer who makes a royal mess of the source code.

In git the law “don’t merge to trunk” is achieved with Dictator and Lieutenants Workflow or other models (available because to the distributed nature of git)

Developers *cannot* merge to trunk, because the trunk is in a repository out of their control. Yet, they can merge to their trunk and be sure that Dictator won’t see any conflict when (eventually) merging their job to the Official Trunk.

TFS can’t allow this.
Seriously: suggesting the use of one dev branch is just a poor workaround. The real solution is managing the complexity through a different workflow. Unfortunately, TFS has only one Workflow: centralized and trunk-based. Take a look to http://progit.org/book/ch5-1.html to have an idea of other possibilities.

(by the way: using git, or mercurial, it’s common to have 30 or more branches per day, per developer. I know you would cry “Hey, it would produce a hell!”. You are right. With TFS it would be a nightmare. Not with DVCS. Just try.)

Thanks for sharing the image. I’m not familiar with git. And the diagram is very interesting. Can you please interpret a little of the diagram so I can understand your scenario? Do we start at the top and read downward? I’m guessing that when the lines split we have a branch. And when they join together, a merge. But if that is correct, then the comments don’t make sense. They never say “create branch” or something similar. And does color have any significance?

If there is a link to the documentation on this git too that illustrates the branching, I would enjoy reading more about it.
Thanks

* Joining lines are merges. You can have octopus merges as well. (http://img.skitch.com/20090101-6s52spepjx7qgjaj3yuscuasa.png)
(for example, imagine your developers have developed 120 micro-features; each feature a branch – [in the meanwhile, imagine a TFS repo with 120 branches]. Preparing to deploy, you select which feature to include in the release, and you do a mega-merge including 80 of those 120 features [again, imagine TFS merging 80 branches] [and yes: you can have a workflow that grants you to have 0 conflicts merging 80 branches]

* separating lines are branches. In git you automatically branch if diverging. You checkout latest version. I checkout. You commit. I commit. TFS would stop me (Hey, you have to update and merge!). git would simply diverge and produce a new branch.

To clarify a factual error you made, TFS does not limit you to one dev branch. You can have an unlimited number. And they can be branched off the main trunk or off of each other, in whatever complex pattern you like. I am not trying to say TFS is better than GIT or comparable to a DVCS. They are certainly different, and I’m not qualified to compare them. But I want to clarify factual inaccuracies before they spread because they will make it difficult to learn and compare the tools.

I am not familiar with the DVCS way of working. You seem to feel strongly that it is the way to go. I only know the TFS branching model. And for the situation we had, it was extremely simple to understand and use, and it did not introduce any friction or impede development at all. So it seems like a good choice for the client.

If we were to use GIT, then I’m sure it will all be different. But we were not, and I did not observe any friction with this model, so I see no reason to recommend they replace their version control system and learn a new paradigm since this one is working well for them.

I’m using TFS, git and mercurial.
My company is migrating from TFS to mercurial. I once used CVS, SourceSafe and SVN.
I used each of them pretty intensively.

What I notice is most TFS users

1. Never tried other VCS
2. Are happy with TFS and see no need to thing to another workflow

I just wonder: are those 2 point dependent?

I also noticed

1. Who used TFS *AND* some other VCS feels TFS as a pain in the ass

Hence, it’s hard to summarize “What’s wrong with TFS” if you never looked out of your window. All I could suggest you is: pay attention, while you’re using a single tool the world is changing, and it’s changing so fast. TFS is a very old tool: imagine you are trying to explain to a VB developer why c# and .net are so cool. You would talk about classes, refactoring, reflection, namespaces. What would you reply when that developer will say you: “Never heard of classes, refactoring, reflection and namespaces; but I’m very happy with VB, actually it fit my Company needs”

Naaaaaa, TFS fits your company needs because you are forcing you company needs to fit to TFS limitation. Worst, you are constraining you team workflow to a very linear and poor model.

I can’t summarize in few words why DVCS would change your life. Just put your head out of the windows.
Being in you, I would start withhttp://progit.org/book/

Trying to summarize, here’s why TFS is a pain in the ass (disclaimer: just like the VB developer and the .NET developer, I would expect you will reply to each point “Hey, yes, it’s true, but what’s the big deal? Actually, I like TFS has that characteristic!”

1. TFS tracks single files, not a snapshot of the project
2. Branching is a heavy operation (in git it’s just adding a label)
3. When you merge TFS can’t track the operation
4. There’s no 3-way merging tool (http://kdiff3.sourceforge.net/doc/screenshot_merge.png) (By the way: try KDiff3 and integrate it in VS, it’s amazing)
5. It’s terribly slow
6. There’s no way to share code with another developer without checkin in to trunk
8. If Developer B and Developer B are working and committing their commits will be interleaved. TFS doesn’t give any options to separate them. Just imagine you’d need to accept Dev B’s commits and discard Dev A’s ones. Just impossible, one they committed, uh? Well, it’s an ordinary, simple operation with git.
9. There’s a central repository
10. You can’t have 2 (working and communicating) copies of a TFS checkout on the same computer
11. You can’t commit or branching while offline

David, I think one other thing to add to the consideration of whether to use branching or just work from the trunk is the capability of TFS to retroactively branch. So if you are working on only one feature and even if your code is a big mess, you can go back and find the label of your last release and branch from that. I think that effectively removes the need for a single development branch. The story changes as soon as you have multiple lines of development.

The team I’m on currently is supposed to create a branch for every feature but in practice we usually end up just running in one branch and treating it like a dev branch. Usually things seem to devolve like that I think because it just isn’t worth the hassle. Even with a DVCS I think you end up with “branching noise” that adds little value. I think the strength of DVCS comes in when you really are in a distributed environment (e.g. working on an open source project) or with distributed teams. For your standard single collocated development team, I’m not sure it adds much value.

Regarding your current team, when you say you are “supposed” to create a branch per feature, do you mean that is a policy set by someone? I agree with your actual practice that the fewer branches the better. In this particular case, I consider the rule wrong, and your team’s practice superior. In fact, my driving principle is to reduce the delay in integration which means working in one shared branch (whether it’s a dev branch or the main trunk). So why would they establish a “preferrred” branching policy that goes against that? I would be interested in the reasoning.

Regarding your point on the value of DVCS, I agree with your description of the context in which DVCS makes more sense and when a Centralized Version Control System makes sense.