Featured in DevOps

Adin Scannell talks about gVisor - a container runtime that implements the Linux kernel API in userspace using Go. He talks about the architectural challenges associated with userspace kernels, the positive and negative experiences with Go as an implementation language, and finally, how to ensure API coverage and compatibility.

More Feature Branching Means Less Continuous Integration

Many teams now implicitly discard continuous integration due to ever-easier feature branching and an under-appreciation of trunk based development, says Steve Smith. At the Agile Tour London 2015 he will talk about the death of continuous integration.

InfoQ interviewed Steve Smith about different branching approaches and how they can be combined with continuous integration, and how using build feature branching can hamper continuous integration and continuous delivery.

InfoQ: Can you briefly describe some of the different branching approaches that are being used?

Steve Smith: Sure. Over the past year I’ve written a series of articles on some very different version control strategies, all of which I’ve used at some point in my career. I wanted to assess their historical context, establish a taxonomy for different styles of Feature Branching, assess their compatibility with Continuous Integration - and answer a question from a colleague: "why would I do Trunk Based Development when I’m doing Build Feature Branching so well?"

In the 1990s old school, long-lived Release Feature Branching was popular with tools such as ClearCase, Perforce, and MKS. Developers worked on shared feature branches for weeks, months, or maybe even years with entire epics developed and tested on branches before being released into production. Afterwards there would be a painful, time-consuming merge into Trunk plus regression test. I did Release Feature Branching for 2 years at a company using MKS, and I do not recommend it.

From the early 2000s onwards Trunk Based Development has been used with tools such as Subversion and Perforce. Developers work on Trunk with multiple commits a day in small, incremental steps. Testing happens on Trunk, and production releases either happen from Trunk or short-lived release branches. Parallel feature development is accomplished using techniques such as Feature Toggles and Branch By Abstraction backed by user preferences, runtime configuration, etc. I did Trunk Based Development for 7 years at 3 companies using Subversion, Mercurial, and Git and I highly recommend it to everyone - colleagues, family, friends, strangers in the street.

In the mid-2000s Distributed Version Control Systems (DVCS) became mainstream, and while Trunk Based Development is equally applicable to VCS and DVCS the lower cost of branch creation in DVCS led to more lightweight Feature Branching strategies emerging. One variant called Integration Feature Branching became used with tools like Git and Mercurial. Developers work on individual feature branches for ideally a day before merging to a long-lived Integration branch for testing, after which point the Integration branch would be merged into Trunk or in the case of Git Flow feature release branches prior to Trunk. I did Integration Feature Branching for 6 months using Git Flow, and I do not recommend it at all.

Finally, since the late 2000s Build Feature Branching has burst onto the scene thanks to Git, Mercurial, and particularly GitHub Flow. Developers work on individual feature branches for ideally a day, with features then merged into Trunk for testing and production release. I’ve been doing Build Feature Branching for 1 year at a client using GitHub Flow, and I would recommend it - albeit it very cautiously.

InfoQ: Can you elaborate how these different branching approaches can be combined with continuous integration?

Steve Smith: Firstly, Continuous Integration is not a tool - it is a practice where every member of a team commits to Trunk at least once a day, which might be verified by a build server such as Jenkins or TeamCity. Therefore we can assess a version control strategy based on its ability to facilitate daily commits to Trunk (call Master if you want, it doesn’t matter). Firstly, Release Feature Branching and Integration Feature Branching are both clearly incompatible with Continuous Integration as the former involves months on a shared branch and the latter involves a plethora of branches prior to Trunk.

Trunk Based Development is synonymous with Continuous Integration, and for good reason - when every team member is committing to Trunk multiple times a day Continuous Integration is achieved. There are also other benefits to explore, such as team members being nudged towards decomposing the codebase into a smaller, modular Evolutionary Architecture and Feature Toggles being used for Canary Releases and Dark Launching.

Build Feature Branching is really interesting. On the face of it individual feature branches being reviewed and merged into Trunk seems compatible with Continuous Integration, but after much thought I’ve concluded that Build Feature Branching is probably incompatible with Continuous Integration - feature branches end up being longer than a day due to developers picking up more changes, reviews take longer than a day due to developer workload, and builds become slower and more broken as a result of developers working asynchronously from the Trunk merge and build.

InfoQ: At the Agile Tour London you will talk about the "Death of Continuous Integration". Why death?

Steve Smith: During the #IsTDDDead debate in 2014, I looked around at my team and saw everyone practising Test Driven Development using GitHub Flow and I thought "Test Driven Development is alive and well, but Continuous Integration is in real trouble".

Build Feature Branching is phenomenally popular now, and I would guess that is due to a) Git and GitHub being such good tools and b) companies increasingly turning to the open-source community for ideas and inspiration. In the Stack Overflow 2015 survey 11,519 of 16,694 (69%) of developers reported Git as their source control tool, and GitHub Press in July 2015 reported 10 million users and 26 million repositories. Now, Git is a very nice tool, and so is GitHub. But Continuous Integration is phenomenally important - it is the foundation of Continuous Delivery - and I believe a lot of teams will struggle to implement a long term, sustainable Continuous Delivery programme if they blindly adopt Build Feature Branching.

InfoQ: Can you share some resources that InfoQ readers can use to learn more about branching and continuous integration?

Need another name

Your message is awaiting moderation. Thank you for participating in the discussion.

Vendors and teams often silently redefine Continuous Integration to mean something different in context of feature branch workflow. Something like automated builds or pre-merge validations, etc.I prefer using another name for this usage pattern, e.g. "Jenkins" instead of "CI server".

Re: Good overview, but strange conclusions

Your message is awaiting moderation. Thank you for participating in the discussion.

Hi Thomas

I went into some detail above, some more in my linked blog posts, and even more in my talk - but in short...

Continuous Integration means *every* member of a team committing to master *at least* once a day. If that can be accomplished with a version control strategy that involves feature branching, all well and good. However, I believe it is highly unlikely that will be the case for the majority of teams.

Continuous Integration is orthogonal to Jenkins, TeamCity... they are all just build servers. Yes, you can build branches automatically. Yes, you can merge automatically.... but they all just tools. The important thing is the practice of Continuous Integration, and the use of tools that encourage rather than discourage the practice.

Re: Good overview, but strange conclusions

Your message is awaiting moderation. Thank you for participating in the discussion.

How about practicing merging the trunk/master branch to feature branch on daily basis and doing the CI on every feature branch. Once successful, merge it to master and then run the CI and other automates component testing.

Re: Promiscuous Merging and Continuous Build

Your message is awaiting moderation. Thank you for participating in the discussion.

Hi Amardeep

You are talking about 2 separate things:

1. merging from trunk to a branch on a regular basis is known as Promiscuous Merging and is a good idea2. building branches on a build server is known as Continuous Build or Continuous Isolation, and is an OK idea

Re: Good overview, but strange conclusions

Your message is awaiting moderation. Thank you for participating in the discussion.

@Thomas: I think the conflict comes from the tension that if you have branches, the point of them is to separate some commits from all the others. Whereas for continuous integration, the point is to combine everybody's commits together.

If a developer is going to push changes from their branch into an master or integration branch, then first they will have to pull all changes from the integration/master branch. If they are doing this regularly, then what purpose is the branch serving? Isn't it functionally identical to pulling from then pushing to github master?

Re: Good overview, but strange conclusions

Your message is awaiting moderation. Thank you for participating in the discussion.

@Jonathan: A branch can be very practical, and it doesn't have to lead to more seldom integration with the central master. The purpose it serves is to enable the developers being able to commit (and push) as early as they like, and to group a set of commits as belonging together, annotated by the merge-commit that tie them into master.

Re: Good overview, but strange conclusions

Your message is awaiting moderation. Thank you for participating in the discussion.

@Thomas. Thanks for the reply.

I don't understand yet how a branch "enables developers to commit (and push) as early a they like". If I don't use a branch, then I can still do that. Sorry if I'm being dumb here, I'm not deliberately trying to misinterpret.

The idea that a branch can be useful because the resulting merge commit might present useful information is a new one to me. I haven't seen this used, in practice. I'll have to ponder on it a little.

I guess my position is that if the branch *does* lead to more seldom integration, then it's causing a problem. And if it *doesn't*, then you are pulling and pushing from integration all the time, so why can't you just do that directly, without the branch as an intermediary.

Re: Good overview, but strange conclusions

Your message is awaiting moderation. Thank you for participating in the discussion.

@Jonathan: I had another epiphany today about this. Perhaps it conveys the point I'm trying to make better.

We agree that periodically we want to integrate our changes, and it is good to keep this period short. At some point while coding, we think "now I'm happy with my work, and I want to integrate it", and historically in SVN-times, that would mean creating a single commit of all the local changes and pushing it.

Now these days, Git gives us a bit more flexibility. First of all, we can make multiple commits before we integrate, and this is nice because we can comfortably split our refactorings from functional changes in separate commits, and generally maintain a higher level of commit hygiene (a commit should change one thing, good commit messages, etc). It gives us more chances to review our commits before integrating them.

Now, I still haven't said anything about branches, but when you think about it, every time we integrate, it is merging our local branch with the remote one. Now you could argue that I'm just being technical, that's not the kind of branches you're talking about here. But if you were not working by yourself on these commits, but perhaps remote pairing with another developer, you could have interactions of commits being suggested, ammended and discussed between you before you choose to integrate it with the master branch, thereby committing it to the permanent history of the project.

I feel I'm not explaining this as clearly as it could be done (it is a bit late in the eve), but trying to reword it: The history that does get integrated into the main branch should be of a higher quality than the first attempted commits that are made on local and remote work-in-progress branches. Shying away from the use of such branches can lead to premature integration, leading to poor commit hygiene and history.

It also just occurred to me, that avoiding branches can reduce the tendency to share and collaborate on code before it gets integrated. If I work without branches, always pushing to master, my unit of work is the sum of local changes I have on my workstation, and the only collaboration on that will have to be done pre-commit, typically by pair programming.

And just to bring it into continuous integration, or rather the continuous building/testing part of it - it's possibly that anyone working with a single branch/early integration policy are likely to have their code tested later and less than those working with branches, as their work will fester for a longer period of time in the local changes before they deem it ready to be pushed (commit anxiety).

Re: Good overview, but strange conclusions

Re: Good overview, but strange conclusions

Your message is awaiting moderation. Thank you for participating in the discussion.

Hi Jonathan and Thomas

(InfoQ truncated my comment edit so I am reposting)

I have to say - unsurprisingly - that I largely agree with Jonathan and disagree with Thomas. I would like to pick up on a couple of Thomas' points.

Re "we agree that periodically we want to integrate our changes" - we do not want to periodically integrate, we want to integrate as often as possible. Mike Roberts once said "continuous means more often than you think" and I strongly agree with that sentiment. The more often everyone commits to trunk, the smaller the batch size and that unlocks many benefits.

Re "this is nice because we can comfortably split our refactorings from functional changes in separate commits, and generally maintain a higher level of commit hygiene" - commit hygiene is important, but with Trunk Based Development you *have* to have good commit hygiene. Just as Trunk Based Development encourages people to collaborate, to communicate, and to design their work well it also encourages small commits and good commit messages.

Re "shying away from the use of such branches can lead to premature integration, leading to poor commit hygiene and history" - as outlined above I disagree with that. During 7 years of Trunk Based Development at 3 different companies I did not see that. I think poor commit hygiene is more to do with bad developer practices, and I would argue working on a branch can facilitate such practies.

Re "avoiding branches can reduce the tendency to share and collaborate on code before it gets integrated. If I work without branches, always pushing to master, my unit of work is the sum of local changes I have on my workstation, and the only collaboration on that will have to be done pre-commit, typically by pair programming" - we must have a different definition of sharing and collaboration! As I outlined in the original blog series, the above interview, and in a talk I'm doing at the moment I do not believe branching encourages collaboration. It *can* happen with Build Feature Branching e.g. GitHub Flow but it is optional, whereas with Trunk Based Development it is mandatory. I would also say that pre-commit collaboration is preferential to any kind of post-commit collaboration as it avoids rework. n developers on n branches will diverge in n directions over time, due to Conway's Law

Re: Good overview, but strange conclusions

Your message is awaiting moderation. Thank you for participating in the discussion.

Hey @Thomas.

Thanks for taking the time to lay out your thoughts so carefully. I've read it very slowly and carefully, and plan to do so once again later today over a cup of tea, to make sure I'm absorbing what you're trying to say. (Always be open to being shown that I'm wrong, is my motto. :-)

I can see that we share a lot of high-level goals, such as improving collaboration, and high standards of commit hygiene.

However, I don't agree that the best way to achieve high levels of hygiene is to commit locally, then review and fix up before merging. I think it's to make sure the initial commits are correct up-front, using TDD and (yep, as you predicted) pairing.

In your opening para, where you say "At some point while coding" - my opinion is that optimally, this point is as early as possible, i.e. after every commit. This makes the content of every developer's branch end up being identical to the content that would be on master, if every developer was working from it.

I don't agree that branches in general increase the tendency for developers to collaborate. While branches do make it *possible* for developers to see and review each other's work before it gets merged, on the flipside, they also make it *possible* for developers to create things in isolation from each other - obviously this is in fact the whole point of a branch in the first place.

Meanwhile, without branches, every developer is *forced* to merge with (and pass tests with) every other developer's changes. If dev A creates a new utility function, then dev B can start using it (and talking about it, fixing it, improving it) as soon as it is committed. IMHO, this actually creates and enhances opportunities for communication and collaboration.

Re: Good overview, but strange conclusions

Your message is awaiting moderation. Thank you for participating in the discussion.

> The more often everyone commits to trunk, the smaller the batch size and that unlocks many benefits.

I agree on this, but we obviously we should avoid breaking the build, and polluting the history of the project with unclear commits, and series of commits. How many times haven't we all pushed to master, only to realize that we forgot to squash a commit, or left a misleading note in the commit message? Have you ever been annoyed to track the development of one feature across 5 commits intermingled with everything else that was going on that the same time?

> .. with Trunk Based Development you *have* to have good commit hygiene.

I would say you *need* to, and I would also argue that trunk-based development with no option to work with feature branches makes it harder to achieve this.

> I would also say that pre-commit collaboration is preferential to any kind of post-commit collaboration as it avoids rework

I think this will culminate in a religious discussion of those who prefer focusing on review *before* commits are integrated, and those who do not. Our personal anecdotes will add little to something so culturally and context specific as this. I think my points will resonate heavier with the distributed teams in the "age of GitHub" and open source projects with very high quality thresholds (Git, Chromium, kernel, etc), while yours will do so among co-located, agile teams and startups. My points should shine in critical libraries and products used by many, while yours will be more popular in intensive projects with business deadlines. Many companies deal with multiple sets of these factors, so they should figure out when to pick which strategy.

I can only speculate that over time, organization will move to having smaller and smaller modules maintained by fewer people (microservices, anyone?), with patches being contributed by a much larger set of either colleagues or even open-source contributors. We see this a lot already, perhaps especially in the Node ecosystem, where nearly all of the runtime code is open-source modules, with just a little being actual application specific. In this future world, I can imagine more review, more code reading - and using - than writing. And with it, the contribution threshold will gradually grow higher, making most commits mere suggestions that may or may not be integrated to the master branch, at the maintainer(s) discretion.

Just to add an anecdote of my own: I actually do work in my day-job with a mostly trunk-based strategy, and this works pretty good for our kind of team, in our particular business. We did have one colleage leave a year back, and he went to work for a company, a small startup at the time, where they review all changes before merging to master (think Gerrit). He said since that he would never go back to the old pushing straight to master way, and insists that productivity and cross-team productivity is much stronger as a result of this branch-review-heavy process.

Re: Good overview, but strange conclusions

Your message is awaiting moderation. Thank you for participating in the discussion.

@Jonathan Please see my last reply to Steve as I think that I address some of your thoughts there as well. We agree on many points, up to the stance of whether code should be isolated from the master branch until it has been reviewed appropriately.

My argument is that your approach only allows isolation as long as the code remains uncommitted/pushed (and therefore must be reviewed pre-commit with pairing, etc), and that my approach *allows* review, while not necessarily increasing the period of integration if the situation allows it (i.e. simple, small changes can be integrated quickly without much review).

And as I wrote in my reply to Steve, it's hard to say which of these strategies will be put to best use by which teams, but I think the industry generally will move towards "my" side of the argument over time, as proficiency with tools like Git, Jenkins and Gerrit increases.

Re: Good overview, but strange conclusions

Your message is awaiting moderation. Thank you for participating in the discussion.

Hi Amardeep,

If I understand your suggestion then it would be an additional layer of "continuous" (daily) merge and build that you would like to add. Benefit would be that possible merge conflicts will pop up earlier. Is that why you would do this?

Also, how does this work when there are many feature branches? Conflicts between feature branches won't pop up until they are merged back to the main branch, right?

Re: How is developing on a trunk more productive?

Your message is awaiting moderation. Thank you for participating in the discussion.

Hi Ben,

I use the terms trunk, master, mainline or main branch interchangeably.

I have written about the advantages of working on, and reviewing branches before merging them to the main branch many years ago: blog.tfnico.com/2010/08/why-distributed-revisio... - at the time it was a bit of a SVN vs Git argument, but this part relates especially well to our discussion here:

You might argue that Subversion users can keeping their changes locally in the workspace, but there's a major constraint: They can't collaborate on their feature. This means missing out on a lot of co-operation, creative process, ideas, code-review and refactoring before the feature ends up in the mainline.

There probably are better explanations around the web though. I googled a bit but I couldn't find anything right now. I did find this nice guide on how to do it in practice here:

Re: How is developing on a trunk more productive?

Your message is awaiting moderation. Thank you for participating in the discussion.

Hi Thomas,

I'm unsure if you've answered my question, which might be because I didn't formulate it well enough.

I fully support that people work together in teams, which means that they will share stuff to get feedback fast and frequently. They can use code reviews, walkthroughs, pair programming and tools to improve the quality of their code.

What I don't understand is how this is related to a branching strategy. Whether you use feature branches or commit directly to the main branch in my opinion should not impact collaboration. I mean, if a programmers wants to have feedback, (s)he doesn't have to commit to get it? You can ask a team member to look at you code, work in pairs, etc, without checking in code.

And even if you would commit code before doing a code review (bit strange to commit before pair programming I guess), how does it make a difference if you commit to a feature branch or to a main branch?

Why not ask your colleague for feedback directly in stead of using a pull request

Your message is awaiting moderation. Thank you for participating in the discussion.

Hi Thomas,

Reading the blog below the second link triggers me:

"There are many tools that will help you with code reviewing, and I can't give you The Perfect Tool For The Job. What currently works well for many people is using the approach with Git where you send a pull request. Most Git-tools (such as Github and Stash) have excellent interfaces for going through the diff between a feature/bugfix branch and the main branch. They allow you to add comments with every line of code, so that the author of the code knows exactly what you're commenting on and is able to fix the right code easily. Similar tools also exist for many of the other version control systems out there."

This blog post suggests to use a tool (it mentions Git, but I guess it can be any tool) to ask for a code review. Maybe I'm old fashioned, but if I want feedback from somebody who's in the same room with me, I'll walk over and ask him.

Of course it will be different when working in remote teams, but even then I would expect that there should be easier ways to ask for feedback than using a CM tool.

Re: How is developing on a trunk more productive?

Your message is awaiting moderation. Thank you for participating in the discussion.

Hi Ben,

Whether you use feature branches or commit directly to the main branch in my opinion should not impact collaboration. I mean, if a programmers wants to have feedback, (s)he doesn't have to commit to get it? You can ask a team member to look at you code, work in pairs, etc, without checking in code.

Yes, for certain kinds of feedback you have to push somewhere. Examples: Review from a remote team member. An automated build-system that analyses & tests your code. An automated job that deploys your commit as a docker container, letting you play out how it would run and act in a production-like environment.

And even if you would commit code before doing a code review (bit strange to commit before pair programming I guess), how does it make a difference if you commit to a feature branch or to a main branch?

I see committing to the main branch as chiseling something into stone. It cannot be undone, and we will always have to bear the weight of each commit made while reading the code log. I might be in the minority among the commenters who see a commit log as an important part of the development inventory, where we should try to keep waste (e.g. unused code & reverted commits) to a minimum.

In addition to the indirect pains of having noise in the log, there's also the added cost of friction on the team. If it turns out that you push to the main branch prematurely, you may break the build, or worse: break some runtime feature without anyone noticing it before much later. This could cost hours of your and your colleagues' time, as well as increase the risk of bugs leaking into production.

Re: How is developing on a trunk more productive?

Your message is awaiting moderation. Thank you for participating in the discussion.

Hi Thomas,

Thanks for your extensive answer. This is a useful discussion, much appreciated!

I agree that when working remote tools can support the communication, trigger people that you want feedback, and help to work time and place dispersed. When teams are collocated I prefer that people walk over and communicate directly as much as possible, but in the end it is op to people to find out what is most effective.

Once code is committed you can run builds which can give an extra check.

Re: Why not ask your colleague for feedback directly in stead of using a pu

Your message is awaiting moderation. Thank you for participating in the discussion.

This blog post suggests to use a tool (it mentions Git, but I guess it can be any tool) to ask for a code review. Maybe I'm old fashioned, but if I want feedback from somebody who's in the same room with me, I'll walk over and ask him.

Of course it will be different when working in remote teams, but even then I would expect that there should be easier ways to ask for feedback than using a CM tool.

What are your experiences with this?

My experience is that when you get feedback/review on your own workstation, there are some disadvantages:

You are incurring a cost on the colleague in question, interrupting and forcing a context-switch on her (i.e. pull-based, or asynchronous review would not destroy her concentration).

You have not formulated and created a commit (or commits) - so you're leaving that out of being reviewed.

You're increasing the chance for "works-on-my-machine!" mistakes by having it reviewed on your own environment.

You're browsing and reviewing your changes in *your* environment, with your editor/IDE preferences, screen setup, your keyboard & mouse control, etc. This makes the review very biased towards the way you navigate the code, giving the reviewer less control and power in the review.

Of course, there is the advantage of being there with them face to face, with the highest possible bandwidth of communication, and instant feedback. However, for some kinds of review and situations, the disadvantages above outweigh the advantage.

Re: How is developing on a trunk more productive?

Your message is awaiting moderation. Thank you for participating in the discussion.

So I think we are aligned on this, right?

Yeah, I think we both understand both approaches. You remind me of another advantage to do branch-based code review over pre-commit workstation review: you can have N reviewers on a pull-request, while a workstation review is usually maxed to 1-2 reviewers. This could make a difference on super-critical code changes.

Re: Why not ask your colleague for feedback directly in stead of using a pu

Your message is awaiting moderation. Thank you for participating in the discussion.

Hi Thomas,

Thanks for your extensive motivation.

My experiences are different. Although people can have different setups of their machines, screen, and even use different tools, I've seen people who adjusted to that quite easily and quickly. Sitting together at one workstation usually wasn't a problem.

I recognize the context switch problem, if people are in flow you want them to keep working. Solution would be to agree upon a time frame for review/feedback. It would be instant then, but can still be quick. Direct communication also enhances the quality of the feedback (as you also mentioned in your answer).

I don't understand what you mean with "You have not formulated and created a commit (or commits) - so you're leaving that out of being reviewed". What would a commit comprise, and what can be the benefit of reviewing it?

Re: Why not ask your colleague for feedback directly in stead of using a pu

Your message is awaiting moderation. Thank you for participating in the discussion.

What would a commit comprise, and what can be the benefit of reviewing it?

What files/changes the commits include, and the commit message. Does each commit change one thing and does the commit message explain it well? Is the commit message well-written and understandable? Stuff like that.

Re: How is developing on a trunk more productive?

Your message is awaiting moderation. Thank you for participating in the discussion.

Having multiple reviewers on a workstation can be difficult. I remember doing many code review meetings with one client where attendants printed the code and made remarks on paper. It sounds very inefficient but the group discussions often made it worthwhile to do it. It was also a great way to pull in and clarify coding guidelines when the code review revealed that they weren't understood.

You mentioned that "committing to the main branch as chiseling something into stone" which "cannot be undone. What about doing a roll back? "Shouldn't be to hard to do with modern version management tools.

Re: How is developing on a trunk more productive?

Your message is awaiting moderation. Thank you for participating in the discussion.

You mentioned that "committing to the main branch as chiseling something into stone" which "cannot be undone. What about doing a roll back? "Shouldn't be to hard to do with modern version management tools.

In Git, there are two ways of rolling back an already published change:

The first way is to do a `git revert`, meaning you create a new commit which does the exact opposite of the commit you wish to revert. If you are reverting a series of commits, you can revert them with a series of reverts. You can also revert merges, but this is a bit dangerous if you don't know what you're doing. As you can imagine, reverting this way can leave commits in your history of little value.

The other way is to reset, or delete history. Typically this is what one does when somebody has messed up bad, and pushed something which should never have been committed (i.e. their mp3 collection into the company code repo, or passwords or secret keyfiles). This requires you to push a rewrite to the central repository, which will lead to everyone who has already fetched the faulty commits into having to reset there as well. This is quite messy, as it usually involves the whole team having to do a little local Git wizardry before they can continue working.

As both approaches are quite ugly, it's best to avoid having to roll back commits in the first place IMHO.

But, if a incomplete or faulty commit is done on a feature branch I guess you have similar problems. There might be less people involved though, only the ones working on the feature. On the other hand, given that there are less people working on that branch, it may also take longer to discover it. Preventing such problems would be best, I guess we agree on that.