Darcs retrospective, and the future

GHC has been using darcs for version control since the beginning of 2006. It has not been all plain sailing, so in this page we will record our experiences with darcs, and attempt to objectively evaluate whether we would be better off with a different version control system. In the event that we do switch, we need to track exactly what needs to change, so this page will also list those dependencies.

Problems we currently experience with darcs

Conflicts and merging. This is the biggest problem we encounter, and is also the #1 priority for
Darcs development. Any non-trivial branch is affected, and essentially the workaround is to discard
the history from the branch when merging, and use ordinary diff/patch tools. Keeping history is
possible, but impractical for branches with more than a few patches.

Speed. many operations are impractical (annotate, darcs changes <file>), and many operations just take "too
long" (i.e. long enough that you go and do something else rather than wait for it to finish,
which incurs a context-switch cost). We can't use Trac's darcs integration or darcsweb, for example,
because both rely on invoking darcs changes <file> (for that matter, that's not completely true for the
​trac darcs plugin as it does not execute that command
on a per-file basis, but rather it loads and caches into its own database the result of darcs changes -v
on the "not-yet-loaded" changesets, visiting every patch in the repository just once.
It caches also the actual content of each file touched by any browsed changeset, to compute the unidiff.).

bugs: we run into darcs bugs other than the conflict/merging bug on a regular basis.

user interface issues: e.g. in a conflict there's no way to tell
which two patches are conflicting with each other(!)

Windows support: is quite flaky still. (well, it's certainly better than it used to be, and
at least some Windows users don't consider it to be bad).

The GHC developers have sufficient problems with Darcs that a change would be beneficial

We want to stick with distributed version control, and have a widely-used and well-supported system, so Mercurial and Git are the only real
contenders

Mercurial and Git and percived as being mostly feature-and-performance comparable, although git is more popular

More investigation of the Mercurial option for GHC is needed, especially in light of reported poor support for Windows with Git. This
work is ongoing

Important workflows

Cherry-picking patches

This is how we maintain the stable GHC branch. Particular fixes are pulled from the HEAD. When the desired patches don't depend on undesired patches, darcs takes care of this automatically, as demonstrated below. Otherwise, with darcs, the patch has to be merged by hand.

# Make a repo with a single file with lines 1,3,5,7,9 in
mkdir repo1
cd repo1
bzr init
printf 'Line1\nLine3\nLine5\nLine7\nLine9\n' > file
bzr add file
bzr commit -m patch1 --author [email protected]
# Now we fix a bug, and in the process add some debugging prints.
printf 'Line1\nFix2\nLine3\nDebug4\nLine5\nFix6\nLine7\nDebug8\nLine9\n' > file
# We want to record our fix, but not the debugging prints.
# Using the new interactive plugin: https://launchpad.net/bzr-interactive
#bzr commit -m the_fix -i
# .. and it won't let us cherry pick each line. It's all or nothing!
# bzr shelve is a better supported plugin that allows something similar to this
# (shelve the debug prints, commit what remains, unshelve the debug prints
# and revert them)
# but it also doesn't let us pick between the changes in this hunk. They are
# welded into one hunk because they are close together.
# Get rid of the debug prints
bzr revert
# Now file contains lines 1,2,3,5,6,7,9

amend-record

So, you make your lovely patch, it all looks good, so you record it. Then you do a build to make sure it works, and during the build or testsuite run you find that the patch wasn't quite right after all. You could just add a little 2-line patch, but that isn't very pleasant: It's nice if, as far as possible, all intermediate compiler states are buildable. Also, people might pull the first patch but not the second when cherry-picking, leading to head-scratching down the line. It's much nicer to be able to just amend-record the fix into your original patch.

The same is available for Git. The command is called git commit --amend. You usually checkout the commit you want to edit into a branch, do the changes, then rebase the remaining patches on top of this. Example coming soon...

I can't find a way to do this directly with Mercurial. You can of course do hg rollback and then add a new commit. The Mercurial Queues extension is also able to do this (hg qrefresh) but it is rather complicated to use.

It doesn't appear to be possible with Bzr either. You have to do bzr uncommit and then bzr commit, similarly to Mercurial.

File renames

We often want to cherry-pick a change where the file has been renamed on one branch or the other. This should work without any extra intervention from the user, and does under darcs.

Apparently git didn't realise that "file" had been renamed to "file1" in one branch, because its contents had also changed sufficiently. In fact, if you add enough other stuff to the file so that both versions are similar, then the merge works, which is deeply worrying.

This goes wrong with git version 1.5.2.5. I wouldn't be surprised if other versions work, but the underlying issue is that git doesn't store information about file and directory renames, and has to rely on heuristics to recover the information when necessary. Converting a darcs repo into a git repo is a lossy conversion - it discards information about renames.

The transplant command wiki page ​http://www.selenic.com/mercurial/wiki/index.cgi/TransplantExtension contains the text "Three-way merge doesn't cope particularly well with transplanted patches - it will tend to generate false conflicts",
which doesn't fill me with confidence. However, we only want to use transplant to maintain a branch (e.g. 6.8) which we won't merge back into the one we are pulling from (e.g. HEAD), so this may be a non-issue

Notes On Conversion

Currently using Tailor. Problems encountered:

Darcs outputs XML without an encoding header. Patched Tailor to append Latin-1 encoding to the XML output. This will be sent to the tailor author

In hg.py, replace the line self._hgCommand('tag', tag) with self._hgCommand('tag', tag, force=True) because we seem to be trying to apply duplicate tags at some points. Don't know quite how this is possible!

bisect support would require git modules to also pick the correct version of libraries. Keeping this in sync is not easy, atm.

uses its own protocol for network transmission (http works but is slower, however, other hosting services are available, e.g., github)

Darcs / Git Command Comparison

darcs whatsnew -s

git status

darcs whatsnew

git diff

darcs record

git add --patch (goes through all changes)/git add -i (starts with a file-based view) Git add only marks changes for commit. This can be nicer if you want to check some things first before you commit them.

git commit (do the actual commit)

darcs record -a -m foo

git commit -a -m foo

darcs pull

git pull then git cherry-pick/gitk + select patches using mouse. It's probably best to have one local branch correspond to the remote branch and then cherry-pick from that. You can also create local names for several remote repositories.

Revisions form a DAG (more like a tree with merge-points) rather than patchsets

Supports convenient "centralised-style" commit-remote-by-default as well as "distributed-style" commit-local-by-default. Just 'bind' or 'unbind' your branch whenever you want.

Simple clear UI

Has rebase

Disadvantages

Revisions form a DAG (more like a tree with merge-points) rather than patchsets (this is a subjective point, which is why it's in both lists. Which model do you believe in?)

Cherry-picking isn't very "native" to the data model. Support for this is very poor.

UI is rather different from darcs (which current contributors are used to).

Benchmarks

These benchmark figures were obtained with a warm disk cache on a clean tree, using OS X 10.5:

Annotate

Log

Status

Clone

Git

0.936s

0.523s

0.030s

0.580s

Hg

0.230s

3.772s

0.178s

9.455s

Bzr

2.131s

7.278s

0.312s

49.788s

Darcs

47.080s*

2.115s

0.053s

28.276

Footnotes:

darcs annotate fails with Stack space overflow: current size 8388608 bytes., so you don't get an answer

darcs get fails with Unapplicable patch due to the case-insensitivity of HFS+, so you don't get a clone

These figures were obtained with a warm disk cache on a clean tree, using a Windows XP Parallels virtual machine running under OS X 10.5:

Annotate

Log

Status

Clone

Git

1.191s

0.681s

0.180s

3.035s

Hg

0.420s

2.844s

0.221s

16.383s

Bzr

2.954s

7.731s

0.350s

62.190s

The Bzr clone time is high because it does an actual copy rather than just using hard links, by design (you can use the shared repository feature instead). However, even on the other commands it seems to be about twice as slow as Hg, which is on average somewhat slower than Git.

Note that this is a very limited benchmark: it doesn't even test merging / pulling or the cost of cloning over a network.

Eliminated alternatives

Darcs

#darcs: 39 members

Advantages to staying with darcs:

Community consistency: essentially the Haskell community has standardised on darcs, so it would be
an extra barrier for contributors if they had to learn another VC system.

Merging, when it works, is done right in darcs.

Disadvantages to staying with darcs:

Uncertain future: no critical mass of hackers/maintainers. The technical basis is not well enough
understood by enough people.

Reason for elimination: persistent performance and algorithmic problems, see above.

Dependencies on darcs

The following is intended to be a complete list of the things that would need to change if we were to switch away from darcs, in addition to the conversion of the repository itself, which I am assuming can be automatically converted using available tools.

The following code/scripts would need to be adapted or replaced:

The darcs-all script

The push-all script

The aclocal.m4 code that attempts to determine the source tree date

.darcs-boring

The buildbot scripts

checkin email script: /home/darcs/bin/commit-messages-split.sh

Trac integration (the GHC Trac does not currently integrate with darcs, however)