My personal blog

Bridge support in git for mercurial and bazaar

I’ve been involved in the git project for some time now, and through the years I’ve had to work with other DSCMs, like mercurial and monotone. Not really as a user, but investigating the inner workings of them, mostly for conversion purposes, and I’ve written tools that nobody used, like mtn2git, or pidgin-git-import, but eventually I decided; why not write something that everybody can use?

So, I present to you git-remote-hg, and git-remote-bzr. What do they do?

You can pull, and in fact push as well. Actually, you wouldn’t even notice that this remote is not a git repository. You can create and push new branches… everything.

You might be thinking that this code being so new might have many issues, and you might screw up a mercurial repository. While that is true to some extent, there’s already a project that tries to act as a bridge between mercurial and git: hg-git. This project has a long history, and a lot of people use it successfully. So, I took their test framework, cleaned it up, and used to test my git-remote-hg code. Eventually all the tests passed, and I cloned big mercurial repositories without any issue, and the result were exact replicas, and I know that because the SHA-1’s were the same 🙂

So you can even keep working on top of clones you have created with hg-git, and switch back and forth between git-remote-hg and hg-git as you wish. The advantage over hg-git, is that you don’t need to use the mercurial interface at all, you can use git 🙂

To enable the hg-git compatibility mode you would need this:

% git config --global remote-hg.hg-git-compat true

What about the others?

Surely I must not be the first one try something this cool, right? I don’t want to dwell into all the details why none of the other tools suited my needs, but let’s give a quick look:

This is a very nice tool, but only allows you to export stuff (fetch an hg repo), and you have to manually run the ‘git fast-import’ command, setup the marks, and so on. Also, it’s not very well maintained.

You need quite a lot of patches on top of git, so your vanilla version of git is not going to cut it. In addition to that it doesn’t support as many features; no tags, no bookmarks. Plus it fails all my extensive tests for git-remote-hg.

Nope, not really. At least not when using the hg-git compat mode, because all the extra information is saved in the commit messages.

Update: I forgot to mention the only bidirectionality problem: octopus merges. In git a merge can have any number of parent commits, but in mercurial only two are allowed. So you might have problem converting from a git repository to a mercurial one. It’s the only feature that hg-git has, that git-remote-hg doesn’t. As for bazaar, it also allows multiple parents, so it should work fine, but this hasn’t been thoroughly tested.

The only consideration is that in the case of mercurial branches are quite different from git branches, so you need to be really careful to get the behavior you want, which is why it’s best to just ignore mercurial branches, and use bookmarks instead. When you create git branches and push them to mercurial, bookmarks will be created on whatever branch is current.

Trying it out

Soon they might be merged into upstream git, so they would be distributed as other contrib scripts (e.g. /usr/share/git), and eventually possibly enabled by default. They were distributed as other contrib scripts, and they were going to be enabled by default, until the maintainer decided not to, for no reason at all.

It is interesting that the issue of hg->git has been so convoluted. I also found and used one
called git-hg-again. And it worked. Apparently github also has their own called gexport,gimport.
Bit I dunno, I got a headache now. I just wanted to fork a bitbucket repo on github, jeez.

The only thing that I found a bit annoying was that after pushing, “git status” says my repo is still ahead of the remote. This is despite the fact that the remote contains the changes I’ve pushed. I need to do a fetch in order to sync the two up. Pure git doesn’t have this behaviour. Perhaps “git push” could be changed to do “git push && git fetch”?

Felipe, your bridge doesn’t gracefully handle importing tags with spaces in the tag name. These are legal in bzr, but not in git, so it’s not really clear how to resolve this situation. Probably it’s best to ignore them with a warning but continue.

Currently the following happens when importing the the tag “Name with space”:
WARNING: TODO: fetch tag ‘Name’
error: refs/tags/Name does not point to a valid object!

Let me know if I’m doing something wrong here, but it seems there is a massive showstopper: you can’t seems to merge branches that originate from different remote and push the result back upstream: I’m getting a KeyError in blob_marks at https://github.com/felipec/git/blob/fc/remote/bzr/contrib/remote-helpers/git-remote-bzr#L537. Could this be caused by the marks files being separate per remote s.t. it can’t find any the revisions and blobs from the merged-in branch?

It looks like this has made it into the git tree, under `contrib/remote-helpers`. If installing from source, you’ll need to install it explicitly: it’s not enabled by default, nor is the `/usr/share/git/remote-helpers` directory created automatically (that’s probably a feature of whatever distro Simon’s using).

Felipe: I figured you’d just up and moved on, you were so quiet. The guys working on gitifyhg have been fairly active and responsive to bug reports. I’m grateful for the work you’ve done, but, all other things being equal, I’m going to go with the product that fixes the problems I have when I use it. That’s not a one-time decision, though, so if they don’t get things done then I’ll come back. Or fork both of your projects!

@FelipeC:
Thank you for the great code, and sorry for missusing the comment section for bug reporting. I didn’t know about the tracker (Sound like a lame excuse by now…).

As of the comment of Paul (10328) I now use gitifyhg. They give credit to you for using your code base. Is there a reason for two projects. Are you still active on your code? Can’t the two of you join to one project again?

Now to the error I reported earlier. I already posted how I worked around it of the gitifyh page [1] but here it is again

The repository I cloned lives directly in the home directory of its dedicated user, this is
/home/repoUser
Therefore, login into the machine using ssh droped me in the correct place for the repo, and hence, the URI I used to clone was

git clone gitifyhg::ssh//repoUserSSH

where repoUserSSH is the ssh shortcut (~./ssh/config) for the remoe machine. This was good enough for cloning, but the pull failed. Changing the clone to

A few variations such as just adding the remote then doing a separate fetch didn’t work either. I am able to clone the mercurial repo, however. Ultimately I’m trying to add a remote to an existing repo to then insert it as a subtree dependency. Suggestions as to how I could achieve this would be appreciated!

@Jakub Narebski The same way hg-git does it; by using Mercurial extras, and in Git adding the extras in the commit message. I started trying that with git notes, but that required changes in git fast-export, and you know how well received changes in Git are. Round-tripping only works properly in hg-git mode.

@FelipeC: By “extras in the commit message” do you mean extra commit headers (IIUC like Kiln Harmony) or at the end of commit message like git-svn?

I’m not sure if git notes would be good solution for converting Mercurial commit to Git one. It could be a good solution for annotating a commit which didn’t came the same on round-trip, to note changes that Mercurial brought. BTW what are problems that `git fast-export` have with git-notes? Are they not composed of ordinary git objects?

“And you know how well received changes in Git are.” — no, I don’t. I had a quite a long hiatus in contributing to Git and watching Git mailing list, and my return to it (e.g. reviewing gitweb patches) got interrupted by HDD failing 😦

BTW. I do wonder why Kiln Harmony uses IIRC base64 encoding to stuff multiline extra fields in Git extra headers instead of using gpginfo like multiline header (for merging in signed tags) that current Git has…

“extras in the commit message” do you mean extra commit headers (IIUC like Kiln Harmony) or at the end of commit message like git-svn?

I mean extra commit headers. Mercurial allows arbitrary key-values to be stored in an “extra” dictionary.

I’m not sure if git notes would be good solution for converting Mercurial commit to Git one. It could be a good solution for annotating a commit which didn’t came the same on round-trip, to note changes that Mercurial brought. BTW what are problems that `git fast-export` have with git-notes? Are they not composed of ordinary git objects?

We need some way to store the extra information, either notes or in the commit message.

The problem with fast-export is that we need that extra information before we create the Mercurial commit. So, for example; a file was explicitly renamed, we need that information as-is before creating the Mercurial commit. We can extract that from the commit message that fast-export passes, but it doesn’t pass the notes associated with that commit until much later.

“And you know how well received changes in Git are.” — no, I don’t. I had a quite a long hiatus in contributing to Git and watching Git mailing list, and my return to it (e.g. reviewing gitweb patches) got interrupted by HDD failing

Interesting, my external HDD where I had all my work also failed.

Nowadays Junio rejects patches without even looking at them. Check this patch series for example:

We need some way to store the extra information, either notes or in the commit message.

The problem with fast-export is that we need that extra information before we create the Mercurial commit. So, for example; a file was explicitly renamed, we need that information as-is before creating the Mercurial commit. We can extract that from the commit message that fast-export passes, but it doesn’t pass the notes associated with that commit until much later.

On the other hand at least in some cases importer would need notes related to commit after a commit, to perhaps rewrite them to point at changed during import commit they refer to. Well, that’s what I think, but I don’t know this area of code and those issues.

Nowadays Junio rejects patches without even looking at them. Check this patch series for example:

What I can see here is you wh^W complaining, but I don’t see any response from Junio. Which (the las of response) is perhaps a response in itself… though this might be because lack of tuits. “What’s cooking in git.git (Jun 2013, #02; Tue, 4)” includes fc/transport-helper-no-refspec (2013-05-21) (!)

I’ve used it for weeks now and never had many problems, today though many of my repos (>30) started acting up, same message as Marko Koivusalo gets:
fatal: mark :11632 not declared
fast-import: dumping crash report to .git/fast_import_crash_61709
fatal: Error while running fast-import

I didn’t update my Git version or anything else on my Mac, checking out an old version of the repo and running pull again doesn’t help, neither does deleting branches (including master) and checking out from origin/master again, I’ll update to the newest git-hg bridge and hope I can resolve the issue.

i didn’t do anything special to get into this state. on one of my machines, i did do an update to git 1.8.5.3 earlier today, and another machine that accesses the same local repository is still running 1.8.4.2. everything was working fine (fetches, merges, rebases, etc) until suddenly i started getting this error whenever i try to fetch.

I just pushed a branch to the mercurial remote but instead it created a bookmark which caused the default branch to have 2 heads which needed to be merged using hg directly.
Deleting the branch using git didn’t work, pushing to branches/mybranch also didn’t work. Is this not possible or how can I push a local branch to my remote mercurial repo?

Played around a little with a local mercurial repo and cloned it, the way it works is the following:
Create branch in the mercurial repo: hg branch myBranch
Commit at least once: hg ci -m “Created branch myBranch”
Pull in your git clone repo: git pull
Switch over to your branch: git checkout branches/myBranch

Important is the branches/ prefix and that you need to create the branch on the mercurial side, pushing and creating from git doesn’t work, also you can’t delete the branch.
For me it also didn’t work to push from a branch which was named differently than the remote branch like:
git push origin branchX:branches/myBranch