Moving repositories managed by GitLab

Sometimes you need to move all repositories managed by GitLab to
another filesystem or another server. In this document we will look
at some of the ways you can copy all your repositories from
/var/opt/gitlab/git-data/repositories to /mnt/gitlab/repositories.

We will look at three scenarios: the target directory is empty, the
target directory contains an outdated copy of the repositories, and
how to deal with thousands of repositories.

Each of the approaches we list can/will overwrite data in the
target directory /mnt/gitlab/repositories. Do not mix up the
source and the target.

Target directory is empty: use a tar pipe

If the target directory /mnt/gitlab/repositories is empty the
simplest thing to do is to use a tar pipe. This method has low
overhead and tar is almost always already installed on your system.
However, it is not possible to resume an interrupted tar pipe: if
that happens then all data must be copied again.

If you want to compress the data before it goes over the network
(which will cost you CPU cycles) you can replace ssh with ssh -C.

The target directory contains an outdated copy of the repositories: use rsync

If the target directory already contains a partial / outdated copy
of the repositories it may be wasteful to copy all the data again
with tar. In this scenario it is better to use rsync. This utility
is either already installed on your system or easily installable
via apt, yum etc.

Thousands of Git repositories: use one rsync per repository

Every time you start an rsync job it has to inspect all files in
the source directory, all files in the target directory, and then
decide what files to copy or not. If the source or target directory
has many contents this startup phase of rsync can become a burden
for your GitLab server. In cases like this you can make rsync's
life easier by dividing its work in smaller pieces, and sync one
repository at a time.

In addition to rsync we will use GNU
Parallel. This utility is
not included in GitLab so you need to install it yourself with apt
or yum. Also note that the GitLab scripts we used below were added
in GitLab 8.1.

** This process does not clean up repositories at the target location that no
longer exist at the source. ** If you start using your GitLab instance with
/mnt/gitlab/repositories, you need to run gitlab-rake gitlab:cleanup:repos
after switching to the new repository storage directory.

Parallel rsync for all repositories known to GitLab

This will sync repositories with 10 rsync processes at a time. We keep
track of progress so that the transfer can be restarted if necessary.

First we create a new directory, owned by 'git', to hold transfer
logs. We assume the directory is empty before we start the transfer
procedure, and that we are the only ones writing files in it.

Now we can start the transfer. The command below is idempotent, and
the number of jobs done by GNU Parallel should converge to zero. If it
does not some repositories listed in all-repos-1234.txt may have been
deleted/renamed before they could be copied.

Parallel rsync only for repositories with recent activity

Suppose you have already done one sync that started after 2015-10-1 12:00 UTC.
Then you might only want to sync repositories that were changed via GitLab
after that time. You can use the 'SINCE' variable to tell 'rake
gitlab:list_repos' to only print repositories with recent activity.