… interesting tidbits of release engineering.

Inter Repository Operations

[This is an experiment in publishing a doc piece by piece as blog entries.
Please refer to themain pagefor additional context.]

Mozilla, like most operations, has the Repositories of Record (RoR) set
to only allow “fast forward” updates when new code is landed. In order
to fast forward merge, the tip of the destination repository (RoR) must
be an ancestor of the commit being pushed from the source repository. In
the discussion below, it will be useful to say if a repository is
“ahead”, “behind”, or “equal” to another. These states are defined as:

If the tip of the two repositories are the same reference, then
the two repositories are said to be equal (‘e‘ in table
below)

Else if the tip of the upstream repository is a ancestor of the
tip of the destination repository, the upstream is defined to be
behind (‘B‘ in table below) the source repository

Otherwise, the upstream repository is ahead (‘A‘ in table
below) of the source repository.

Landing a change in the normal (2 repository case: RoR and lander’s
repository), the process is logically (assuming no network issues):

Make sure lander’s repository is equivalent to RoR (start with
equality)

Apply the changes (RoR is now “Behind” the local repository)

Push the changes to the RoR

if the push succeeds, then stop. (equality restored)

if the push fails, simultaneous landings were being attempted,
and you lost the race.

When simultaneous landings are attempted, only one will succeed,
and the others will need to repeat the landing attempt. The
RoR is now “Ahead” of the local repository, and the new
upstream changes will need to be incorporated, logically as:

When an authorized committer wants to land a change set on an
hg RoR from git, there are three repositories involved. These are the
RoR, the git repository the lander is working in, and internal
hggit used for translation. The sections below describe how this
affects the normal case above.

Land from git – Happy Path

On the happy path (no commit collisions, no network issues), the steps
are identical to the normal path above. The git commands executed by the
lander are set by the tool chain to perform any additional operations
needed.

Land from git – Commit Collision

Occasionally, multiple people will try to land commits simultaneously,
and a commit collision will occur (steps 3a, 3b, & 3c above). As long as the
collision is noticed and dealt with before addition changes are
committed to the git repository, the tooling will unapply the change to
the internal hggit repository.

Land from git – Sad Path

In real life, network connections fail, power outages occur, and other
gremlins create the need to deal with “sad paths”. The following
sections are only needed when we’re neither on the happy path nor
experiencing a normal commit collision.

Because these cases cover every possible case of disaster recovery, it
can appear more complex than it is. While there are multiple (6)
different sad paths, only one will be in play for a given repository.
And the maximum number of operations to recover is only three (3). The
relationship between each pair of repositories determines the correct
actions to take to restore the repositories to a known, consistent
state. The static case is simply:

Simplistic Recovery State Diagram

Note

The simplistic diagram assumes no changes to RoR during the
duration of the recovery (not a valid assumption for real
life). See the text for information on dealing with the
changes.

States “BB” & “BA” are not shown, as they represent invalid
states that may require restoring portions of the system
from backup before proceeding.

In reality, it is impractical to guarantee the RoR is static during
recovery steps. That can be dealt with by applying the process described
in the flowchart to restore equality and using the tables below to
locate the actions.

The primary goal is to ensure correctness based on the RoR. The
secondary goal is to make the interim repository as invisible as possible.

This “shouldn’t happen”, as it implies the git repository
has been restored from a backup and the “pending landing” in
the hggit repository is no longer a part of the git history.
If there isn’t a clear understanding of why this occurred,
client side repository setup should be considered suspect,
and replaced.

Lander shot themselves in the foot - they have 2 incomplete landings
in progress. If they are extremely lucky, they can recover by
completing the first landing (“hg push RoR” -> “eB”), and proceed
from there.

The deterministic approach, which also must be used if
landing of first change set fails, is to back out second
landing from hggit and git, then back out first landing from
hggit and git.) Then equality can be restored, and each
landing redone separately.

DVCS Commands

Next Step

Active Repository

Command

pull from RoR

hggit

hgpull

pull to git

git

gitpullRoR

push from git

git

gitpushRoR

push to RoR

hggit

hgpush

Note

that if any of the above actions fail, it simply means that we’ve
lost another race condition with someone else’s commit. The recovery
path is simply to re-evaluate the current state and proceed as
indicated (as shown in diagram 1).