… interesting tidbits of release engineering.

The canonical commit/push/land cycle at Mozilla

[This is an experiment in publishing a doc piece by piece as blog entries.
Please refer to themain pagefor
additional context.]

Untangling the terminology

In the old days, before DVCS, “commit” only had only one real purpose.
It was how you published your work to the rest of the world (or your
project’s world at least). With DVCS, you are likely committing quite
often, but still only occasionally publishing.

In the early days of hg at Mozilla, you would “push your changes”. But
with the advent of social coding sites such as bitbucket & github, you
often push your changes there as well. Or you push to our try servers.

Informally, Mozillian’s use the term “landing” to describe committing a
change to one of the official repositories (also known as “repository of
record” or RoR). So while most of the online documentation refers to
“commit”, I’ll use the less ambiguous term “landing” here.

The Evolvolution of the Landing Cycle

It happened slowly, so we may not have noticed it, but the distinctions
between “commit” and “land” have changed over time. Can we benefit from
embracing these changes and extending them further? I believe so, even
though that might require us to view our toolsets differently.

The old version

With centralized VCS, the cycle was roughly like:

The new version

With DVCS, the cycle has been expanded to account for all the “social
coding” that goes on between developers directly and indirectly via
(public or private) hosted repositories.

NOTE

Nothing has changed w.r.t. how developers interact with the RoR!

So What?

Besides demonstrating my skill level at graphviz, what can be learned
from these diagrams? I think something very special:

There are two separate work flows here, and they interact only in
well defined ways.

or

The work flow of a development is distinct from the workflow of
landing changes into the official repositories.

I assert that we can make the following conclusion from that
observation:

Conclusion

There is no hard requirement that both the developer and the
official repositories use identical tooling. What is the hard
requirement is that the contract between the two continues to be
met.

This really shouldn’t be news worthy - it is actually the way things
have always been. Before DVCS, every developer did their own source code
management on their computer. Maybe it was multiple checkouts, tarballs,
patch sets, but chances are it had nothing to do with the tool used for
the RoR. Developers even managed to exchange information with each
other, again via tarballs, emailed patches, directory permissions and a
host of other methods outside of the RoR tooling. (Personally, my first
production use of hg was to follow the pattern Mozilla documented for
using hg with CVS. That was wonderfully freeing, as it gave me
the safety net of local commits and local branches. As a developer, I
wanted and could make good use of far more flexibility than any well
designed release system would let me have.)

But wait, there’s more

What isn’t shown above is that there is another interface on
the other side of the repositories of record. That interface is the
contract for using the buildfarm/testfarm/tbpl/L10N machinery that is
being run by various groups, all pulling from the repositories of
record. (For a zoomed view of this diagram, see this
post.)

Because this “backend API” exists, even if we change the tooling used to
land changesets, it would have to be done in a manner that continues to
support the automated usage of the RoRs. That workflow is not a “natural
fit” for any DVCS (arguably especially not git). [1]

Another way to position that is, if we can find a clean, quick, way for
developers to continue using hg as the only repository of record, we
isolate them from tracking and complying with changes in the RoR ->
buildbot contract. That’s the path I’m currently following.

Yes, the “backend API” could, theoretically, be changed. At
the very least, that is a large (hence long term - possibly
years) project, due to both the number of systems involved, and
the need to keep those systems up 24x7x365.

To provide more timely improvements in the process, I’m focussed
on shorter term deliveries.