since last UDS, we planned to have autolanding of unity and related
components branch to trunk. This has nice side-effects of enabling us
doing extra checking, running autotests before the branch is landed, and
pushing every commits to a ppa for users to get latest crack before it
lands into precise. This short cycle of feedback will enable everyone to
spot regressions way easier and create some rapid error/fix loop
compared than what we had in the past.

Jean-Baptiste helped a lot on the infrastructure side to get a vm
running and a jenkins slave called "unity-merger" to support this. We
created as well a launchpad "unity-merger" account bot which has gpg and
ssh access to the unity-team ppa and trunk. If you see merges and
messages from a bot callled "unity-merger", don't run away! ;)

So, what happens?

1. From now on, you should not, at any mean, merging yourself a branch
to trunk. Commit access will be removed if we see abuse there, but we
try to trust people and that won't be necessary, isn't it? :)
2. For every merge request, once someone is happy with the branch and is
sure it can be safely merged to trunk, he can change the *global* status
of the merge request from "Needs Review" to "Approve".
3. A fire-tarmac cron job is run every 15 minutes and will look at every
projects in parallel (but only one merge per project at a time to avoid
conflicts) and run a special tarmac branch that I setup (more on that below)
4. Tarmac will basically ensure that the branch is mergeable, try to
merge it, check some criterias which will be extended in the future, and
then runs a jenkins job
5. Tarmac will then monitor the generic jenkins job (this one doesn't
depend on the project which will be run), and from it:
5.0 copy all the branches to other dirs and handling the fact that
jenkins is runned by another user without spoiling the upstream tree.
5.1 take the proposed branch, apply the packaging, figuring a correct
versionning for the ppa, run autoreconf/libtoolize to take the latest
dependencies (if you are using autotools, cmake project and projects
without any separate packaging branches are also supported). 5.2 Then,
it will spawn a pbuilder chroot, take all needed dependencies as well as
a local repository with latest nux, dee, bamf… from previous builds,
adds some specific test dependencies like google-tests and build from it.

5.3 Finally, it runs a make check on the project to ensure tests are passing

Note that the packaging rules are a little bit relaxed in the pbuilder:
no need for exact symbol checking and such. However, it needs the
packaging to be in synced with the upstream branch, but packaging heavy
changes like soname bumps should only happen once a cycle (and that was
what happened on the past), there will be no automatically ABI checks
though.
6. If all of this is successfull, it will then take a commit message
from the merge description, adds a "By <author>, reviewed by <reviewer>.
Fixes: <bugs_fixed>.", commit and push to trunk. If a bug is attached to
the branch, this one is automatically set to "Fix committed"
7. It dput a packaged version in the unity-team daily ppa:
https://launchpad.net/~unity-team/+archive/ppa for broader audience
(which will then build on the three architectures: i386, amd64 and
finally armel). It won't wait on the end of it building though are we
are already ensured it builds on amd64 through steps 5.

Then… it will go to the next branch of the same project

If jenkins exits in error or that other criterias are not met, the
branch will be rejected, then, the result of this private jenkins is
copied automatically on a public jenkins mirror
(https://jenkins.qa.ubuntu.com) and a comment is added to the merge
request while reverting the status to "needs review".

If jenkins doesn't respond (let's say it's under maintainenance), the
branch is not rejected, just ignored for now and no more merge will be
processed until jenkins is up again.

Limitations

As we are running all projects in parallels to avoid having to wait too
much before a branch from a project is merged, we need to have in mind
that there is no dependency automagic check, like:

1. you add a new API to nux
2. you use this API in unity.

1. needs to be merged (and so built) successfully. Please, do not
approve 2. until 1. status is "merged". Then, the local repository in
the pbuilder will automatically take the right Nux version with the
additional API. If you set it to "approved" before the nux branch is
merged, you can have great changes seeing your branch rejected because
it will fail to build. Be patient before approving the second branch, as
the merger is running every 15 minutes, you should get a merge soon. :-)

Future improvments
I will add additional checks soon in the future:

- ensure that every branches have at least a bug attached. There will be
the possibility to add "NOBUGNEEDED" to bypass it, but we will have a
report to avoid abuses (and that will enable us to have the bug
automatically set and components added). No more unclosed bugs! Having
real metrics of number of bugs closed, and such…
- ensure that every branches have tests (with a test directory of file
changes). There will be the possibility to add "NOTESTNEEDED", with the
same report to avoid abuses.
- we can even ensuring that the contributor signed the CLA checking for
the right team if the team is put up to date again on launchpad.
- we can also imagining that after UIF or FF, if the bug linked to the
branch authenticated as a UIFe or FFe is not acked by the release or
documentation team, it is rejected automatically.
- finally, we can get a "ready to release" time, where no approved
branch are merged automatically, when set in this mode, we can directly
choose which branch to merge and pick only those that are helping
getting closer to the release (a freeze-like mode in some way).

For developers, in a nutshell, no more direct merge, think to set the
"approved" status in the merge request and all should be smoothed.
Of course, as this has been tested on a test project in the infra, we
are enabling projects one after another.

Changes that were needed

This is more for interested technical people wanting to know what was
needed to be done

Tarmac branch

I have made a bunch of modification to tarmac so that it can support
this workflow, those are:
1. a jenkins plugin to communicate with a jenkins instance from tarmac.
This support monitoring a job, waiting on the result, and launching new
jobs. Of course, a full test suite is provided.
2. ensure that tarmac can be run by project, but that only one tarmac
instance can exist at some point (to avoid trying to get two merges at
the same time)
3. add a new tarmac exception type to not refuse the merge but ignore it
for now and go to the next branch. This is used for plugin which can
have transient errors like http connexion (like the jenkins one)

4. enabling merge description to be used as commit message for the branch
I'll push all those fixes and enhancements upstream.
fire-tarmac script

This script is the one piloting tarmac jobs, it basically uses a
directory and run one tarmac for each project in parallel.

Consequently, adding a new project is basically:
- adding a jenkins job using the above generic script
- creating a directory with the corresponding tarmac.conf file
Jenkins job

There is a new shell script which is project independant for doing all
what is described in step 5. It also uses some pbuilderrc tweaks and
pbuilder hooks that are installed on the same vm.

All projects are not activated on purpose right now, do not hesitate to
ping me if you have an approved merge request that isn't triggered
automatically. I prefer to monitor the first run myself as we know we
will have rough edges in the coming days. :)

Jean-Baptiste is writing a documentation from our notes while setting up
the infrastructure so that we can share with other projects we want to
have the same workflow.