[PATCH v1 3/3] travis-ci: run Git bisect on failed tests

Junio usually pushes many commits at once to the public "pu"/"next"/
"master" branches. If a test fails then it is not obvious what commit
caused the failure. Therefore we run Git bisect with the merge base
between the failing rev and its more stable branch ("next" for "pu",
"master" for "next", and "maint" for "master") as good ref to find the
offending commit. This is only enabled on "github.com/git/git" because
there we can assume that all relevant branches are up to date.

Re: [PATCH v1 3/3] travis-ci: run Git bisect on failed tests

> Junio usually pushes many commits at once to the public "pu"/"next"/
> "master" branches. If a test fails then it is not obvious what commit
> caused the failure. Therefore we run Git bisect with the merge base
> between the failing rev and its more stable branch ("next" for "pu",
> "master" for "next", and "maint" for "master") as good ref to find the
> offending commit. This is only enabled on "github.com/git/git" because
> there we can assume that all relevant branches are up to date.

Just FYI, 'git log next..pu' is not a very interesting range unless
you also use the "--no-merges" option.

Is there a good way to tell the CI that, instead of testing 'pu' (or
a specific branch in general), test all new commits that appear on
the first-parent chain between 'master'..'pu'?

It would be ideal if CI can do this every time 'master' and 'pu'
gets updated.

- Find all commits that appear on the first-parent chain between
'master..pu'.

- For each of them:

- Test it. This tests the merge result and can catch breakages
introduced by mismerging.

- Also test its second parent if it is a merge. This tests the
tip of each topic branch in isolation. CI gets a bonus point
if it can remember that the tip of the topic did not move and
the commit already passed the test to skip it ;-)

- Optionally, for those topics that failed the "tip of the topic
branch" test, "bisect master..$topic" to see where it breaks.

The thing is, "bisect" works only when you have a _single_ cause of
breakage, but because 'pu' is a pile of unsorted raw material yet to
be sifted into dirt and gem, once you see a failure at the tip of
'pu', mechanical 'bisect run' may not be a very useful tool, as you
do not know if there is only one breakage you are looking for.

Because the unit of integration we use is a topic branch (not an
individual commit on a topic branch), "this topic is broken", and
"this topic itself may be good, but merging it breaks 'pu'" are far
more interesting and relevant piece of information than "this commit
is what broke 'pu'".
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [hidden email]More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: [PATCH v1 2/3] travis-ci: disable verbose test output

> From: Lars Schneider <[hidden email]>
>
> The verbose output clutters the Travis CI webview and is not really
> useful since test debugging usually happens on a local machine.

I have not really been using the Travis CI results, so perhaps my
opinion does not count. But in other systems, I have found that the more
verbose the CI output, the better, simply because you will inevitably be
faced with a test that breaks on CI and not your local machine, and you
will have no way to get more details.

I don't know if Travis provides a better way to hide the output in the
non-failing cases.