Support extracting steps from annotated logs

Description

In general, putting too much knowledge about how builds are
done into buildbot config files is considered harmful because
it makes it hard for non-buildbot-admins to update or test the
build procedure used by buildbot.

One approach to dealing with this is to have buildbot run a single
step that just runs a script that is under full control of the
developers (as opposed to buildbot admins). That's good, but
then you lose visibility into build steps, and the waterfall is
a bit more opaque.

Chromium currently does this, and solves the big-opaque-single-step-waterfall problem by parsing the logfile and synthesizing steps from annotations inserted into the log by the build script.

Change History (8)

I do like the idea of allowing the build config to be controlled by version controlled files. But I'm not sanguine about synthesizing steps, though. I think, mainly, because I think this would increase the api surface that could be considered public significantly.

Well, so the issue is that right now, buildbot doesn't have a well-defined public api, and various configurations make assumptions about how buildbot works (which incidentally makes them hard to upgrade if those assumptions are invalidated, hence mozilla being stuck on 0.8.2 for example). The point is, that allowing steps to be synthesized means that steps can exist with state that have never been run, and steps are created in the middle of a build, and probably other things that I haven't thought of. And so code that deals with those things needs to be prepared to handle those situations. That is what I meant by API surface, which still exists, even if it is not reflected in a particular public method.

The key advantage of the annotations technique is that it's totally tool-agnostic.
It doesn't require developers to change how they do local builds.
It only requires them to add echo or print statements or whatever their tool uses
to jam text into the log file.

The travis approach requires them to change how they build, and is likely to be rejected by the average developer.

I'm not against something like this in general, nor am I saying that buildbot_travis is the proper solution. I am just leary of the implementation pointed at here, and am pointing out other approaches in the same design space.

Also, for inclusion into buildbot, I am wary of possible ambiguity in the parsing. For examplke if there was attempt to use it for metabuildbot (where the the control text may also occur in the output).

I agree with tomprince. I think this is a good idea (it's been discussed before, but I didn't know it was implemented!), but will be better to merge in the 0.9.x series (when at least the status-reporting API is well-defined) or even 1.0.x (when process APIs get nailed down).

Throughout 0.8.x, we've adopted a lot of interesting tools that turn out to create edge cases, bugs, or contradictions after they're merged. I'd rather *avoid* doing so again in the 0.8.x series, so that we can focus on defining things properly in 0.9.x. Once those definitions are well understood, we will have the tools to properly analyze bigger changes like this, and identify any violated assumptions or unintended consequences.

We should think of this as a design goal as we build those releases, though.