graceful shutdown and triggered builds don't play as expected (0.8.4p1)

Description

we have several builders that act as coordinators for a set of triggered builds (so that binaries across platforms can be submitted in a single changelist).

we noticed that if you hit graceful shutdown after the coordinator has started, but before it triggers the sub-builds, you're stuck - the sub-builders stay pending, and the triggering builder never proceeds, so you never shut down, but you stop doing work.

I'll submit a well-tested patch for this issue versus buildbot master this week, if the maintainers will welcome it. I'm aware that you all would prefer a patch versus the "nine" branch, but I need to patch a branch to which we can actually upgrade; this is a production problem for us.

In short, if a graceful-shutdown command is sent to a buildbot, which then runs a trigger-and-wait step, that buildbot cluster will then cease to do anything useful indefinitely.

The deadlock consists of three blocking parts:

The graceful-shutdown command blocks, waiting for all builders to finish.

The trigger-and-wait step blocks, waiting for its triggered builds to finish.

The triggered build blocks, waiting for the graceful-shutdown to complete before *starting* itself.

My essential strategy is to edit part 3. We'll make sure that builds which have steps blocking on them are allowed to run, even if we're in the process of a graceful shutdown.

This image demonstrates the necessary behavior. When the "clean shutdown" command comes in, queued builds and non-blocking triggered builds pend until after the master restarts, while blocking triggered builds are allowed to run.