This feature is built-in in the Python standard library, in the resource module so that
was easy to add. The next step here would be to see how Circus could interact with tools
like cgroups.

stderr and stdout streaming

That feature is a must have, and we worked quite a bit on it to make sure it's fast
even with hundreds of processes being watched : the ability to stream the standard
output and standard error streams of all processes back into Circus.

For example, if you want all processes to write their stdout continously
into a file, you can write:

redirect_to_circus basically redirects the stream to
whatever class you've configured in stdout_stream.class.
You can provide your own class if you want to implement
a specific stream handler.

We've implemented this stream redirector with one thread per
watcher that operates the select() calls, but also have a Gevent
implementation that uses greenlets instead of threads and
Gevent's own select() implementation.

The threaded version is the default one, but you can pick the
gevent backend with the stream_backend option.

Flapping

Yet another important feature to have: the ability to detect
that a process that's launched constantly dies. That happens
when the process command line is wrong or when a resource the
program uses is not reachable for example.

Tools like daemontools will simply try again and again to run
the service, eating in the process the whole CPU of your server.

With this feature built-in and enabled by default, Circus detects
the flapping of processes, and try again later then eventually
quit trying.

Of course, the flapping dance is published in the PUB/SUB
channel so it's easy to build a script that will send you
an alert when it happens.

What's next

We've been doing extensive load testing for a Mozilla project that's
going in production next week, and Circus seems to be quite stable so
far. It's handling around 150 processes per server right now, and
everything's working like a charm.

I think Circus has reached a level now where it could replace tools
like Daemontools in some of our set ups.

The next major step is to refactor a bit the tool to make it more
pluggable.

For example, the flapping detection is built-in, but could be
factored out as a plug-in since it subscribes to the PUB/SUB channel
to get notified when a process is restarted, and could control
everything as a client like circusctl.

In other words, it would be nice to provide a base class that gets
notifications and acts upon. The auto-grow feature I want
to add --a feature where Circus adds automatically workers depending
on the load-- could be a plugin based on that class.