Automating systems at the FSF

The sysadmins at the FSF have put great effort into consolidating our server infrastructure over the past 12 months. We have retired more than a dozen servers in the process, and now host most of our infrastructure on three potent machines with many CPU cores, ample RAM, and plenty of disk space. As you may know, we use Xen to virtualize our servers. Virtualization allows us to securely partition our servers into many virtual machines, each dedicated to a limited number of tasks.

We have also embarked on a journey to automate our systems
configuration as much as possible. We have selected Puppet, a systems
configuration management and automation tool, to help in that task.

One of the advantages of Puppet is that one can start small. Once a
system is under Puppet's control, it is easy to expand the Puppet
configuration over time, and thus automate more and more of it.
We started out with a very limited configuration that
defined some settings that are common to all our systems -- for
instance, making sure the sshd configuration is secure. Puppet makes
it easy to differentiate rules based on "facts" about a machine, like
the version of the operating system it runs, or whether the
system is a physical server or a virtual machine. Each system under
Puppet's control gets its own configuration stanza, so it is also
possible to do things that are specific to one machine.

Here are some of the things that we now do with Puppet, rather than by
hand:

install appropriate software packages, and standard GNU/FSF
configurations for them

create and manage user accounts

distribute SSH public keys and SSL certificates

update xen-tools configurations on virtualization host systems

We create new virtual machines with xen-tools, which pulls in the
Puppet packages. We then add the new virtual machine to our Puppet
configuration, which pulls in the default GNU/FSF configuration
without any additional work on our part. We save a lot of time setting
up new machines, and we get peace of mind: all our systems under
Puppet's control are guaranteed to have our standard configuration.

We are currently working towards the goal of generating our automated
systems monitoring configuration from our Puppet configuration. This
requires us to migrate more service configurations to Puppet. With
enough of that done, it should be possible for Puppet
to know that, for instance, www.gnu.org runs a web server on port 80. With that knowledge, Puppet can instruct our monitoring hosts to check for the availability of that service, all without manual intervention from the sysadmins.

I would like to conclude this article with a brief word of thanks to
Bernie Innocenti, who left the FSF for
another job in September. We are extremely grateful for his
contributions as an FSF sysadmin; he was instrumental in the server
consolidation and Puppet setup efforts described here.