Mon, 04 Jan 2016

Automating Deployments: Why bother?

At my employer, we developed a new
software architecture. This involved developing and deploying several new
components, many of them following the same pattern: A daemon process listing
on a message bus (RabbitMQ, in case you're wondering) and also talking to
existing applications: A database, an Active Directory service, a NetApp
cluster or a VCenter, you name it.

Shortly after the development of these components begun, it was decided
that a different team than before should operate the software we developed.
The new team, although dedicated and qualified, was also drowning in other
work.

As we had them deploy the first few components, it became clear that each
new deployment distracted them from doing what we wanted most: build the
infrastructure that we and our software needed.

As programmers, automating thins is much of our daily business, so why not
automate some steps? We already had a Jenkins running for executing tests, so
the next step was to automate the builds.

Since our systems run Debian
GNU/Linux, and we build our applications as Debian packages, distributing the
software meant uploading it to an internal Debian mirror. This proved to be a
trickier than expected, because we use debarchiver
for managing the Debian repositories, which doesn't give immediate feedback if
an upload was successful.

After that, a deployment involved only an apt-get update &&
apt-get install $package, which at first we left to the ops team, and
later automated too - though in the production environment only after a manual
trigger.

Many of the manual and automatic deployments failed, usually due to missing
resources in the message bus, so we automated their generation as well.

Reduced feedback cycles

So at $work, automating deployments first was a means to save time, and a
means to defend the architectural freedom to develop several smaller
components instead of few small components. Later it became a means to improve
reliability.

But it quickly also became a tool to reduce the time it takes to get feedback on new features.
We found it notoriously hard to get people to use the staging environment to
try out new features, so we decided to simply roll them out to production, and
wait for complaints (or praise, though we get that less often).

Being able to quickly roll out a fix when a critical bug has managed to
slip into the production environment not only proved useful now and then, but
also gave us a feeling of safety.

I'm writing a book on automating
deployments. If this topic interests you, please sign up for the Automating Deployments
newsletter. It will keep you informed about automating and continuous
deployments. It also helps me to gauge interest in this project, and your
feedback can shape the course it takes.