Application Deployment at Deliveroo

Our services are rapidly growing in number and we need a scalable way of managing the mayhem. Over the course of a year, our site was served as a single monolith and is now composed of over 50 services.

“When a monolith is split into microservices, the responsibilities of the infrastructure
organization for providing a stable platform for microservices to be developed and run on grows
drastically in importance. The infrastructure teams must provide microservice teams with stable
infrastructure that abstracts away the majority of the complexity of the interactions between
microservices.”

Production-Ready Microservices by Susan J. Fowler

Breaking up the monolith and new feature initiatives means that we are adding (or maybe I’m just
discovering) two new services a week. Over the past nine months, we have been making strides to
take ownership of our application infrastructure.

Making it easy to deploy new apps, with consistency, is key to preventing this devolving into mayhem.
With an infrastructure description language, like Hashicorp’s Terraform,
we can describe applications programatically. With Infrastructure as Code,
maintaining the service platform itself can be done with GitHub PRs.

The end result is that all of our services can now be developed and deployed with consistency.
There is no playbook for “adding new apps”. Teams can create a PR and we all wait for Terraform to
do its thing.

Adding more resources

My team maintains a bunch of extra templates for common resources that applications can use.
For example, adding a postgresql database.

This can be extended to any Terraformable resource, not just AWS. The template modules can abstract
common details and let us pick which parameters we can allow diverge across our organisation.

Beyond Infrastructure as Code

“In cloud native infrastructure, you must hide underlying systems to improve reliablilty. Cloud
infrastructure, like applications, expects failures of underlying components to occur and is
designed to handle such failures gracefully. This is needed because the infrastructure engineers
no longer have control of everything in the stack.

Infrastructure is ready to become cloud native when it is no longer a challenge. Once
infrastructure becomes easy, automated, self-serviceable, and dynamic, it has the potential to be
ignored. When systems can be ignored and the technology becomes mundane, it’s time to move up the
stack”

Cloud Native Infrastructure - Justin Garrison & Kris Nova

We don’t control everything about our applications in Terraform. While keeping a git log of all
changes can be useful, not all changes need to be serialised through my team.

Higher frequency changes such as deploying a commit or updating variables are tasks that are controlled
by dashboards and control panels. With an CI/CD workflow, a team may choose to have the system
autodeploy whenever the GitHub Merge button is clicked.

We think we have this abstraction at the right level. The particular details about the platform
can be concealed in the templates, while application choices are established for each instance.

About Ben Cordero

I’m a platform engineer building tools and provisioning infrastructure for everyone else.