Chat / Contact

Latest News from 100TB

Infrastructure As Code

It’s 4am and the production server has gone down. It’s simple enough to spin up a new instance, except that in your sleep deprived state, you can’t quite remember the exact configuration that production was running on. Try this first - no, wait, that was supposed to be the dev environment.

Wouldn’t it be nice to have something manage all this configuration for you? No, there are no robots coming to take over the devops team yet. I’m talking about using Infrastructure as Code to automatically and consistently manage infrastructure configuration.

What is Infrastructure as Code?

Infrastructure as Code (IaC) is the process of managing and configuring servers and environments through human-readable definition files. When done properly, it allows infrastructure to be rebuilt consistently and to be automated - no more logging in as root on production servers.

This can save your team a lot of pain, especially when working across multiple environments.

In this post, we’ll cover some of the benefits of developing IaC both in production and development environments, and introduce some of the many tools important in an IaC workflow.

The case for setting up IaC in production

As your product becomes popular, the server load will increase. At some point it’s more cost effective to scale horizontally by adding more servers behind a load balancer. As you add servers, the effort required to keep them up to date increases as well. Patching and testing servers becomes a time consuming job, and each new server you add needs to have the same versions configured and installed.

At this point, rather than manually setting up each new server configuration, it’s time to automate your infrastructure.

Automatic scaling requires storing your infrastructure in code. Your orchestration tool will read your definition files each time they provision a new server and set up your desired configuration. It’s more consistent than manually setting up config files in different environments with each new change.

The same system works for auto-healing servers - why get woken up in the middle of the night by a failing server alarm when a new server can be automatically provisioned to replace the failed one? A definition file is more reliable than a bleary-eyed developer.

Employing IaC in production servers means faster recovery time when things hit the fan, more predictable scaling and more cost efficient growth.

Infrastructure as Code in development

Hopefully the case for IaC in production is clear, but there are many advantages to IaC within a development workflow as well. Using code to define and deploy configuration ensures consistency across development environments even when needs are constantly changing.

The age-old “works on my machine” problems have plagued developers since forever. Not so long ago, it was common practice amongst PHP developers to develop and test locally on a WAMP server and then deploy to a linux staging machine for (hopefully) some QA. To prevent issues between environments, there were so many caveats and gotchas you needed to remember. For example, Windows file system is case insensitive, but Linux file systems like ext3 aren’t. If you tried to load aFile.php on your local machine it would work, but on production? Not so much. With IaC, you can guarantee a consistent environment and production parity.

IaC gives you a reliable way to synchronize and track changes made in an environment, which is crucial when working in a team. Changing a configuration file, added a library, upgrading an existing dependency - all of these things need to happen across all development machines to avoid configuration drift. With the right implementation, you can allow developers to customize their own environments, but lock critical application dependencies for everyone.

By defining your infrastructure in code, you are able to use all the same tools that programmers use for code management. As an example, you’re now able to add your infrastructure definition files to a version control system like git and enjoy all of the benefits programmers have had for years: improved error tracking, fast and reliable rollbacks, track changes and examine your infrastructure history. The business case for storing infrastructure in version control may be even better than storing code there: configuration files have a tendency to be arcane and counter intuitive - having the tooling for code reviews prevent a typo making its way to production and taking your application down.

Tests can be fantastic source of confidence for a development team, but they can also be a cause of frustration. When deadlines are tight, and you have some working code with tests passing locally but failing on the build machine, it can be tempting to comment out the tests and add a TODO. Even with the best of intentions, it’s rare that deadline pressure eases and the tests remain commented and unhelpful. Using Infrastructure as Code means you can compare the development and test environments to see what may have changed and with immutable builds; you can easily start a test environment to manually run tests and investigate errors.

In summary, development environments benefit from IaC in three ways:

Automated configuration means that every environment interacts with code in the same way, preventing “it worked on my machine” conversations.

Configuration changes are trackable and can be rolled back if needed.

Failing tests due to configuration can be compared across environments to pinpoint changes. Test environments can be easily spun up to investigate errors.

Convinced? What’s the story

Ready to start moving to IaC? Outlined below are the popular technologies used today to achieve IaC, together with some example implementation. Solutions for IaC generally fall into one of three camps: configuration management, virtualization, or containers. Each has its own set of advantages and disadvantages, so it’s important to select a solution that’s the correct fit for your business. The solutions can be combined as well - it’s quite common to install dependencies when building a container, and then use a configuration management tool to add various configuration files. In fact, tools like Ansible provide both containerization and configuration management wrapped up into one cohesive package.

Configuration Management Tools

Configuration Management Tools make changes to a running system. Conceptually, it’s very similar to automatically running a set of scripts against a server (although modern tools add many features to help with targeting groups of servers, error resiliency and logging). In fact, one of the most popular open source configuration management tools, Chef, started life as a set of bash scripts. Chef is a client/server architecture; you store “cookbooks” containing “recipes” on a server which describe changes to a system in Ruby files. Clients will periodically check in with the server to update cookbooks and apply new changes. These changes can be tasks like adding configuration files, installing software, flushing caching or any other operation you might want to perform, and it’s easily extendable if you can write Ruby code.

One of the major drawbacks to these tools is that it’s hard to guarantee the state that the system will be in when the tool runs, due to the fact that the tools make changes to a running system. Many tools will try to be idempotent when they update the system, but this can create extra work. When provisioning new servers in the case of auto scaling, only using a configuration management tool can be quite slow. All software must be installed which takes time - it’s not possible to create a snapshot of a running system.

Other popular configuration management tools include Puppet and Salt.

Virtualization

A virtual machine is an emulated computer system, where an operating system runs either on virtualized hardware or through a hypervisor. This allows physical hardware to be shared between isolated operating systems, meaning you can run many stacks safely on one bare metal server. Virtualization providers excel for development environments because they are so conceptually simple and the tooling is mature. However, they are slower to provision and are more resource intensive compared to containers. For this reason, virtual machine environments are not recommended in production.

Vagrant is a popular tool for building and managing virtual machine environments in a single workflow. It sits on top of a virtualization provider like VirtualBox or Hyper-V to provide a reproducible, portable environment. It starts with a Vagrantfile - a human editable config file which specifies which software to install and how to install it. Once you have your base, you use provisioners and configuration management tools to customize configurations, install new software and add any other personalizations you need for productivity.

These boxes can easily be shared with another member of your team, making it simple to give a designer a working environment without having to worry about creating a full stack from them to use. They just run ‘vagrant up’, and they have a working, isolated clone of your development environment. It runs across different platforms, so even if you develop on Windows and a colleague runs Linux, Vagrant has you covered.

Containerization

Containers are conceptually very similar to virtual machines, except that they share the underlying kernel rather than sharing underlying hardware. This makes them very resource efficient and very fast to provision (you don’t have to “boot” an operating system - it’s already there), but at the expense of portability. Containers still provide process isolation, so you can run multiple stacks on one bare metal server. Check out our blog on containerization for more advantages.

Dockeris a popular production ready containerization platform. A Dockerfile allows you to specify build steps for a container in much the same way that Vagrant does. When you’re happy with a Dockerfile, you can build it and publish the built image to a public or private repository for use by orchestration platforms like Kubernetes or simply provision the image directly. From a pre-built image, provisioning a new container can be done in seconds which is very attractive for auto-scaling architectures. Docker also caches the result of each build step, so changing a configuration file and building a new image can be done quickly.

Because containers are so lightweight, it’s easy to split up your architecture into multiple containers which gives you increased flexibility down the road. For instance, if your stack requires a web server, a database server and a key-value store, you should create three different containers for each service. (This will allow you to easily scale individual services as needed.) To help with this, Docker provides a simple orchestration tool called docker-compose which allows you to provision multiple dependent containers at once.

Repositories exist for official Docker containers which you can extend if you need to add customizations. Official containers exist for many popular open source projects including Nginx, Redis, Jenkins and Postgres and with Docker-Compose, you can quickly provision a complex stack with just a few lines of setup.

Before you get started

Hopefully this post has convinced you to start testing how Infrastructure as Code can be useful for your business. A few last tips before you start building:

When getting started with IaC, it’s important to be strict. It’s easy to find yourself in a situation where you have modified your environment dynamically instead of through code. This means that your changes will be lost when rebuilding your environment.

To prevent this, it’s important to make sure that the code is the easiest way to make changes. If you’re using a provisioner like Chef to add configuration files, make sure that the Chef client is running regularly to overwrite any local changes. If your configuration is in your Docker compose file, destroy and rebuild your containers regularly - it will keep you honest, and give you confidence that everything is up to date.

Infrastructure as Code will help keep your configuration consistent, and will do so more effectively than human deployed environments. It’s time to jump on board the IaC train.