Since the summer of 2012 I've dreamt of having an in-house Heroku service.
Ideally it would remove any systems admin work away from developers and run a large number of staging environments ad-hoc in an inexpensive fashion. Our live services run on bare-metal for performance reasons but our staging environments don't have such speed requirements.

At any time, we can have several pull requests open with features that are being developed. Being a geographically dispersed team, it's handy to be able to access these services from a remotely-visible domain, interact with them and provide feedback to one another.

My ideal scenario would be to create an automated tool which would, on-command, checkout out a pull request, push it into a VM, install it and configure nginx to point traffic to it via a unique hostname. Job queues and databases would be exposed via external services and their attributes exposed via environment variables inside each container.

An SSH-receiver script is something we can put together with just about no code at all. Gitreceive is one such project that does just that. With our SSH public keys installed we could speak to the stage deployment machine via git, just as we would with Heroku.

Key characteristics of these staging environments is that they're elastic, short-lived and should be very simple to create and destroy. As of this writing I could provision a server with 24GB of DDR3 RAM for $53.28 per month from Hetzner's online auction but I could also get an SSD-based, 512 MB RAM virtual machine on Digital Ocean for $0.0069 an hour.

The SSD should be quick enough for us to allocate a large SWAP partition. The staging site shouldn't come under heavy load so memory performance isn't the biggest concern. Being able to launch a staging environment in minutes and only pay for the number of hours we need it live would be a huge cost savings when multiplied across all of our projects. Keeping fixed costs low is the key to a healthy company.

Hetzner works out to $2.22 per GB of RAM per month versus $10 a month from Digital Ocean but Digital Ocean has per-hour granularity in it's billing, Hetzner only does monthly billing. If we switch off our staging environments we should save money. Technically we could automatically turn them off after 4 hours since the database service would be external to the container itself.

With most Virtualisation solutions you need to allocate a static amount of memory for your container. But with LXC, only the amount of memory that you're using will be allocated so it will allow for much denser hosting. With this in mind I set out to build a prototype using Docker that would spawn, install a python application that speaks HTTP via IPv4 and proxy that to the outside world.

I first installed the tugboat gem so I could interact with Digital Ocean's API via the command line. Digital Ocean already had a variety of golden images to boot new Virtual Machines with:

I opted to spawn VMs using the Ubuntu 13.04 x64 (350076) golden image. It's more efficient to use 32-bit memory pointers when you have less than 4GB of memory but I wanted to replicate our live environment as much as I could.

Once the command was given to spawn the VM I waited for a minute before the status switched to active:

The gunicorn process only speaks IPv4 as far as I can tell so I needed a way to stop IPv6 from being available and hopefully that would force Docker to bind on IPv4 interfaces. That didn't work so I tried to setup IP forwarding so IPv4 and IPv6 would transmit back and forth between one another:

This didn't seem to help either so I raised a bug report with dotcloud.

As of this writing the bug is still outstanding but I did find a work around: Using LXC directly and proxy traffic using iptables. There was a fantastic write up in Digital Ocean's help section on how to do this.

There are still a number of pieces to this system to sort out but the bottom line is that we're on the way to having our own staging cloud which could cost as little as a few dollars a month. If any of the above chimes with you and you're interested in working with these sorts of systems, please drop me a line.