Periodic updates on my software startup endeavor

Manage All Application Environments With Vagrant

What if I could use Vagrant to configure and create reproducible
environments for development, test, and production? This is the question I
thought to myself while working on the planned activity to setup a
MongoDb cluster. After spending time testing the cluster with VirtualBox and
Vagrant and then switching to knife-solo to bootstrap droplets on
Digital Ocean, I opted to dig further into my question.

With the release of Vagrant 1.1 in mid-March, it was now possible to manage
virtual machines backed by providers other than VirtualBox. HashiCorp, the
creator of Vagrant, provided a compelling preview showcasing Vagrant
managing Amazon EC2 servers. The timing was right to investigate the
possibility of using Vagrant as the tool to define my environment and
subsequently set it up on either VirtualBox or Digital Ocean. As such, I
redirected my remaining few weeks for the first sprint to tackle this challenge.

Value & Exit Criteria

Chef had proven extremely useful for defining configuration for
individual servers. Vagrant would compliment Chef by providing a common,
consistent interface for managing an environment of servers. This would help
further minimize system regressions and reduce the operational cost for managing
multiple environments.

To claim success on the task, I defined the end state. As a developer and
administrator, I wanted to run the following commands to stand up a
six-server environment with VirtualBox or Digital Ocean respectively:

The environment had to be defined with a single Vagrantfile creating a proxy
server, two Node.js application servers, and a MongoDb replica set. After
provisioning with a single Chef application cookbook, the system had to be
fully operational accepting requests.

Vagrant Plugins

In my previous job, we had used Vagrant to create reproducible development
environments for the development team. Beyond creating a basic Vagrantfile
to spin up a machine provisioned with Puppet or Chef, my knowledge of Vagrant
was limited. As I dove into the internals of Vagrant to meet my objective, I
was pleasantly surprised to find an extensible plugin framework. The framework
supports new configuration, commands, virtual machine providers, and
action hooks. In fact, most of Vagrant’s built-in features are implemented
as plugins.

Operational use cases for creating and managing an environment
could be implemented as a set of plugins. I looked to reuse existing ones where
possible and created new ones when needed. The four plugins that helped
achieve my objective are discussed below.

Digital Ocean Provider Plugin

To create, rebuild, and destroy Digital Ocean droplets, I needed a Vagrant
provider plugin. Luckily, John Bender had already established a GitHub
project that delivered the basic functionality. Unfortunately, it was missing
support for SSH keys (it used the insecure Vagrant key) and relied on a root
account for subsequent provisioning. John was gracious enough to support my
modifications ultimately transitioning the project over to me. Adding
features to this plugin was a great exercise in understanding the fundamentals
of Vagrant and I highly recommend anyone interested to browse the source code.

create a new user account during droplet creation (allowing me to disable
the root account during provisioning)

provision a droplet with the shell or Chef

While this plugin met 80% of my objective, creating new droplet instances
introduced a problem. An IP address is not assigned to a droplet until after
it is created, however, my Chef recipes required knowledge of it upfront.

Host Manager Plugin

To solve this dilemma, I opted to reference servers in my Chef recipes using
host names and leverage a Vagrant plugin to synchronize a /etc/hosts file across
the environment to resolve the names. I created a Vagrant 1.1 compliant
plugin, vagrant-hostmanager, that hooked into the up action for a new
server. After server creation, a line containing the server’s new
IP address and host name is added to the /etc/hosts file of each active
server within the Vagrant environment. Once the vagrant up command completes,
all servers defined within the Vagrantfile can resolve to one another using
host names.

Due to this approach, I had to disable provisioning when calling vagrant up.
That is why my end state included the --no-provision switch and a second
command to provision the servers.

MongoDb Plugin

Now that servers within my Digital Ocean environment could resolve to one
another, I could move forward with configuring a replica set. Before I had
started my investigation into Vagrant, I had attempted to use the mongodb
cookbook. Unfortunately, the cookbook required a Chef server to search for
replica set members and I had opted for Chef solo a few weeks earlier.
Additionally, it felt awkward that a recipe provisioning a single MongoDb
replica set member would also attempt to initiate a replica set if all
members were available.

A Vagrant plugin seemed to be a better fit — it had access to the
Vagrantfile containing configuration for all servers in the environment and
therefore could act on the environment itself. I created a plugin,
vagrant-mongodb, that provided an adminstrator with new configuration
options in the Vagrantfile to describe a replica set. Here is a brief example:

With a replica set defined within a Vagrantfile, the plugin will check
if all members are available, and if so, initiate it. By default,
this happens automatically after executing vagrant provision, although this
behavior may be disabled and a custom command executed instead.

Berkshelf Plugin

With the three plugins above, I could now meet my objective. However, I took
the time to clean up my use of Chef from earlier weeks. Since I was replacing
aspects of knife-solo with Vagrant, I had the opportunity to use
Berkshelf for cookbook dependency management. I collapsed my
marinara-kitchen project into a single application cookbook called
marinara-cookbook containing the recipes I wrote earlier. I defined my
cookbook’s dependencies in the metadata.rb file:

With the Berkshelf Vagrant plugin installed, all cookbook dependencies
are automatically available to each server during the provisioning process.

Vagrantfile

With plugins to manage Digital Ocean droplets, synchronize a /etc/hosts file
for server resolution, and initiate MongoDb replica sets, I was ready to
define my application’s environment in a single Vagrantfile. Below is my
file that works with both VirtualBox and Digital Ocean (with a few manual
tweaks required until Vagrant 1.2 is released):

You will note the declaration of private networks and static IP addresses
in the file. This enables the machines to communicate with one another
when running locally using VirtualBox. Currently, I have to ignore the
private IP addresses when using Digital Ocean.

Vagrant 1.2

When Vagrant 1.2 is released, a user will be able to configure
override attributes for a specific provider. In the file above, I can move
the ignore_private_ip configuration into the Digital Ocean provider without
manually changing values.

More importantly, Vagrant 1.2 introduces the possibility of executing actions
in parallel for a multi-server environment. This will cut down my current time
to build a new six-server environment from 15 minutes to ~3 minutes.

What’s Next

Overall, I’m quite pleased with the outcome. With this sprint complete, I
can now create and manage a multi-server environment for development, test, and
production with the combination of Vagrant and Chef. Over the next six weeks
I’ll be shifting my focus from infrastructure to product management
defining my product’s scope and drafting a user experience in preparation
for development. Stay tuned for more.