Blueprint

Blueprint is a simple configuration management tool that reverse-engineers servers. It figures out what you’ve done manually, stores it locally in a Git repository, generates code that’s able to recreate your efforts, and helps you deploy those changes to production.

APIs

Philosophy

Blueprint was born out of frustration with development environments, deployment processes, and the complexity of configuration management systems.

Blueprint insists development environments realistically model production and that starts with using Linux. Blueprint only works on Debian- or Red Hat-based Linux systems. We recommend VirtualBox, Vagrant, Rackspace Cloud, or AWS EC2 for development systems that use the same operating system (and version) as is used in production.

On top of the operating system, we recommend using the same web servers, databases, message queue brokers, and other software in development and production. This brings development visibility to entire classes of bugs that only occur due to interactions between production components.

When development and production share the same operating system and software stack, they also share the same interactive management tools, meaning developers and operators alike don’t need to maintain two vocabularies. Well-understood tools like apt-get/dpkg, yum/rpm, and the whole collection of Linux system tools are available everywhere. Blueprint is unique relative to other configuration management in encouraging use of these tools.

What’s common to all configuration management tools is the desire to manage the whole stack: from the operating system packages and services through language-specific packages, all the way to your applications. We need to span all of these across all our systems. To pick on RubyGems arbitrarily: RubyGems is purposely ignorant of its relationship to the underlying system, favoring compatibility with Windows and a wide variety of UNIX-like operating systems. Blueprint understands the macro-dependencies between RubyGems itself and the underlying system and is able to predictably reinstall a selection of gems on top of a properly configured operating system.

When constructing this predictable order-of-operations used to reinstall files, packages, services, and source installations, Blueprint, along with other configuration management tools, takes great care in performing idempotent actions. Thus Blueprint prefers to manage the entire contents of a file rather than a diff or a line to append. Idempotency means you can apply a blueprint over and over again with confidence that nothing will change if nothing needs to change.

Because Blueprint can reverse-engineer systems, it is of particular use migrating legacy systems into configuration management. It doesn’t matter when you install Blueprint: changes made to the system even before Blueprint is installed will be taken into account.

This installs Python 2.6 from EPEL side-by-side with Python 2.4 and so won’t break Yum.

Reverse-engineering systems with blueprint-create

By now you’ve hopefully created a development environment running the same operating system (and version) that you use in production and installed your production stack. If this is your first time configuring production databases and web servers, don’t be shy about copying-and-pasting from the guy at the next desk or a staging or production system (perhaps even multiple systems).

Once Blueprint itself is installed you can reverse-engineer your development system with a single command.

blueprint create name

Blueprint will list packages managed by APT, Yum, RubyGems, Python’s easy_install and pip, PHP’s PEAR and PECL, and Node.js’ NPM. It will also determine which configuration files in /etc have been added or modified from their packaged versions and collect files in /usr/local that are part of any software packages installed from source (typically via GNU make(1)). Finally, it will build a list of conditions under which System V init or Upstart services should be restarted, including package upgrades and configuration changes.

All of this information is encoded in a blueprint(5) JSON document and zero or more tar(5) archives and stored in a branch called name in the local Git repository ~/.blueprints.git.

Any blueprint in the local Git repository may be applied to the system with blueprint-apply(1):

blueprint apply name

Example

Suppose you want to install a basic Ruby stack for running a Sinatra application called example proxied by Nginx on Debian-based Linux. Install the prerequisite packages:

Inspecting blueprints

Once a blueprint has been created and stored in the local Git repository, it’s easy to inspect what’s been included both in its raw JSON form and in more friendly formats.

First, print the entire blueprint(5) JSON document:

blueprint show name | less

The output of blueprint-show(1) can be a bit overwhelming, which is one of the reasons the following four commands were created.

blueprint-show-files(1) lists the pathname of each file included in the blueprint on its own line.

pathname

blueprint-show-packages(1) lists the manager (“apt”, “yum”, “rubygems1.8”, etc.), name, and version number for each package included in the blueprint on its own line. If two versions of the same package are included (as is possibly with RubyGems and some other package managers), each will be printed on its own line.

managerpackageversion

blueprint-show-services(1) lists the manager (“sysvinit” or “upstart”) and name of each service included in the blueprint on its own line.

managerservice

blueprint-show-sources(1) lists the source installations included in the blueprint. The root directory (for example, “/usr/local”) and tarball filename are printed to standard error and the contents of the tarball are printed to standard output via tar(1)’s tv options.

We’re off to a great start but there are some extraneous files and packages here that need cleaning up.

Ignoring particular resources

Rather than requiring you enumerate every detail of your infrastructure in code, Blueprint reverse-engineers most of these details from running systems. There are times, though, when it’s a bit too verbose.

Inspired by similar features in version control software, Blueprint allows you to enumerate files, packages, services, and sources that should be ignored in a file format inspired by gitignore(5). See blueprintignore(5) for details.

Blueprint looks for these rules in /etc/blueprintignore and ~/.blueprintignore. Rules in /etc/blueprintignore will be included in the blueprint itself, thereby propagating to other users of the blueprint. Rules in ~/.blueprintignore will remain local though each revision of a blueprint will store the full set of rules used to create it.

Files may be specified by fully-qualified or relative pathnames, possibly including glob syntax:

/etc/foo
foo/*
*.foo
[abc]/[xyz]

Packages must be specified by their manager and name, prefixed with :package::

:package:apt/build-essential

When a package is ignored, packages on which it depends are also ignored.

Services must be specified by their manager and name, prefixed with :service::

:service:sysvinit/ssh

Sources must be specified by fully-qualified pathnames:

:sources:/usr/local

You can ignore and unignore particular files within a source directory to fine-tune what’s included in the tarball.

Any rule may be negated by prefixing it with a !, which overrides defaults and well as previous matching rules - the last matching rule wins.

Example

/etc/apt/sources.list and /etc/hosts really don’t belong in the blueprint since they’re part of the operating system and we haven’t customized them. Add their pathnames to /etc/blueprintignore:

/etc/apt/sources.list
/etc/hosts

/etc/mysql/debian.cnf is generated by the MySQL server package in its postinst maintainer script. Add its pathname to /etc/blueprintignore:

/etc/mysql/debian.cnf

/etc/nginx/sites-enabled/default is part of the basic Nginx installation and again we don’t particularly care about it. Add its pathname to /etc/blueprintignore:

/etc/nginx/sites-enabled/default

build-essential brought a lot of friends along that are clouding what’s really important in this blueprint. Recall that ignoring a package also ignores its dependencies. The opposite is not true: unignoring a package leaves its dependencies alone. Ignoring and immediately unignoring build-essential, ruby, and ruby-dev will slim down the blueprint without any loss in completeness.

Running sudo blueprint create example again will commit a new blueprint that takes these rules into account.

Rules files and blueprint-rules

As complexity grows, you’re likely to pass an inflection point when it becomes easier to enumerate the resources you care about, not the resources that should be ignored. When the time comes, Blueprint is ready. The rules syntax used to ignore particular resources can be turned around and used to enumerate the resources to include in the blueprint. See blueprint-rules(5) for details.

The blueprint-rules(1) command reverse-engineers the system just like blueprint-create but limits the resources included in the blueprint to those in the rules file.

blueprint rules pathname

The rules files are a bit of a hybrid between the traditional Blueprint approach of reverse-engineering and the typical configuration-as-code approach of other configuration management tools.

A good exercise for the reader is creation of a separate blueprint for the MySQL server and its configuration. It’s installed alongside the web stack in development but that’s unlikely to be the case in production, so having two blueprints could be advantageous. The next chapter introduces another way to create two blueprints from one system.

Diffing, splitting and pruning existing blueprints

Refactoring is a major part of software development practice and configuration management shouldn’t be left out in the cold. Blueprint provides several tools that can be used to refactor your blueprints into more modular, maintainable, focused artifacts.

blueprint-diff(1) takes direct advantage of the subtraction operator available on Blueprint objects in the underlying library.

blueprint diff minuendsubtrahenddifference

Resources that appear in minuend but not subtrahend will be included in difference and committed to the local Git repository under that name.

blueprint-split prints each resource in src and prompts you for a choice of dest-a or dest-b. The resulting dest-a and dest-b blueprints are committed to the local Git repository.

blueprint split srcdest-adest-b

blueprint-prune instead prompts you to include or ignore each resource in src in dest.

blueprint prune srcdest

There are some limitations in both of these tools, however. You can’t currently split the files within a source tarball. The best workaround is to use rules files to create blueprints that contain only the files you want.

The same could have been done with blueprint-split to create example-nginx and example-mysql or base and custom — any distinction you desire.

Rendering templates of configuration files

Not all systems are created equal. Certainly your m2.4xlarge AWS EC2 instance packs a bit more CPU than a virtual machine running in a corner of your laptop and your configurations should be able to cope with these operational extremes.

Blueprint allows configuration files (found in /etc) to be specified as templates and (optionally) data rather than static content. These templates are rendered by a special portable dialect of the mustache(5) template language called mustache.sh. See blueprint-template(5) for details.

In their simplest form, these templates allow substitution of system parameters.

{{FOO}}

You can also substitute the output of a shell command.

{{`echo foo`}}

More complex uses can iterate over the lines of output from a shell command.

FQDN: the fully-qualified domain name according to hostname(1)’s --fqdn option.

PRIVATE_IP: the first private IPv4 address assigned to any interface.

PUBLIC_IP: the first public IPv4 address assigned to any interface.

There are several more, documented in blueprint-template(7). You can provide your own global data by adding source-able shell scripts with the .sh suffix to /etc/blueprint-template.d.

Any pathname may have an associated template, pathname.blueprint-template.mustache. An additional source-able shell script private to just this template may be placed in pathname.blueprint-template.sh.

To render a template locally (likely during development), use blueprint-template(1).

blueprint template pathname

Example

Our Unicorn configuration statically assumes four workers is the best configuration. In reality, Unicorn workers are a function of the number of CPU cores available.

Configure Unicorn to use one worker per CPU in /etc/unicorn.conf.rb.blueprint-template.mustache:

worker_processes {{CORES}}

Configure Unicorn to use four workers per CPU in /etc/unicorn.conf.rb.blueprint-template.mustache:

worker_processes {{`expr 4 \* $CORES`}}

Or, you can extract computation out of the template and into /etc/unicorn.conf.rb.blueprint-template.sh:

export WORKER_PROCESSES="$(expr 4 \* $CORES)"

/etc/unicorn.conf.rb.blueprint-template.mustache becomes:

worker_processes {{WORKER_PROCESSES}}

Now Blueprint can scale this Unicorn configuration up and down as the system allows or requires.

Controlling service restart conditions

The last step in applying a blueprint is to restart all the services whose configuration changed. Most configuration management tools require you to connect the dots explicitly but Blueprint finds many of those relationships automatically.

System V init and Upstart services are included in blueprints when their init script or configuration file, or the package that contains the init script or configuration file, are included in the blueprint. From there, Blueprint searches for resources that, when changed, should cause the service to restart.

When the package that contains the service init script or configuration file is upgraded, the service should be restarted.

When other files in the package that contains the service change, the service should be restarted.

When files in directories in the package that contains the service change, the service should be restarted.

When fully-qualified pathnames (be they files or directories) mentioned in the service init script or configuration file are changed, the service should be restarted. This accounts for both changes to individual files and changes in source tarballs.

Other files may be added to the list to watch by naming them in a comment in the service init script or configuration file.

Example

Suppose our example application reads /etc/database.yml for database connection credentials. The Unicorn application server must be restarted to read changes to this file. Mention /etc/database.yml in the Upstart configuration:

Generating POSIX shell scripts

Beneath the blueprint-apply(1) command briefly introduced before there is the -S option to blueprint-show(1) which generates a POSIX shell script that can apply a blueprint to any system.

Dependencies are especially painful when bootstrapping new systems so Blueprint takes great pains to generate dependency-free shell scripts. They extract source tarballs, place file contents and adjust owners/groups/modes, install packages, and restart services as necessary according to the algorithm described in blueprint(5). Even template rendering is handled without any dependencies.

The generated shell script for the blueprint name is written to a file in your working directory as name.sh or name/bootstrap.sh if the blueprint contains templates or source tarballs.

blueprint show -S name

This shell script and its associated files can drive deployment, provisioning, or other development environments without even having to install Blueprint first.

Sharing and distributing blueprints

DevStructure runs a free Blueprint Server at devstructure.com but this service will not be available after June 30th, 2012. Learn more about running your own Blueprint Server

We at DevStructure saw the workflow unfolding around these generated shell scripts and in them an opportunity for a macro don’t-repeat-yourself optimization. Thus were born blueprint-push(1) and blueprint-pull(1).

blueprint-push uploads the JSON document and all source tarballs referenced by a blueprint to a Blueprint Server, which stores them behind a long secret key in AWS S3. The URL that may be used to pull the blueprint later is printed to standard output.

blueprint push name

The first time you run this command it will prompt you with the contents of /etc/blueprint.cfg which you can optionally put in place. If you do, Blueprint will reuse the same secret the next time blueprint-push is called. Configuring a secret this way allows you to push revisions to your blueprints which can then be pulled from a known location.

blueprint-pull downloads the JSON document and all source tarballs referenced by a blueprint that has been pushed and stores them in the local Git repository. Typically, it accepts the URL printed by blueprint-push but if you configure a default secret in /etc/blueprint.cfg, you can pull blueprints by only their name.

blueprint pull url

blueprint pull name

Just as blueprint-show’s -S option helps bootstrap systems with zero dependencies, devstructure.com can generate shell scripts remotely so Blueprint doesn’t have to be installed ahead of time. You can bootstrap a new system from a pushed blueprint in one command:

curl https://devstructure.com/secret/name/name.sh | sh

Example

Push the example blueprint to devstructure.com:

blueprint push example

Now configure a production system to apply the latest revision to example every half hour (this behavior should be familiar to Puppet and Chef users). In root’s crontab:

Generating Puppet modules and Chef cookbooks

Blueprint can also streamline the development workflow in Puppet- or Chef-managed environments by generating complete Puppet modules or Chef cookbooks.

Generate a Puppet module in name/manifests/init.pp:

blueprint show -P name

Generate a Chef cookbook in name/recipes/default.rb:

blueprint show -C name

These modules and cookbooks may be included directly in any Puppet or Chef environment or be used as the starting point for further development — the code is formatted according to the style guidelines of the respective communities.

Note, however, that the file templates used by Blueprint are incompatible with Puppet and Chef and so can’t be included in the generated modules and cookbooks.

Integrating with AWS CloudFormation

Production environments are much more than a rack of servers these days and nowhere is that more apparent than AWS. Blueprint integrates with AWS CloudFormation to provision and bootstrap entire infrastructures declaratively.

The --cfn option to blueprint-show(1) generates the skeleton of a CloudFormation template that provisions a single EC2 instance running Amazon Linux (an RPM-based distribution supported by Amazon) which will apply the blueprint during its first boot.

Deploying your application with Blueprint

The example running throughout this tutorial takes advantage of Blueprint’s default handling of /usr/local to package up an example web application. As an application grows, this can become muddled by other packages you install from source.

Blueprint doesn’t dictate how you deploy your applications but here are a few options play very nicely with Blueprint.

Build Debian packages or RPMs for your application. These packages can include service init scripts or configuration files which Blueprint will take into account when restarting services after a deploy.

Use Blueprint to scaffold your application deployment and some other method (possibly via SSH tools like Capistrano or Fabric) to actually deploy.

And remember to use templates to render configuration files appropriately in development, staging, and production.

Local Git repository

Blueprints are stored in the local Git repository ~/.blueprints.git. Direct access isn’t typically needed but Blueprint comes with the blueprint-git(1) tool that simplifies the parameters to git(1) needed to use this repository.

Example

Clone the entire local Git repository into blueprints in the working directory:

blueprint git clone

Show the diff, from Git’s point-of-view, between the previous two revisions of the example blueprint:

blueprint git show example

The JSON document is pretty-printed for storage so Git’s diffs will actually be meaningful.

Running your own Blueprint Server

DevStructure runs a free Blueprint Server at devstructure.com but this service will not be available after June 30th, 2012.

Running a Blueprint Server opens up the blueprint-push(1) and blueprint-pull(1) commands so you can store your blueprints remotely and share them with your team.

The Blueprint Server protocols and endpoints are documented for adventurous users that want to implement their own client or server.