Background - Immutable infrastructure

As discussed in a previous blog entry, there are two main approaches to managing virtual machine software. One approach is to use the base images supplied by your cloud provider (which usually just have the base operating system installed), and then entirely manage the software stack using configuration management software such as Chef, Ansible, Puppet or Salt. The other approach is to adopt "immutable infrastructure", and do part of your configuration management ahead of time (for example, to install all the necessary software), then capture new base images with the software pre-installed. You might then use configuration management software to do the last bit of configuration to reflect the runtime details of your infrastructure.

Both approaches have advantages and disadvantages, but the advantages of using custom images include the following:

You can improve reliability, as you're only running the installer once, then shipping around that pre-installed image. Installing each VM individually can lead to inconsistent servers, if some of those installs encounter (say) a download error.

It's simple to manage the versions of all the software in your image, which helps ensure that all your servers are on exactly the same versions.

Of course, maintaining (large) VM images is a challenge in itself, but tools like Packer can help, as we'll demonstrate next.

Prerequisites

There are a few tools you need to install and configure in order to follow the instructions in the later sections:

Download and install Packer

Download, install and configure the AWS command line clients

This blog entry is going to use the popular open source tool Packer, which allows you to create virtual machine images using a variety of different cloud providers (for example, Amazon Web Services, OpenStack and Microsoft Azure). Packer is a command line tool which can be downloaded here.

You will also need to install and configure the AWS command line clients, instructions for which can be found here. Once you've configured the AWS client, the Packer tool will be able to re-use some of the same configuration, so you won't need to tell Packer about your AWS environment directly.

Building an AMI using Packer

Packer is an open source tool which can be used to create Amazon Machine Images (AMIs), or images for other clouds (see here for an example of using Packer to create an MQ image with OpenStack). Packer runs on your laptop/workstation, and does the following: creates a virtual machine (for example, on AWS) using a pre-existing base (or "source") image, runs a "provisioner" script to install software, then takes a snapshot of the VM and saves it as an AMI.

Packer uses a template file to describe what we want in our image, and some options for our cloud provider (in this case, AWS). Here's an example template, which you should copy to a file named "mq-

The "install-mq.sh" script can be found on GitHub, and should be downloaded and saved in the same location as the Packer template above.

Note that this template uses a specific AMI for the base image: ami-4d76e23e. At the time of writing, this was the latest AMI produced by Canonical for Ubuntu 14.04, in the "eu-west-1" region. These base AMIs are frequently updated with patches, and are different for each region. It is highly recommended that you check the Ubuntu Amazon EC2 Locator for the latest AMI ID for your region. In this case, I chose the "HVM" AMI on the Ubuntu site, as this is what Amazon recommends when using the latest instance types. Also note that the "amazon-ebs" Packer builder, used in this example, is used to build EBS-backed AMIs. You can use the "amazon-instance" builder to create an S3-backed AMI. See the Packer documentation for more details.

After you've saved the template file and corresponding "install-mq.sh" script, you're ready to run the following command to build the image. The command will take several minutes to complete, as it will download and install MQ.

packer build mq-aws.json

Once this command complete, you should see an image called ibm-mq-nnnnnnnn has been created, where "nnnnnnnn" is a timestamp. You can get a list of AMIs owned by your account by checking the AWS EC2 web console, or by using the following command:

aws ec2 describe-images --owners self

This image is what we'll be using as the basis for new virtual machines in the next section.

Deploying a queue manager using CloudFormation

AWS CloudFormation provides a way to manage a collection of related AWS resources described by a JSON template. You describe your resources, such as EC2 instances and EBS volumes in a JSON file, and save it in an S3 bucket. CloudFormation can then use the template to create a "stack" of resources. This stack can later be updated by providing a new version of the template (for example, if you wanted to change the AMI used, as you have a new version with security fixes). Note that OpenStack Heat is very similar to CloudFormation, and you can see an example of Heat with MQ here.

Here is a summary of the simple CloudFormation template we will be using in this blog entry, as seen in CloudFormation Designer:

an EC2 instance, using the AMI we created earlier. User data is supplied for use by cloud-init. The user data copies some text into script files in the image, and then runs it. The code, included in the template, formats the EBS volume if necessary, mounts it, and creates a queue manager.

a "VolumeAttachment" to attach the EBS volume to the instance. A particular Linux device, /dev/xvdf, is specified in the template. This is the recommended practice for AWS as documented under "Device Naming on Linux instances".

an EC2 security group to allow network traffic to MQ port 1414

If you save the template as a file, you can create a new stack in CloudFormation using the following command. You must make sure you replace the parameter values for “KeyName” and “AMI” to be your own SSH key pair, and to use the AMI you created with Packer:

If you create a new stack in CloudFormation using this template, all four resources will be created and connected together appropriately. You can see the public IP address of the new instance in the "Outputs" section. If you want to SSH into the instance to see the running queue manager, remember that the Ubuntu-provided AMIs expose the “ubuntu” user, and not “root”.

Conclusion

This blog entry showed how to run a simple MQ queue manager using CloudFormation. The use of an AMI allows simple updates (for example, for patching), by just building a new version of the AMI, then updating the CloudFormation stack. This is a good grounding for running MQ on AWS, but there is clearly more that we could do. Stay posted for future blog entries.