The Ceph Blog

In an effort to show people how OpenStack and Ceph work together I decided it would be fun to build an Amazon Machine Image (AMI) for their Elastic Compute Cloud (EC2). This would give folks the ability to showcase the integration, end users a working example, and developers a potentially easy starting environment to build on. Of course, I’m not a devops expert, so many of the assumptions and init scripts will probably make the seasoned experts cringe and gnash their teeth. Just remember, patches welcome!

For those who like to skip to the end of the story feel free to head over to Amazon and search for ‘ceph-openstack‘ in the public images or for AMI ID ‘ami-2fbf3746‘ and boot away with the following constraints:

m1.xlarge instance size

create a new or use an existing ssh key

make a relatively permissive security group (I just made it wide open for all ports)

All of the following details are contained (without explanation) in a text file in /home/ubuntu/. The rest of you can read through a bit of my hacky solutions to some of the challenges presented by EC2. Some of these issues were:

Static hostname (ec2 doesn’t really play well by default but some of the service like rabbitMQ play nicer with a static hostname)

Dynamic IP (Ceph MON doesn’t like to have the IP change)

Clean OpenStack install (DevStack was great for this, but a few modifications had to be made)

Pre-Built bootable Ceph volume (One of the key points of Ceph/OpenStack is the ability to boot from Ceph volume)

Each of these was handled by an init script that I created in /usr/local/ec2/ and added to rc.local. Now on to the meat!

Making Your EC2 Hostname Static

While EC2 typically hands you a hostname based on one of a few criteria (looking something like domU- or ip-), I wanted a static hostname that I could control so that some of my services would play nice together. Thankfully Amazon provides a handy API that you can utilize to get certain pieces of information on-the-fly. I created a /usr/local/ec2/ec2-hostname.sh script that would update both /etc/hostname and /etc/hosts with the appropriate info. The finished script is as follows:

This should give us a hostname of ‘cephdemo’ that uses the IPv4 address and the public hostname (ex: ec2-23-20-118-120.compute-1.amazonaws.com) to resolve. This should give us a good basis to start installing the other services, starting with Ceph.

Installing Ceph

Since this was to be a simple demo with everything running on a single machine, I chose to follow the 5 minute quickstart guide on Ceph.com. You should have everything there that you need. The only alteration for this demo box was the removal of the MDS in /etc/ceph/ceph.conf since we would not be installing any metadata servers.

Making Ceph MON IP Dynamic

When you spin up monitors for your Ceph cluster they expect a static IP, so I had to do a few gymnastics to make sure that when you spin up a new AMI (with a new IP) it would start Ceph with a functional monmap. To do this we need to grab the current IP address and inject a replacement monmap into our Ceph cluster. Thankfully Ceph has a monmaptool that can help us do this. The resulting /usr/local/ec2/update_mon_ip.sh script looks like this:

This updates /etc/ceph/ceph.conf so that clients will know how to access the monmap, removes the old mon.a and re-adds it with the new IP, injects the new monmap, and restarts Ceph. We should now have a running Ceph cluster any time we need to reboot.

Install OpenStack

For the purposes of this demo I wanted a nice clean OpenStack that was installed and running any time a new instance was created. The DevStack work from the Rackspace Cloud Builder guys is absolutely awesome for quick and dirty OpenStack installs. It isn’t designed to spin up a production environment, but it went a long way to getting me familiar and comfortable with OpenStack in very short order. The easiest way to grab the DevStack stuff is with git, so I went ahead and installed that:

#> sudo apt-get install git -y

Now we can clone into devstack:

#> cd ~/
#> git clone git://github.com/openstack-dev/devstack.git

There is just a bit of setup work before we run the magic install script, so we navigate into ~/devstack and create a localrc file which will contain the IP ranges, network info, and all of your service passwords. Eventually it should look something like this:

The only other change was a slight tweak to the stack.sh install script for the purposes of our custom EC2 hostname work. I wanted it to use the fully qualified domain name (the ec2- public DNS) instead of just the IP. This way when we use the VNC capability from the web-based dashboard it will be able to resolve properly. So, edit ~/devstack/stack.sh, look for the following two lines, and substitute in the $(hostname –fqdn) bit:

That should take care of Ceph auth for glance and cinder, the only thing left to do is make sure libvirt can talk to Ceph. Unfortunately libvirt handles auth via a secret key, so we have to do things a tad differently:

And voila, we should have all of the auth pieces handled. Now we just need to tell OpenStack to use Ceph.

Configuring OpenStack

In order for OpenStack to use Ceph we only need to change a couple lines in the required conf files. We’ll do this by editing the /etc/glance/glance-api.conf and /etc/cinder/cinder.conf files and perform the following changes:

A restart OpenStack should be all that’s left. Unfortunately, DevStack doesn’t install services or anything permanent, instead choosing to run everything in a series of screen windows. They did, however, make a rejoin-stack script that will restart the screen sessions based on your install. So, restarting a devstack install should look something like this:

Welcome to your new Ceph-backed OpenStack install! The rest of the work was prepping a bootable volume, making backups of the conf files for future installs, and creating startup scripts to do all of this setup auto-magically when you create a new instance.

Creating a Bootable Volume

In order to create a bootable volume by hand now that we’re connected to Ceph RBD I’ll need to spin up an instance and attach a blank volume. Unfortunately, with Folsom we’ll always need an image that is stored in Glance that is the same of any volume that we want to boot from. The way OpenStack stores all the metadata means you can’t “just” boot from volume, it needs the associated metadata, even though it doesn’t get anything from the image when it boots. This is changing, but for the versions I used in building this demo that’s still the case.

For this example I spun up the included CirrOS image so I wouldn’t have to import a new volume, but anything can be used, be it an Ubuntu image or whatever your flavor du jour happens to be. Just remember that you’d have to include steps in the init script that follows to recreate the image from the glance CLI so that you could have the metadata you needed for your example volume.

We now have a bootable volume stored in Ceph. You should be able to look at the volume info, or do an ‘#> rbd ls -p volumes’ from the ec2 command line and see the name of your “golden” volume that we’ll use later in building our init script.

Automating Startup

Now that we have everything set up and running, we just need to automate the process of startup. First we need to grab the current cinder and glance conf files, since they’ll get overwritten in the fresh install that we’ll install at startup:

We should have everything here that we need to create a fresh, working copy of OpenStack and plug it back in to Ceph (the Ceph auth stuff wont get wiped out on a fresh OpenStack install). I created a /usr/local/ec2/restack.sh script to do this for me:

With our clean OpenStack install we’ll need to create a volume and copy in our “golden” volume so that our demo instance has a bootable volume to play with. I did this via a /usr/local/ec2/create_volume.sh script but had to write in a wait to make sure the DevStack install was done first:

All that’s left is to tell our system to run these four scripts on startup and publish the AMI. Now, one fatal flaw that immediately jumps out at me is: “if I restart my machine it’ll keep making new Ceph volumes without cleaning them up.” So, you’ll either need to delete them by hand (if you are hell-bent on restarting this box a bunch) or just destroy the machine and spin up a clean one from the AMI. Adding the scripts to startup was just a simple edit to rc.local as follows:

From here I just created the AMI from the EC2 dashboard and made the permissions public. Feel free to poke around with Ceph and OpenStack and hit us up with any Ceph questions you might have via the mailing list or IRC. Happy Ceph-ing.