The Problem

The Beauty of OpenStack

OpenStack is a thing of beauty, isn’t it? Just look at all those cleanly defined services, perfectly atomic, able to run standalone … it’s simply amazing. What more could developers and operators ask for in a cloud?

The Reality of OpenStack

Except, that it’s not exactly like that. All those services heavily rely on each other and given the rate of change OpenStack is experiencing the degree of complexity only stands to increase. The problem is that OpenStack has many services that are dependent on one another and managing the lifecycle is difficult and inefficient because of this.

Let’s look at an example of updating the keystone service, OpenStack’s identity management service. It is difficult to know whether or not deploying a new version of Keystone into an existing OpenStack deployment will cause problems because of compatibility with others services. It’s also difficult to move backwards and expensive to roll back a deployment of a new keystone service with today’s tools. Operators don’t want to use extra racks of hardware to test an upgrade of a service if they can avoid it and no lifecycle management tools that try to imperatively deploy and roll back can do so as reliability as we’d like between OpenStack releases.

At this point you might conclude that I have a personal vendetta against OpenStack. Although this could be justified after the many nights I’ve spent installing, configuring, and upgrading OpenStack I can assure you that’s not the case. In truth, OpenStack is not a beautiful and unique snowflake. Lots of different infrastructure platforms face this same problem and so do many application platforms.

The Many Paths to OpenStack Lifecycle Management

Today, there are many ways to manage the lifecycle of OpenStack services, but the two most prevalent can be loosely grouped into two categories: build based and image based deployments.

Build based lifecycle management uses a build service, such as PXE, and is typically coupled together with a bunch of lifecycle management tools and almost always uses some type of configuration management whether that’s Puppet, Chef, Ansible, or others.

This approach is generally inefficient because each OpenStack service is placed onto a different physical piece of hardware or at least a different operating system.

It is possible to combine multiple services on a single operating system, but this can get tricky. How does the lifecycle management tool know that OpenStack Service A in the image above won’t conflict with OpenStack Service B in terms of resources required, ports required, file systems, etc? It takes an awful lot of logic in a lifecycle management tool to know this and given the rate of change experienced in a community like OpenStack, lifecycle management tools have a hard time keeping up and delivering what users would like to deploy. Could virtual machines be used here? Possibly, but virtual machine are heavyweight and also lack rich metadata or require large infrastructures and agents loaded into those virtual machines to get metadata. In other words, VMs are too heavy and they also lack the concept of inheritance.

Finally, build based deployments can be slow. Copying each package back and forth over the wire is not the most efficient way of deploying at scale.

Image based deployments solve the problem of slow performance that build based systems have by not requiring each package to be installed. Typically an image based system has some sort of image building tool that stores images in a repository and these images are then streamed down to physical hardware.

However, even while using images, incremental updates can be slow due to the large size of images. Also, the expense of pushing a large image around for small incremental updates doesn’t seem appropriate.

Even more importantly, image based deployments don’t solve the fundamental problem of complexity that understanding the relationships between OpenStack services presents. This problem is only moved earlier in the process and must be solved when building the images themselves instead of at run-time.

There is one other consideration that should be taken when looking at building a lifecycle management solution for OpenStack and that is that OpenStack doesn’t live alone. The last thing most operators want is yet another way to manage the lifecycle of a new platform. They’d like something that they can use across platforms from bare metal, to IaaS, and possibly even in a PaaS.

What Atomic, Docker, and Kubernetes Bring to the Party

Wouldn’t it be great if there was a solution for managing the lifecycle of Openstack services that was:

Isolated, lightweight, Portable, and Separated

Easily Described run-time relationships

Could run on something thin and easy to update

Worked to manage the lifecycle of services beyond OpenStack too

That’s exactly what the combination of Docker, Kubernetes, and Atomic can provide to the existing lifecycle management solutions.

Docker provides a level of abstraction for Linux Containers through APIs and an “Engine”. It also provides an image format for sharing that supports a base and child image relationship allowing for layering. Finally, Docker provides a registry for sharing docker images. This is important because it allows developers to ship a portable image that operators can deploy on a different platform.

Kubernetes is an open source container cluster manager. It provides scheduling of Linux Containers using a master/minion construct. It uses a declarative syntax to express desired state. This is important because it allows developers to provide a description of the relationships between different Linux Containers and let’s the cluster manager do the scheduling.

Atomic provides just enough of an operating system to run containers in a secure, stable, and high performance manner. It includes Kubernetes and Docker and allows for users to update using newly developed update mechanisms such as OSTree. Here is a quick video that shows how easy it is to deploy atomic (in this case on OpenStack) and also how easy it is to upgrade Atomic. Watch OGG

So when you put these pieces together what you end up with is something that looks (at a high level) like the diagram above. OpenStack developers are free to develop on a broad choice of platforms (Linux/Vagrant/Libvirt pictured) and can publish completed images to a registry. Operators on the other side would pull the kubernetes configurations into their lifecycle management tools and the tools would launch the pods and services. This would trigger Docker running on Atomic to pull the images locally and deploy containers with the OpenStack services. Services are isolated and (we are fairly certain given our experience with our OpenShift PaaS) lots and lots of containers could be run on a single operating system to maximize density of Openstack services. There are LOTS of other benefits including ease of rollback, deployment and update speed, etc, but this alone should be enough for anyone looking at running an OpenStack cloud at scale to be interested.

Show me the Demo!

Here are several demonstrations that illustrate the scenario above. These are a demonstration of the OpenStack Kolla project and were produced in 2 weeks time by a group of amazing developers who saw the potential these technologies had.

First there is building the images and pushing them to a registry. Watch OGG

After deploying OpenStack countless times I can say that when you see each schema automatically created in MariaDB and endpoints, services, etc automatically created all in under a minute it is an amazing feeling!

“I’m Sold, What’s Next?”

In the end, the combination of Docker, Atomic, and Kubernetes show the promise of alleviating some of the pain OpenStack developers and operators have experienced. There are still a lot of unanswered questions, but we feel that this combination of technologies shows promise and are excited that they have found a home in the TripleO project through Kolla.

I decided to test Docker Hub’s automated build feature to see if I could have automated docker images created from a project relevant to Red Hat Cloud Infrastructure (RHCI), Red Hat’s private IaaS cloud solution. RHCI combines datacenter virtualization based on Red Hat Enterprise Virtualization (RHEV), scale out IaaS based on Red Hat Enterprise Linux OpenStack Platform (RHELOSP), and cloud management based on CloudForms. These come from the upstream communities of oVirt, OpenStack, and ManageIQ.

If you are interested in why containers could be so beneficial to an Infrastructure as a Service solution you could read my previous post, “Why containers for OpenStack Services?”. The bottom line is that moving more logic about the lifecycle of the IaaS services into the application layer (Think PaaS for IaaS) could solve many problems and help IaaS become much easier to manage.

Keystone Docker Image

The natural choice for the first service to attempt to containerize was the identity service, Keystone. Keystone has (relative to other openstack projects) few moving parts and is also required by most of the other services since it publishes a catalog of endpoints for the other services APIs.

7. I could check the details of the build, including the Dockerfile used and the output of the build.

Assuming that the image was good and I could run it and setup Keystone rather quickly I decided to focus on another service and then attempt launching the two together (see the results section if you want to spoil the surprise).

Ceilometer Docker Image

I selected the OpenStack telemetry project, commonly known as Ceilometer, for my next test of an automated build of a docker image to take place on commit to my forked repository. Why Ceilometer? After looking at the OpenStack architecture diagram I thought it might be one of the easier services to run in a container (basically, I used a dart board), and since it only requires keystone I thought I might be able to make it happen next. Here are the components of OpenStack Ceilometer (Telemetry) at a glance taken from the OpenStack docs.

The telemetry system consists of the following basic components:

A compute agent (ceilometer-agent-compute). Runs on each compute node and polls for resource utilization statistics. There may be other types of agents in the future, but for now we will focus on creating the compute agent.

A central agent (ceilometer-agent-central). Runs on a central management server to poll for resource utilization statistics for resources not tied to instances or compute nodes.

A collector (ceilometer-collector). Runs on one or more central management servers to monitor the message queues (for notifications and for metering data coming from the agent). Notification messages are processed and turned into metering messages and sent back out onto the message bus using the appropriate topic. Telemetry messages are written to the data store without modification.

An alarm notifier (ceilometer-alarm-notifier). Runs on one or more central management servers to allow settting alarms based on threshold evaluation for a collection of samples.

A data store. A database capable of handling concurrent writes (from one or more collector instances) and reads (from the API server).

An API server (ceilometer-api). Runs on one or more central management servers to provide access to the data from the data store. These services communicate using the standard OpenStack messaging bus. Only the collector and API server have access to the data store.

Here it is in a diagram.

I decided to start with the database and ceilometer collector and then add the API. I went the route of placing all of these services in a single image. I’m aware there is a lot of debate as to whether Docker images should only run a single process or if multiple processes could be beneficial. My intention wasn’t to optimize the image for production, rather it was to test how easy or difficult it was to take a forked GitHub project and get it into an image build in an automated fashion that I could run on my Fedora 20 workstation. Also, I did not plan to add the evaluator, notifier, or any agents to this image. Since most of the agents require other components of OpenStack.

4. I created a Dockerfile in the root of the project following the manual installation of OpenStack Ceilometer. Here is the contents of the Dockerfile. Note I wasn’t able to run the mongod command during the build successfully. More on that later, I just created a post launch script that could be executed after the docker image is launched as a work around.

I can also run them and in relatively short order have keystone and ceilometer running side by side on the same host. These containers are relatively isolated, much smaller then virtual machines, and I don’t have to worry about my local machine getting foobar’d while working on keystone or ceilometer. Some great benefits to developers and (eventually) to ops teams.

Since the keystone service was having issues I wasn’t able to run ceilometer meter-list or other commands (yet), but I do have the processes running in containers. I’ll continue to troubleshoot the keystone issue to see if I can tie these two services together.

Observations

A few thoughts came to mind while running through this exercise.

1. An area that would benefit from tooling is the ability to take an existing docker image and determine how it could be re-based on an existing parent image. For example, after I went through installing python, python-devel, mysql-devel, etc it would be nice if Docker Hub or another tool could tell me that I could save time on builds by using a parent image that already contained those components (no need to `RUN docker yum install` anything). This would save time during build processes. Call it deduplication for Docker!

2. If build times could be kept really short with such tooling it would be REALLY cool to attach an IDE to Docker Hub so that as you typed code into a project on GitHub you could instantly find out the build status. Of course syntax checking could solve some problems in a Dockerfile, but I am thinking along the lines of launching multiple docker builds and testing them with real data (system, UAT, or performance testing scenarios) and returning the result in near real-time. Building a truly integrated development experience into a continuous delivery pipeline could be really powerful (I’m imagining an IDE showing you that the line you just wrote caused a failure when run with 3-4 other docker image builds and launched on AWS, GCE, etc or that the performance was degraded).

3. Extending docker files to have pre-requisites on other docker images would allow users to reference other images required. For example, instead of installing MongoDB on the same docker image it would have been nice to be able to put some statements like this in the Dockerfile.

Perhaps this should live outside the Dockerfile in systemd, geard, heat, or some configuration language (puppet) and orchestration engine (Kubernetes). Whatever the case, once Docker Hub and other automated Docker build services have this functionality building images that depend on other services will be very powerful.

4. Some of the commands, such as running mongod and then adding a user during the docker hub build kept failing. I’m not sure if I am missing something, but it would seem that being able to run mongod during the build process to add users or seed data into the docker image is something that would be useful. Local docker builds also failed at this. Again, this might be something I am doing incorrectly.

One thing is certain in my mind, the future is bright for containerized IaaS services and sooner or later PaaS will drive the lifecycle of IaaS private cloud services and make the life of Ops much easier!

Here is a link to my docker hub builds for ceilometer and keystone if you want to look further.