It's been a really long time since I've written an article, so I really feel like I need to make this a good one. Sometimes I am just awestruck with the amount of awesomeness on GitHub! Without projects like Openstack, Docker, CoreOS, Atomic, Kubernetes, and so many more I can't even count or keep up with, my career would be completely different and nowhere near as enjoyable as it is today. I owe everything to them and because of open source projects, I really enjoy my job. I can truly say that what I do is my passion, and I really enjoy telling others about these amazing projects. I just can't keep all of this good information to myself. With all that said, I have another great project to share with you today.

Openstack + Ansible + Containers: One Incredible Project

As you can tell from my blog posts, I'm a huge fan of cloud automation and containers. Containers are the perfect unit of transport! Now add some Ansible orchestration and the result is...wait, Openstack...and it works as advertised?! I thought to myself this has to be too good to be true, right? Well, it is true. So let me share some information about it, and give you some background on why I think this project is so incredible.

Project Details

Background

In 2015, I saw two videos that drove my curiosity nuts! The first video was a presentation at the Openstack Summit held in Vancouver, and the talk was presented by Rackspace. It was a project originally started in Stackforge, but later moved to GitHub.

In the video, Rackspace developers explained why they were using Ansible for their Openstack Deployments. The result [for me] was a few profound takeaways:

The Ansible deployment uses all upstream GitHub Openstack native repositories; this is a clean, unmodified Openstack native deployment. This means no management wrapper APIs, no "mysterious" wrapper for starting/stoping Linux services, and I can build in my own upstream projects (no plugins that force me to rebuild my environment from scratch)

The deployment was production ready (unlike DevStack)

The deployment was production scale-able (unlike DevStack)

The deployment uses LXC containers as the deployment unit of management (so developers are presented with a more "true" Openstack development framework)

Users can easily modify their Git sources (public or private)

Dude, seriously?! So yeah, these guys got my attention. My thoughts, "I'll check out Ansible when I get a chance," because at the time, my plate was pretty full like everyone else's.

Ansible for Deploying and Orchestrating Openstack

A few months passed before I could actually start looking into this "magical deployment" of an extremely complicated Cloud environment, but before I got hit with another reminder of what I was missing. At the very next Openstack Summit in Tokyo, a few folks compared all of the popular orchestration too, including the new "cool kids" on the scene, Ansible and Salt.

Again, I had some very impactful take-aways from watching another Openstack Summit video! (I'm feeling great at this point).

If the solution is too complex, your company could ultimately lose some of it's intended audience. Sysops teams simply won't use it, and if approached incorrectly (without formal training or operational buy-in) you'll quickly realize that you've wasted resources; both money and peoples time. Sure you have six or seven really strong users of your awesome, programmatic-driven IaC "product X". But if your team has twenty engineers, and those six have become dedicated to writing your deployment plans, what are you gaining? I feel like this is wasteful, and it can even impact team morale. Additionally, now you're going to be tasked with finding talent who can orchestrate with "product X".

Ansible is very operations-driven and the learning curve is extremely easy. If you're hiring sysops personnel, then using Ansible will be the most natural for them. Here is a list of considerations when looking at the learning curve for some IaC solutions:

Ansible: Which takes very little time to learn, and is very operations-focused (operations driven, YAML or CLI-direct-based tasks, uses playbooks in sequence to perform tasks)

Salt: Which breaks into the client-less barrier (very fast, efficient code that uses the concepts of grains, pillars, etc)

Puppet: Which starts to introduce YAML concepts (client/server based model, with modules to perform tasks)

Then there's the part about Openstack Support; meaning who has the most modules for supporting deployments via Openstack, and creating Openstack clusters with the solution itself, the order is as follows:

Ansible: Which is most supported by the Openstack Governance and has two massive projects provided by Ansible:

Puppet: Which is supported by RDO and integrates well with ForeMan to provide Openstack Deployments

Chef: For its Openstack modules/support

Salt: Which doesn’t have great Openstack Module support, and doesn’t have many projects to deploy “vanilla” Openstack Deployments.

Is Ansible the perfect "be-all, end-all"? Of course not. Ansible does seem to treat Openstack as a "first class citizen" along with Puppet, but it seems to beat everyone in terms of general user ease of adoption.

NOTE: One way to see these results on which tools are the most used by the Openstack project for yourself (which is publicly view-able on GitHub), go to https://github.com/openstack and filter the repositories for "Ansible", "Puppet", "Chef" and "Salt" to see what is actually being built via these automation tools. This spoke volumes to me, when I was trying to find the right tool.

Openstack Deployment Options

There are a bunch of options to describe here. Pick yours below.

Manual: Openstack AIO in Less Than 20 Commands

So you're tired of the background and you just want the deployment? Fair enough. Let's get to it.

Install a base Ubuntu 14.04 LTE environment (VM or bare-metal) on your server. Make sure you at least have OpenSSH installed.

Steps 1 and 2 Make some basic host preparations. Update your hosts file for this machine hosts, and any others that may need to be defined (minimal is fine), and make sure that your host has a correct DNS configuration.

Steps 3 - 7 Next, update the host, install and start NTP, configure NTP for your correct timezone, and then install prerequisite packages that the Openstack-Ansible project will require. NOTE: If IPv6 is timing out for your host, then you will need to add '-o Acquire::ForceIPv4=true' at the end of every single command (this means just before each '&&').

Steps 8 - 11 THIS NEXT SECTION IS FOR ADDING VLAN INTERFACES TO YOUR HOST. IF YOU DON'T NEED THIS SUPPORT, OR IF YOU'RE UNSURE, SKIP IT! Next, you will NEED to be root to make the following changes (simply sudo as a privileged user will not work)! Add the following lines to /etc/modules, and then you must reboot.

Once you've run scripts/run-playbooks.sh the entire process will take anywhere from 40-120 minutes to complete. So I would recommend going to get a coffee, or continue reading below.

NOTE:The first thing you're probably going to do is log into Horizon. To view all of the randomly generated passwords, refer to the file /etc/Openstack_deploy/user_secrets.yml

Sidenote: Selecting Another Version to Deploy
Maybe you want to use a different version of Openstack? This project is perfect for that, and you can do it by selecting a different branch before you deploy (Step 13). Openstack-Ansible is intended to be completely customizable, even down to the upstream project repositories.

Make sure you're in the directory /opt/openstack-ansible/ before reviewing or checking out a new branch.

Ansible Tower: Fully Customizable and Manageable

This is a project that I'm currently working on, and Tyler Cross from Ansible has been an amazing resource in getting me started. When I first started using Ansible, I was curious about Ansible Tower. Tower is a management and orchestration platform for Ansible. The Openstack-Ansible project lends itself really well for Ansible Tower because of the deployment flow; they're using variables nearly everywhere throughout. That allows us to build on top of the upstream projects, supports backwards compatibility with older repository deployments, and it allows us to completely customize [nearly] anything about our deployment. This will take some work, but this is all possible because of their amazing Ansible practices!

If you would like to contribute to this effort, please send me an email! I am still learning my way through Ansible and Ansible Tower, but if there is something that you would like to be implemented as a customizable variable don't hesitate to ask me for contribution access.

Mind blown, right? So let's start talking about how we support his environment, and get into the guts of the post-deployment tasks.

Exploring the Environment

You've deployed the beast successfully, but now it's time to understand what you deployed.

The Containers

It's time for us to look around, and see if we like this deployment. The first thing you're going to notice, is that there are...containers? That's right, this whole deployment is scalable at a production level and designed to use containers as the unit of scale. You can choose not to use containers, but there's really no reason to deploy via native services. The container framework not only works, but it works well!

The Ansible Groups and Container Names

Now we really want to look at the next most important thing for you to understand; the Ansible Groups. This is really important when you want to run tasks against your environment using Ansible (rather than doing things manually). If you want to use the containers manually, you can still do this! It's your time to waste, not mine. If automation of tasks is desirable to you, then this is something you'll want to understand better! Luckily, this project makes things so incredibly easy for you. Just navigate to the /opt/openstack-ansible/directory, and run the following script ./scripts/inventory-manage.py -G. An example of this is shown below.

You can see, there are a lot of groups! I create a custom group for my own uses, and I'll explain this better in the Operations section below. For now, I want to tell you more about Openstack services.

Openstack Services

Openstack has a lot of services to keep up with, and adding a lot of containers, groups and other management responsibility may not see to help a whole lot. I can assure you that this has been made easy too.

What we're going to do, is cat and grep out a file, to figure out what services may be running on a particular Openstack node-type.

First, what we're going to do is navigate to the same /opt/openstack-ansible directory we use all the time (you should start seeing a pattern here). Next, we want to list out the contents of the directory /opt/openstack-ansible/playbooks/roles/, and grep for anything containing os_.

Notice the format closely: cat playbooks/roles/os_<nodetype>/defaults/main.yml | grep ": <nodetype>-" even down to the hyphen, because that's really important.So what we're concerned with are the following nova services.

So if you ever need to restart <nodetype>-<service>, you can connect to the aio1_<nodetype>_<service> and perform a service <nodetype>-<service> start|stop|restart. Better yet, we can do this the Ansible way, which [again] is listed below.

Upgrading the Environment

When you're ready to upgrade the environment (to the latest minor versions), perform the following steps.

First, you will want/need to update/synchronize your local repositories with any changes upstream. Make sure to do this in the /opt/openstack-ansible/ directory.

After this has completed, you'll see that two branches were updated (in this case Kilo = 11.2.9 and Liberty 12.0.6). What we will need to do is 'check out' the updated branch. NOTE: Updates can still exist within the same TAG, and you will know this when you see a {{* [new tag]}} indicator.

root@megatron:/opt/openstack-ansible# git checkout 12.0.6
Note: checking out '12.0.6'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b new_branch_name
HEAD is now at 972b41a... Merge "Update Defcore test list function" into liberty
root@megatron:/opt/openstack-ansible#

Make sure to check all of your services when you are done, to ensure that everything is running.

Operations and Advanced Topics

Adding Additional Ansible Roles to Openstack-Anisble

Adding additional Ansible roles is one of the best features about the Openstack-Ansible project! This is how to add/integrate Contrail/Opencontrail, Ironic, and other useful projects into the deployment. This can be done either at initial run, or by simply rerunning the Ansible build playbooks after your infrastructure is already initiated; either way is perfectly fine. To read more, follow the link to the documentation entitled: Extending Openstack-Anisble.

Instance Evacuation

Instances need to be available at all times. What if hardware issues start to arise. This is call "Instance Evacuation" and it is documented here: Openstack Instance Evacuation.

Live Migration

Live Migration is a feature similar to VMWare's vMotion. It allows you to actively transfer an instance from one compute node to another with zero downtime. Configuration changes are required, as I found that Mirantis disables these features in their Kilo release (NOTE: I need verify this wasn't an installation error). See below for further details. Openstack Live Migration.