Mostly about code…

If you use LDAP with Keystone in Juno you can give your implementation a turbo-boost by using LDAP connection pools. Connection pooling is a simple idea. Instead of bringing up and tearing down a connection every time you talk to LDAP, you just reuse an existing one. This feature is widely used in OpenStack when talking to mysql and adding it here really makes sense.

After enabling this feature, using the default settings, I got a 3x-5x speed-up when getting tokens as a LDAP authenticated user.

Using the LDAP Connection Pools

One of the good things about this feature is that it’s well documented (here). Setting this up is easy. The tl;dr is that you can just enable two fields and then use the defaults and they seem to work pretty well.

Then if you want to use pools for user authentication, add this one:
# Enable LDAP connection pooling for end user authentication. If use_pool
# is disabled, then this setting is meaningless and is not used at all.
# (boolean value)
use_auth_pool=true

Experimental Setup

For my experiment I used a virtual keystone node that we run on top of our cloud, pointing at a corporate AD box using ldaps. Using an LDAP user, I requested 500 UUID tokens in a row. We use a special hybrid driver that uses the user creds to bind against ldap and ensure that the user/pass combo is valid. I also changed my OS_AUTH_URI to point directly at localhost to avoid hitting the load balancer. Finally I’m using the eventlet (keystone-all) vs apache2 to run Keystone. According to the Keystone PTL, Morgan Fainberg, “under apache I’d expect less benefit” If you’re not using the eventlet, ldaps, or my hybrid driver you might get different results, but I’d still expect it to be faster.

Results

I then enabled use_pool and use_auth_pool and restarted keystone, the results were quite a bit faster, a 5x speed-up. Wow.

getting 500 tokens
real 1m25.774s
user 0m2.302s
sys 0m1.539s

I ran this several times and the results were all within a few seconds of each other.

I also tried this test using the keystone CLI and the results were closer to 3.5x faster, still a respectable number.

Watching the Connections

I have a simple command so I can see how many connections are being used:
watch -n1 "netstat -an p tcp | grep :3269"

Using this simple code I can see it bounce between 0 and 1 without connection pools.

Using the defaults but with connection pools enabled, the number of connections was a solid 4. Several minutes after the test ran they died after a bit and went to 0.

I’m not sure why I didn’t get more than 4, but raising the pool counts did not change this value. Any ideas on this are welcome. This is because I have 4 workers on this node.

Tokens Are Fundamental

The coolest part of this is that this change speeds everything up. Since you need a token to do anything, I re-ran the test but just had it run nova list, cinder list, and glance image-list 50 times using the clients. Without the pooling, it took 316 seconds but with the pooling it took 231 seconds.

Plans

There are lots of ways to improve the performance of OpenStack, but this one is simple and easy to setup. The puppet code to configure this is in progress now. Once it lands, I plan to move this to dev and then to staging and prod in our environments. If I learn any other interesting things there, I’ll update the post.

When I was a kid, Richard Scarry’s book, “What Do People Do All Day” was one of my favorites. I saw this book at my parents house and I was thinking about trying to categorize everything I’ve worked on in the past few months, so the result of that thinking is this post. “What Do Operators Do All Day?”

Being an operator means that you need to by necessity be a jack of all trades, unless you’re at a very large provider. And so, over the past 6 months, I’ve worked on almost every piece of our cloud, and in almost all cases I learn something new and grow my skillset (which is my favorite part of working on Openstack).

Collecting My Data

Over the last 6 months or so, I’ve resolved 106 JIRA issues, and looking back at these provides a decent picture of where I spend my tracked work time. I’ve also done upstream reviews and commits, for which stackalytics will provide good details. Using this information, I’ll present where I do my work in order from most time to least.

Puppet Automation

I spend most of my time these days working on puppet modules or configuring services with puppet. Some of this work includes:

Fixing/configuring/enabling new features in services like Keystone/Nova/etc

Upgrading our puppet branches from Icehouse to master

Configuring build server & infra (cobbler, puppet, package repos etc)

Configuring/deploying Icinga, or writing new checks

Refactoring and cleanup, like moving all our keystone roles/users to YAML so that they’re simpler to add

Ansible

A close second is Ansible automation. We use ansible to manage our internode dependencies and also to drive our deployments. One example of what we’d use ansible for is to upgrade mysql, one node at a time, managing state between nodes while doing so. Over the past six months I’ve written ansible jobs to:

Deploy a new hand-built version of ovs

Perform a live upgrade mysql from 5.5 to 5.6

Upgrade openstack services from I to J or J to K

Improve to our weekly deployment process

Misc

Some of these tasks don’t show up in Jira, but they do take a good amount of my time.

Travel/training: Openstack conf, RabbitMQ training, etc

Planning: sprint planning, feature planning, expansion planning, etc

Mentoring and on-boarding: we’ve grown a lot and this one cannot be underestimated. I do about 5-10 code reviews per day even when I’m not answering questions

Working on Ubuntu packaging for openstack, we roll (some) of our own

MySQL/Galera DBA-esque work

On Call/Issues

Every few months, I do an on-call rotation for a week, these can be quiet or not, depending on the shape of our monitoring and cloud. Whether it’s a good or bad on call is usually of our own doing however. Even when not on call, I deal with issues though, although we do our best, we occasionally have problems. When you get enough nodes, you’ll get failures. They could be hardware failures, kernel issues, or even simply software failures, we’ve had them all. I could do an entire post on the issues we see here, but the ones that stick out to me as focus areas for software are ovs, mysql, and rabbitmq. Those are probably the three most complex and most important pieces of our software stack, and so they get lots of my attention.

Upstream

I think that the community of one of the best things about OpenStack, so I spend what little time I have left here. I participate in IRC and mailing list discussions as a part of the Operators and Puppet-Openstack community. I also do reviews and submit fixes primarily for puppet-openstack but also for Openstack itself. Although my commits to openstack itself have slowed, I’ve earned my 3rd ATC for Vancouver and I think it’s important to participate in this process.

Summary

One of my first concerns that I expressed when interviewing for this job was that we’d have Openstack setup in a year and then we’d be done. That has been far from the truth. In reality, the life of an Openstack operator is always interesting. There’s no shortages of things to fix, things to improve, and things to learn, and that’s why I love it. Although each release of Openstack generally makes things easier and more robust, they also always add more features along the edges to keep us busy.

Does this list match what everyone else spends their time on? Let me know in the comments.

I learned a bunch of things in Paris: French food is amazing, always upgrade OVS, but the most important thing I learned in Paris?

All operators have the same problems.

Every operator session was a revelation for me. As it turns out, I’m not the only one writing Icinga check jobs, Ansible scripts, trying to figure out how to push out updates, or fighting with ovs. These sessions not only provided by validation to let me know that we’re not doing stuff totally wrong, but also allowed everyone to share solutions. Below are some of the themes that I found particularly interesting or relevant to this premise.

Upgrades

One topic in which operators have a common interest was upgrades. Only a few of the operators have any real experience with this, but its been a pain point before for many. Some have resorted to fork-lift style “bring up a new cluster” upgrades, which is not nice for customers and requires extra hardware. How do we solve this? It seems that some projects have upgrade guides, but finding documentation on a holistic upgrade is difficult, especially gathering information on ordering (Cinder before Nova, for a contrived example). Issues specifically called out in the session were config changes (deprecations and new additions) and database migrations (including rollback). Rollback is especially worrying as its not well tested. There was no solution for this except a resolve to share information on upgrading via the Operators Mailing List.

CI/CD & Packaging

Another issue that operators face after an Openstack deployment is how to get fixes and new code out to the nodes. An upstream bug fix might take a few days, plus a week for a backport, plus a month for the distro to pick it up. This means that even if the operator fixes it themselves, they still have a delay in getting it releases. During this delay you might be impacting customers who may not be that patient. The solution for many is using a custom CI/CD system that builds “packages” of some sort, whether distro packages or custom built vdevs. It was interesting to hear that people have a myriad of solutions here. We use a toolchain quite similar to the upstream Openstack tool chain that outputs Ubuntu packages. However even with this method we still rely on dependencies and libraries provided by the Ubuntu Cloud Archive, as much as possible anyway.

There was a bunch of talks on this subjects, not just operator talks, here are a link to the ones I attended:

If you know other ones that I should go back and watch please comment here.

Automation (Puppet)

We use Puppet to configure and manage our Openstack deployment, so this was an area of interest to me. It was great to see a 100% full room with everyone focused on improving how Openstack is configured with puppet. From discussions on new check jobs to a conversation about how to better handle HA, it was great to see the real sense of community around Puppet.

The second puppet session was more of a “how are you solving X” and “how could it be better”. This was also a great session with some interesting notes.

Towards a Project?

Finally, one of the more interesting things happened later in the week when Michael Chapman and Dan Bode grabbed me in the lobby and wanted me to preview an email that was about to go out. In brief, they were proposing an operations project. This was born of the realization that I’d also made, we’re all solving the same issues. This email led to an impromptu meeting of about 30 people in the lobby of the Meridien hotel and the beginnings of an Operations Project.

Birth of a New Project

It was not the ideal venue, but we still had a great discussion on a few topics. The first was, can we have a framework that will allow us to share code? We all agreed that this project’s purpose is not to bless specific tools (for example Icinga) but instead to allow us to share Icinga scripts alongside other monitoring tools. Although this had been tried before with a github repo it had only a couple contributions, hopefully this will improve. We then dove into all the different tools that people use for packaging and whether or not any could be adapted as general purpose. Having a good tool for Ubuntu/Debian packages seemed to be something in great demand. Adding debian support for Anvil seemed to be something worth investigating further.

Other topics of discussion included:

log aggregation tools and filters

ops dashboarding tools

Ansible playbooks

You can see the Etherpad link below to see what options for each were discussed, but the best idea is to catch up on the mailing list threads which are still ongoing and add if you have to share anything.

For 2014, I had a goal of doing 12 contributions (commits) to OpenStack. I set this goal because I wanted to learn more about the different components of OpenStack and I wanted to contribute back to the project. In order to do this, I continued a pattern that I started when I first began working on Ubuntu, one which I consider to be a great way to come up to speed on and contribute to OpenStack. Sometime around May I stopped counting contributions and consider my goal to have been met(*). Since I consider that a success, I’d like to share my process with you here and hopefully it can inspire someone else.

Step 1: Learn About the Project
The first step was to learn more about the project, reading is interesting, but diving in and using is better, so I started with DevStack. This should be everyone’s first step. When you encounter something you don’t understand, go find a blog post or OpenStack Summit video about the subject.

Step 2: Contributing with Bug Triage
Once I had a basic understanding of the parts of OpenStack, I started looking at bugs. Joining BugSquad and BugControl was the first thing I did when I did community work on Ubuntu and doing triage for OpenStack is also a good first step. Start with this wikipage, especially if you’re new to Launchpad. By triaging bugs you can get an idea where the issues are and get a second level understanding of the projects. A bug report might cause you to ask yourself, why is neutron talking to nova in this way? It could be something that was not obvious from playing with DevStack. During this process you can mark bug dupes, ask follow-up questions, and perform other triage work. For details on OpenStack bug triage work, read more here. The best part about Bug Triage for new committers? There is an unlimited and never-ending supply of bugs. So which to pick? I like to focus on areas that I have a basic understanding of and interest in. Pick a few projects that make sense to you and look there.

Step 3: Finding a Bug
The real mission during bug triage is digging for a golden nugget, your first bug. This is more difficult than you’d think. You need to find a bug that you can fix, something relatively simple, because your goal in fixing your first bug is to learn the process. During your bug triage work you may have seen some bugs tagged with “low-hanging-fruit”, these are issues that have been identified as ones that would be simple to fix, a good first choice. If you don’t see an obvious bug tagged thusly, I’d recommend starting with the python-*client projects. I find that this code is easier to understand, easier to test, and has more unclaimed issues. You can also even just pull the source for one you like and look for FIXME notes. Another idea is unit tests, all projects love more tests. And finally, the docs can always use help.

Step 4: Working on the Bug
After finding your bug, get to work on fixing it. Before doing so you have some pre-work to do, like creating a launchpad account and signing the developer agreement. I’m not going to list all these steps, but it’s not too difficult. The next steps are fairly obvious, testing, reviewing, fixing, etc. A brief interlude on testing: I do all my testing during this process against DevStack or in DevStack. I have an up-to-date DevStack VM with me at all times and a git repo with my config changes for it. I’d recommend you do the same. Note, when you make your DevStack VM, give it enough RAM and disk, 2GB RAM + 4GB swap and a 20GB disk should be enough. As for code reviews, keep in mind that you will have to likely go multiple rounds on your code reviews. Most of my changes go at least 3-4 rounds, some more than 10, but don’t lose hope, it will eventually land and it will be awesome!

Anyway that’s my process for getting to a first commit with OpenStack. Hopefully if you’re just getting started this method will work for you. Good luck finding your bug and doing your first commit! If you have some other ideas on any of the stuff I covered, please comment below.

A quick update on Stacked/Hybrid auth for Keystone. I spent some time this week updating the driver for Icehouse. That branch is available here and includes packaging info. I’ve been too lazy to make a non-packaging copy of the branch but it should be easy to do. During my debugging of the new driver, I realized one thing, even though this driver does a bind using the credentials of the user you are trying to authenticate, you still need a service account to bind with if you are using subtree search. The reason is that the LDAP code does an initial bind to map your user id to a dn so that then it can do the “real” bind. If you see any issues with this code please let me know!

Most large companies have an Active Directory or LDAP system to manage identity and integrating this with OpenStack makes sense for some obvious reasons, like not having multiple passwords and automatically de-authenticating users who leave the company. However doing this in practice can be a bit tricky. OpenStack needs some basic service accounts. Then your users probably want their own service accounts, since they don’t want to stick their AD password in a Jenkins config. So multiple the number of teams that may use your OpenStack by 4 or 5 and you may have a ballpark estimate on the number of service accounts you need. Then take that number to your AD team and be sure to run it by your security guys too. This is the response you might get.

The way you can tell how a conversation with a security guy is going is to count the number of facepalms they do.

In this setup you probably also do not have write access to your LDAP box. This means that you need a way to track role assignment outside of LDAP, using the SQL back-end (a subject I’ve previous covered)

So this leaves us with an interesting problem, fortunately one with a few solutions. The one we chose was to use a “stacked” or “hybrid” auth in Keystone. One where service accounts live in Keystone’s SQL backend and if users fail to authenticate there they fallback to LDAP. For role Assignment we will store everything in Keystone’s SQL back-end. The good news is that Ionuț Arțăriși from SUSE has already written a driver that does exactly what you need. His driver for Identity and Assignment is available here on github and has a nice README to get you started. I did have a few issues getting it to work, which I will cover below which hopefully saves you some time in setting this up.

The way Ionut’s driver works is pretty simple, try the username and password first locally against SQL. If that fails, try to use the combo to bind to LDAP. This saves the time of having to bind with a service account and search, so it’s simpler and quicker. You will need to be able to at least bind RO with your standard AD creds for this to work though.

From this point forward, you probably will only have the issues I have if you have an AD/LDAP server with a large number of users. If you’re in a lab environment, just following the README in Ionut’s driver is all you need. For everyone else, read on…

Successes and Problems

After configuring the LDAP driver, I switched and enabled hybrid identity and restarted keystone. The authentication module seemed to be working great. I did my first test by doing some basic auth and then running a user-list. The auth worked, although my user lacked any roles that allowed him to do a user list. When I set the SERVICE_TOKEN/ENDPOINT and re-ran the user list, well that was interesting. I noticed that since I had enabled the secret LDAP debug option (hopefully soon to be non-secret) that it was doing A TON of queries and I eventually hit the server max before setting page_size=1000. Below is an animation that shows what it was like running keystone user-list:

This isn’t really animated, but is still an accurate representation of how slow it was

It did eventually finish, but it took 5 minutes and 25 seconds, and ain’t nobody got time for that. One simple and obvious fix for this is to just modify the driver to not pull LDAP users in the list. There are too many to manage anyway, so I made the change and now it only showed local accounts. This was a simple change to the bottom of the driver, to just return SQL users and not query LDAP.

I didn’t really want to wait five minutes to assign or list a role, so this needed fixing.

Solutions

It was late on Friday afternoon when I started toying with some solutions for the role issue. The first solution is to add OpenStack users into a special ou so that you can limit the scope of the search. This is done via the user_filter setting in keystone.conf. If this is easy for your organization, it’s a simple fix. It also allows tools like Horizon to show all your users. If you do this you can obviously undo the change to not return LDAP users.

The second idea that came to mind was that every time an AD user authenticated, I could just make a local copy of that user, without a password and setup in a way that they’d be unable to authenticate locally. I did manage to get this code functional, but it has a few issues. The most obvious one is that when users leave the company there’s no corresponding clean-up of their records. The second one is that it doesn’t allow for any changes to things like names or emails once they’ve authenticated. In general it’s bad to have two copies of something like this info anyway. I did however manage to get this code working and it’s available on github if you ever have a use for it. It is not well tested and it’s not very clean in the current state. With this change, I had this output (truncated):

The final idea which came after some discussions was to figure out why the heck a user list was needed to add a role. I dug into the client code, hacked a fix, and then asked on IRC. It turns out I wasn’t the first person to notice it. The bug (#1189933) is that when the user ID is not a UUID, it thinks you must be passing in a name and does a search. The fix for this is to pull a copy of python-keystoneclient thats 0.4 or newer. We had 0.3.2 in our repo. If you upgrade this package to the latest (0.7x) you will need a copy of python-six that’s 1.4.0 or newer. This should all be cake if you’re using the Ubuntu Cloud Archive.

Final State

In the final state of things, I have taken the code from Ionut and:

disabled listing of LDAP users in the driver

upgraded python-six and python-keystoneclient

packaged and installed Ionut’s drivers (as soon as I figure out the license for them, I’ll publish the packaging branch)

reconfigured keystone via puppet

So far it’s all working great with just one caveat so far. If users leave your organization they will still have roles and tenants defined in Keystone’s SQL. You will probably need to define a process for cleaning those up occasionally. As long as the users are deactivated in AD/LDAP though the roles/tenant mappings for them should be relatively harmless cruft.

Over the past few weeks I’ve been working on a new client implementation for keystone. Python is a bit bloated for nimble CLIs like keystone, and so I went back to a language I used at my first job back in 1999, Forth. Forth is interpreted like python but it’s also very simple and memory efficient. And unlike python, forth can fit on a floppy disk. This allows this client to be installed onto satellites and other memory constrained environments. For embedded systems, forth also makes a great alternative to C. Now, onto the client.

Mastering Forth

As you probably know, forth is stack-based, and no real programmer likes someone else managing the stack for them. I’ve taken this paradigm into the client design. So let’s dive in. Below is an example usage so you can see how simple it is:

First we remember the definition of what user-role-list does. Python offers the “help” and “dir” for introspection, forth has ‘ and DUMP which work just as well. First let’s make sure user-role-list is defined:

An update on my previous post about User Enabled Emulation, tl;dr, don’t use it. It’s slow. Here’s what I found:

I just spent parts of today debugging why keystone user-list was so slow. It was taking between 8 and 10 seconds to list 20 users. I spent a few hours tweaking the caching settings, but the database was so small that the cache never filled up, and so I realized that this was not the main issue. A colleague asked me if basic ldapsearch was slow, and no it was fine. Then I dug into what Keystone is doing with the enabled emulation code. Unlike a user_filter, Keystone appears to be querying the user list and then re-querying LDAP for each user to check if they’re in the enabled_emulation group. This leads to a lot of extra queries which slows things down. When I disabled this setting, the query performance improved dramatically to between 2-2.5 seconds, about a 4x speed-up. If you are in a real environment with more than 20 users, the gain will be pretty good in terms of real seconds.

Disabling the enabled_emulation leaves us with no user enabled information. In order to get the Enabled field back, I’m going to add a field to the schema to emulate what AD does with the enabled users. Since this portion of Keystone was designed for AD, this blog post may help clear up what exactly it expects here from an AD point of view. Read that page to the end and you get a special treat, how to use Logical OR in LDAP, see if it makes less sense than the bf language does.

Also to reduce my user count, I did enable a user_filter, which since it’s just part of the initial query does NOT appear to slow things down. You could skip the new field and just use the filter if you want, however it’s not clear what impact a “blank” for Enabled has, other than perhaps some confusion. If it has a real impact, PLEASE comment here!

I’ve been playing around this week with using Landscape to interact with puppet. I really like Landscape as an administration tool, but I also really like using puppet to manage packages and configuration. So the question is how to get puppet to do something based on Landscape. Landscape tags seemed like an obvious choice for this, tag a system as “compute-node” and it becomes a nova compute node, but how do you make this happen?

Standing on Shoulders

After running through a couple of ideas, I was inspired by a great article from Mike Milner. Mike’s blog post uses Landscape tags to do node classification and I wanted to use tags to set facter facts so I needed a few changes.

Make Some Tags

First I went to Landscape and added some basic tags to my box:

Just a couple tags

Get Some Tags

Next, I sat down to write a script that would run on each system and get the tags for the system it was running on. I did this first before I looked into the glue with Facter. This turned out to be pretty since since the Landscape API, is easy to use, well documented, and comes with a CLI implementation that makes testing easy.

ubuntu@mfisch:~$ get_tags.py
nova-compute
vm

Facter Time

Once that worked, it was time to look at Facter. A colleague told me about executable facter facts. The tl;dr for this is, drop a script into /etc/facter/facts.d, make it executable and facter will take the output from it and turn it into facter facts. This was super cool. I had planned on some complex hooks being required, but all I had to do was print the tags to stdout. However, Facter wants a key=value pair and Landscape tags are more like just values, so I decided on a method to generate keys by prepending landcape_tagN, where N is an iterated number. With that change in, I ran facter:

Values or Keys?

The puppet master will not know what to do with “landscape_tag0″ most likely, so we’ll need a convention to make this more useful. One idea that my colleague had was to actually set a tag with an = sign, like this, “type=compute”. Alas Landscape won’t let us do this, so instead we’ll probably just set a convention that the last _ is the key/value boundary. That would map like this:

datacenter_USEast -> datacenter=USEast

node_type_compute -> node_type=compute

foo_bar -> foo=bar

Note the current version of my script that’s in github doesn’t have this convention yet, you’ll probably want to choose your own anyway.

Issues

My script is pretty simple, it retrieves a list of computers, filtering on the hostname, as long as it only finds one match, it proceeds. This matching is the weakpoint of the current iteration of this script. It’s possible to have more than one system registered with the same hostname, so I’m thinking of adding a better filter here when I get time. If you’ll note the original script that I based mine on solved this by making you pass in the hostname or id as an argument. My current plan is to look at the system title in /etc/landscape/client.conf and do a 2nd level filter on that, but even that I don’t think will be guaranteed to be unique.

What is This Good For?

You probably use an automated tool to install and boot a node and it registers with Landscape (via puppet of course). Now what. We have a box running Ubuntu but not doing much else. Let’s say I want a new compute node in an openstack cluster, so all I’d have to do is tag that box with a pre-determined tag, say “compute-node”, let puppet agent run and wait. The puppet master will see the facter facts that let us know it should be a compute node and act accordingly.

Code

Last week I wrote about Keystone using LDAP for Identity. Thanks to a helpful comment from Yuriy Taraday and a quick email exchange I have solved the issue of the empty “Enabled” column. Here’s how it works.

Emulated user enable checking is useful if your LDAP system doesn’t have a simple “enabled” attribute. It works by using a separate group of users or tenants that you must be a member of in order to be enabled. Yuriy has a simple example for this which shows how we can mark user2 as enabled and user1 as disabled.

The default value for foo_enabled_emulation_dn is cn=enabled_foos,$tree_dn, in other words, user_enabled_emulation_dn has a default of enabled_users (note the s).

Keep in mind that even when you remove a user from the enabled_users group they are still a valid user to LDAP/AD. This means that your service account, which you’re using for ldaps authentication, does not need to be a member of this group.

The user_enabled_emulation_* fields appear to be undocumented to me, so I’ll work on that this week so that the official docs are more helpful.