OpenStack – Right Angleshttp://www.danplanet.com/blog
If they're not right...they're wrongThu, 01 Feb 2018 17:30:10 +0000en-UShourly1https://wordpress.org/?v=4.9.6Evacuate in nova: one command to confuse us allhttp://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/
http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/#commentsThu, 03 Mar 2016 19:05:02 +0000http://www.danplanet.com/blog/?p=685Continue reading →]]>If you’ve ever used OpenStack Nova, you’ve probably at least seen our plethora of confusing commands that involve migrating instances away from a failing (or failed) compute host. If you are brave, you’ve tried to use one or more of them, and have almost certainly chosen poorly. That is because we have several commands with similar names that do radically different things. Sorry about that.

Below I am going to try to clarify the current state of things, hopefully for the better so that people can choose properly and not be surprised (or at least, surprised again, after the initial surprise that caused you to google for this topic). Note that we have discussed changing some of these names at some point in the future. That may happen (and may reduce or increase confusion), but this post is just aimed at helping understand the current state of things.

Fundamentals

At the heart of this issue, are the three(ish) core operations that Nova can do (server-side) to move instances around at your request. It helps to understand these first, before trying to understand everything that you can do from the client.

1. Cold Migration

Cold migration in nova means:

Shutting down the instance (if necessary)

Copying the image from the source host to the destination host

Reassigning the instance to the destination host in the database

Starting it back up (if it was running before)

This process is nice because it works without shared storage, and even lets you check it out on the destination host before telling the source host to completely forget about it. It is, however, rather disruptive as the instance has to be powered off to be moved.

Also note that the “resize” command in Nova is exactly the same as a cold migration, except that we start up the instance with more (or less) resources than it had before. Otherwise the process is identical. Cold migration (and resize) are usually operations granted to regular users.

2. Live Migration

Live migration is what it sounds like, and what most people think of when they hear the term: moving the instance from one host to another without the instance noticing (or needing to be powered off). This process is typically admin-only, requires a lot of planets to be aligned, but is very useful if tested and working properly.

3. Evacuate

Evacuate is the confusing one. If you look at the actual english definition of evacuate in this context, it basically says (paraphrased):

To remove someone (or something) from a dangerous situation to avoid harm and reach safety.

So when people see the evacuate command in Nova, they usually think “this is a thing I should do to my instances when I need to take host down for maintenance, or if a host signals a pre-failure situation.” Unfortunately, this is not at all what Nova (at the server side) means by “evacuate”.

An evacuation of an instance is done (and indeed only allowed) if the compute host that the instance is running on is marked as down. That means the failure has already happened. The core of the evacuate process in nova is actually rebuild, which in many cases is a destructive operation. So if we were to state Nova’s definition of evacuate in english words, it would be:

After a compute host has failed, rebuild my instance from the original image in another place, keeping my name, uuid, network addresses, and any other allocated resources that I had before.

Unless your instance is volume-backed*, the evacuation process is destructive. Further, you can only initiate this process after the compute host the instance is running on is down. This is clearly a much different definition from the english word, which implies an action taken to avoid failure or loss.

Footnote: In the case of volume-backed instances, the root disk of the instance is usually in a common location such as on a SAN device. In this case, the root disk is not destroyed, but any other instance state is recreated (which includes memory, ephemeral disk, swap disk, etc).

Client Commands

Now that you understand the fundamental operations that the server side of Nova can perform, we should talk about the client. Unfortunately, the misnamed server-side evacuate operation is further confused by some additional things in the client.

In the client, if you want to initiate any of the above operations, there is a straightforward command that maps to each:

Command

Operation

Meaning

nova migrate

Cold Migration

Power off and move

nova resize

Cold Migration (with resize)

Power off, move, resize

nova live-migration

Live Migration

Move while running

nova evacuate

Evacuation

Rebuild somewhere else

The really confusing bit comes into view because the client has a few extra commands to help automate some things.

nova host-evacuate

The nova host-evacuate command does not translate directly to a server-side operation, but is more of a client-side macro or “meta operation.” When you call this command, you provide a hypervisor hostname, which the client uses to list and trigger evacuate operations on each instance running on that hypervisor. You would use this command post-failure (just like the single-instance evacuate command) to trigger evacuations of all the instances on a failed compute host.

nova host-servers-migrate

The nova host-servers-migrate command also does not directly translate to a server-side operation, but like the above, issues a cold migration request for each instance running on the supplied hypervisor hostname. You would use this command pre-failure (just like the single-instance migrate command) to trigger migrations of all the instances on a still-running compute host.

nova host-evacuate-live

Ready for the biggest and most confusing one, saved for last? The client also has a command called host-evacuate-live. You might be thinking: “Evacuate in nova means that the compute host is already down, how could we migrate anything live off of a dead host?” — and you would be correct. Unfortunately, this command does not do nova-style evacuations at all, but rather reflects the english definition of the word evacuate. Like its sibling above, this is a client-side meta command, that lists all instances running on the compute host, but triggers live-migration operations for each one of them. This too is a pre-failure command to get instances migrated off of a compute host before a failure or maintenance event occurs.

In tabular format to mirror the above:

Command

Operation

Meaning

nova host-evacuate

Evacuation

Run evacuate (rebuild elsewhere)
on all instances on host

nova host-servers-migrate

Cold Migration

Run migrate on all
instances on host

nova host-evacuate-live

Live Migration

Run live migration on all
instances on host

Hopefully the above has helped demystify or clarify the meanings of these highly related but very different operations. Unfortunately, I can’t demystify the question of “how did the naming of these commands come to be so confusing in the first place?”

]]>http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/feed/5Upgrades in Nova: Database Migrationshttp://www.danplanet.com/blog/2015/10/07/upgrades-in-nova-database-migrations/
Wed, 07 Oct 2015 14:35:33 +0000http://www.danplanet.com/blog/?p=523Continue reading →]]>This is a part of a series of posts on the details of how Nova supports live upgrades. It focuses on one of the more complicated pieces of the puzzle: how to do database schema changes with minimal disruption to the rest of the deployment, and with minimal downtime.

In the previous post on objects, I explained how Nova uses objects to maintain a consistent schema for services expecting different versions, in the face of changing persistence. That’s an important part of the strategy, as it eliminates the need to take everything down while running a set of data migrations that could take a long time to apply on even a modest data set.

Additive Schema-only Migrations

In recent cycles, Nova has enforced a requirement on all of our database migrations. They must be additive-only, and only change schema not data. Previously, it was common for a migration to add a column, move data there, and then drop the old column. Imagine my justification for adding the foobars field to the Flavor object was because I wanted to rename memory_mb. A typical offline schema/data migration might look something like this:

If you have a lot of flavors, this could take quite a while. That is a big problem because migrations like this need to be run with nothing else accessing the database — which means downtime for your Nova deployment. Imagine the pain of doing a migration like this on your instances table, which could be extremely large. Our operators have been reporting for some time that large atomic data migrations are things we just cannot keep doing. Large clouds being down for extended periods of time simply because we’re chugging through converting every record in the database is just terrible pain to inflict on deployers and users.

Instead of doing the schema change and data manipulation in a database migration like this, we only do the schema bit and save the data part for runtime. But, that means we must also separate the schema expansion (adding the new column) and contraction (removing the old column). So, the first (expansion) part of the migration would be just this:

Once the new column is there, our runtime code can start moving things to the new column. An important point to note here is that if the schema is purely additive and does not manipulate data, you can apply this change to the running database before deploying any new code. In Nova, that means you can be running Kilo, pre-apply the Liberty schema changes and then start upgrading your services in the proper order. Detaching the act of migrating the schema from actually upgrading services lets us do yet another piece at runtime before we start knocking things over. Of course, care needs to be taken to avoid schema-only migrations that require locking tables and effectively paralyzing everything while it’s running. Keep in mind that not all database engines can do the same set of operations without locking things down!

Migrating the Data Live

Consider the above example of effectively renaming memory_mb to foobars on the Flavor object. For this I need to ensure that existing flavors with only memory values are turned into flavors with only foobars values, except I need to maintain the old interface for older clients that don’t yet know about foobars. The first thing I need to do is make sure I’m converting memory to foobars when I load a Flavor, if the conversion hasn’t yet happened:

When we load the object from the database, we have a chance to perform our switcheroo, setting foobars from memory_mb, if foobars is not yet set. The caller of this method doesn’t need to know which records are converted and which aren’t. If necessary, I could also arrange to have memory_mb set as well, either from the old or new value, in order to support older code that hasn’t converted to using Flavor.foobars.

The next important step of executing this change is to make sure that when we save an object that we’ve converted on load, we save it in the new format. That being, memory_mb set to NULL and foobars holding the new value. Since we’ve already expanded the database schema by adding the new column, my save() method might look like this:

Now, since we moved things from memory_mb to foobars in the query method, I just need to make sure we NULL out the old column when we save. I could be more defensive here in case some older code accidentally changed memory_mb, or try to be more efficient and only NULL out memory_mb if I decide it’s not already. With this change, I’ve moved data from one place in the database to another, at runtime, and without any of my callers knowing that it’s going on.

However, note that there is still the case of older compute nodes. Based on the earlier code, if I merely remove the foobars field from the object during backport, they will be confused to find memory_mb missing. Thus, I really need my backport method to revert to the older behavior for older nodes:

With this, nodes that only know about Flavor version 1.0 will continue to see the memory information in the proper field. Note that we need to take extra care in my save() method now, since a Flavor may have been converted on load, then backported, and then save()d.

Cleaning Up the Mess

After some amount of time, all the Flavor objects that are touched during normal operation will have had their foobars columns filled out, and their memory_mb columns emptied. At some point, we want to drop the empty column that we’re no longer using.

In Nova, we want people to be able to upgrade from one release to the other, having to only apply database schema updates once per cycle. That means we can’t actually drop the old column until the release following the expansion. So if the above expansion migration was landed in Kilo, we wouldn’t be able to land the contraction migration until Liberty (or later). When we do, we need to make sure that all the data was moved out of the old column before we drop it and that any nodes accessing the database will no longer assume the presence of that column. So the contraction migration might look like this:

Of course, if you do this, you need to make sure that all the flavors will be migrated before the deployer applies this migration. In Nova, we provide nova-manage commands to background-migrate small batches of objects and document the need in the release notes. Active objects will be migrated automatically at runtime, and any that aren’t touched as part of normal operation will be migrated by the operator in the background. The important part to remember is that all of this happens while the system is running. See step 7 here for an example of how this worked in Kilo.

Doing online migrations, whether during activity or in the background, is not free and can generate non-trivial load. Ideally those migrations would be as efficient as possible, not re-converting data multiple times and not incurring significant overhead checking to see if each record has been migrated every time. However, some extra runtime overhead is almost always better than an extended period of downtime, especially when it can be throttled and done efficiently to avoid significant performance changes.

Online Migrations Are Hard Worth It

Applying all the techniques thus far, we have now exposed a trap that is worth explaining. If you have many nodes accessing the database directly, you need to be careful to avoid breaking services running older code while deploying the new ones. In the example above, if you apply the schema updates and then upgrade one service that starts moving things from memory_mb to foobars, what happens to the older services that don’t know about foobars? As far as they know, flavors start getting NULL memory_mb values, which will undoubtedly lead them to failure.

In Nova, we alleviate this problem by requiring most of the nodes (i.e. all the compute services) to use conductor to access the database. Since conductor is always upgraded first, it knows about the new schema before anything else. Since all the computes access the database through conductor with versioned object RPC, conductor knows when an older node needs special attention (i.e. backporting).

]]>Upgrades in Nova: Objectshttp://www.danplanet.com/blog/2015/10/06/upgrades-in-nova-objects/
Tue, 06 Oct 2015 13:35:51 +0000http://www.danplanet.com/blog/?p=511Continue reading →]]>This is a part of a series of posts on the details of how Nova supports live upgrades. It focuses on a very important layer that plays several roles in the system, providing a versioned RPC and database-independent facade for our data. Originally incubated in Nova, the versioned object code is now spun out into an Oslo library for general consumption, called oslo.versionedobjects.

As discussed in the post on RPC versioning, sending complex structures over RPC is hard to get right, as the structures are created and maintained elsewhere and simply sent over the wire between services. When running different levels of code on services in a deployment, changes to these structures must be handled and communicated carefully — something that the general oslo.messaging versioning doesn’t handle well.

The versioned objects that Nova uses to represent internal data help us when communicating over RPC, but they also help us tolerate a shifting persistence layer. They’re a critical facade within which we hide things like online data migrations and general tolerance of multiple versions of data in our database.

What follows is not an exhaustive explanation of versioned objects, but provides just enough for you to see how it applies to Nova’s live upgrade capabilities.

Versioned Objects as Schema

The easiest place to start digging into the object layer in Nova is to look at how we pass a relatively simple structure over RPC as an object, instead of just an unstructured dict. To get an appreciation of why this is important, refer back to the rescue_instance() method in the previous post. After our change, it looked like this:

Again, the first two parameters (self and context) are implied, and not of concern here. The rescue_password is just a string, as is the rescue_image_ref. However, the instance parameter is far more than a simple string — at version 3.0 of our RPC API, it was a giant dictionary that represented most of what nova knows about its primary data structure. For reference, this is mostly what it looked like in Juno, which is a fixture we use for testing when we need an instance. In reality, that doesn’t even include some of the complex nested structures contained within. You can imagine that we could easily add, remove, or change attributes of that structure elsewhere in the code or database without accounting for the change in the RPC interface in any way. If you end up with a newer node making the above call to an older node, the instance structure could be changed in subtle ways that the receiving end doesn’t understand. Since there is no version provided, the receiver can’t even know that it should fail fast, and in reality, it will likely fail deep in the middle of an operation. Proof of this comes from the test structure itself which is actually not even in sync with the current state of our database schema, using strings in places where integers are actually specified!

In Nova we addressed this by growing a versioned structure that defines the schema we want, independent of what is actually stored in the database at any given point. Just like for the RPC API, we attach a version number to the structure, and we increment that version every time we make a change. When we send the object over RPC to another node, the version can be used to determine if the receiver can understand what is inside, and take action if not. Since our versioned objects are self-serializing, they show up on the other side as rich objects and not just dicts.

An important element of making this work is getting a handle on the types and arrangement of data inside the structure. As I mentioned above, our “test instance” structure had strings where integers were actually expected, and vice versa. To see how this works, lets examine a simple structure in Nova:

Here, we define what the object looks like. It consists of several fields of data, integers, floats, booleans, strings, and even some more complicated structures like a dict of strings. The object can have other types of attributes, but they are not part of the schema if they’re not in the fields list, and thus they don’t go over RPC. In case it’s not clear, if I try to set one of the integer properties, such as “swap” with a string, I’ll get a ValueError since a string is not a valid value for that field.

As long as I’ve told oslo.messaging to use the VersionedObjectSerializer from oslo.versionedobjects, I can provide a Flavor object as an argument to an RPC method and it is magically serialized and deserialized for me, showing up on the other end exactly as I sent it, including the version and including the type checking.

If I want to make a change to the Flavor object, I can do so, but I need to make two important changes. First, I need to bump the version, and second I need to account for the change in the class’ obj_make_compatible() method. This method is the routine that I can use to take a Flavor 1.1 object and turn it into a Flavor 1.0, if I need to for an older node.

Let’s say I wanted to add a new property of “foobars” to the Flavor object, which is merely a count of the number of foobars an instance is allowed. I would denote the change in the comment above the version, bump the version, and make a change to the compatibility method to allow backports:

The code in obj_make_compatible() boils down to removing the foobars field if we’re being asked to downgrade the object to version 1.0. There have been many times in nova where we have moved data from one attribute to another, or disaggregated some composite attribute into separate ones. In those cases, the task of obj_make_compatible() is to reform the data into something that looks like the version being asked for. Within a single major version of an object, that should always be possible. If it’s not then the change requires a major version bump.

Knowing when a version bump is required can be a bit of a challenge. Bumping too often can create unnecessary backport work, but not bumping when it’s necessary can lead to failure. The object schema forms a contract between any two nodes that use them to communicate, so if something you’re doing changes that contract, you need a version bump. The oslo.versionedobjects library provides some test fixtures to help automate detection, but sharing some of the Nova team’s experiences in this area is good subject matter for a follow-on post.

Once you have your data encapsulated like this, one approach to providing compatibility is to have version pins as described for RPC. Thus, during an upgrade, you can allow the operator to pin a given object (or all objects) to the version(s) that are supported by the oldest code in the deployment. Once everything is upgraded, the pins can be lifted.

The next thing to consider is how we get data in and out of this object form when we’re using a database for persistence. In Nova, we do this using a series of methods on the object class for querying and saving data. Consider these Flavor methods for loading from and saving to the database:

Now, if you’re familiar with any sort of ORM, this doesn’t look new to you at all. Where it comes into play for Nova’s upgrades is how these objects provide RPC and database-independent facades.

Nova Conductor

Before we jump into objects as facades for the RPC and database layers, I need to explain a bit about the conductor service in Nova.

Skipping over lots of details, the nova-conductor service is a stateless component of Nova that you can scale horizontally according to load. It provides an RPC-based interface to do various things on behalf of other nodes. Unlike the nova-compute service, it is allowed to talk to the database directly. Also unlike nova-compute, it is required that the nova-conductor service is always the newest service in your system during an upgrade. So, when you set out to upgrade from Kilo to Liberty, you start with your conductor service.

In addition to some generic object routines that conductor handles, it also serves as a backport service for the compute nodes. Using the Flavor example above, if an older compute node receives a Flavor object at version 1.1 that it does not understand, it can bounce that object over RPC to the conductor service, requesting that it be backported to version 1.0, which that node understands. Since nova-conductor is required to be the newest service in the deployment, it can do that. In fact, it’s quite easy, it just calls obj_make_compatible() on the object at the target version requested by the compute node and returns it back. Thus if one of the API nodes (which are also new) looks up a Flavor object from the database at version 1.1 and passes it to an older compute node, that compute node automatically asks conductor to backport the object on its behalf so that it can satisfy the request.

Versioned Objects as RPC Facade

So, nova-conductor serves an important role for older compute nodes, providing object backports for compatibility. However, except for the most boring of calls, the older compute node is almost definitely going to have to take some action, which will involve reading and writing data, thus interacting with the database.

As I hinted above, nova-compute is not actually allowed to talk directly to the database, and hasn’t for some time, even predating Versioned Objects. Thus, when nova-compute wants to read or write data, it must ask the conductor to do so on its behalf. This turns out to help us a lot for upgrades, because it insulates the compute nodes from the database — more on that in the next section.

However, in order to support everything nova-compute might want to do in the database means a lot of RPC calls, all of which need to be versioned and tolerant of shifting schemas, such as Instance or Flavor objects. Luckily, the versioned object infrastructure helps us here by providing some decorators that turn object methods into RPC calls back to conductor. They look like this:

With these decorators in place, a call to something like Flavor.get_by_id() on nova-compute turns into an RPC call to conductor, where the actual method is run. The call reports the version of the object that nova-compute knows about, which lets conductor ensure that it returns a compatible version from the method. In the case of save(), the object instance is wrapped up, sent over the wire, the method is run, and any changes to the object are reflected back on the calling side. This means that code doesn’t need to know whether it’s running on compute (and thus needs to make an RPC call) or on another service (and thus needs to make a database call). The object effectively handles the versioned RPC bit for you, based on the version of the object.

Versioned Objects as Database Facade

Based on everything above, you can see that in Nova, we delegate most of the database manipulation responsibility to conductor over RPC. We do that with versioned objects, which ensure that on either side of a conversation between two nodes, we always know what version we’re talking about, and we tightly control the structure and format of the data we’re working on. It pays off immediately purely from the RPC perspective, where writing new RPC calls is much simpler and the versioning is handled for you.

Where this really becomes a multiplier for improving upgrades is where the facade meets the database. Before Nova was insulating the compute nodes from the database, all the nodes in a deployment had to be upgraded at the same time as a schema change was applied to the database. There was no isolation and thus everything was tied together. Even when we required compute nodes to make their database calls over RPC to conductor, they still had too much direct knowledge of the schema in the database and thus couldn’t really operate with a newer schema once it was applied.

The object layer in Nova sadly doesn’t automatically make this better for you without extra effort. However, it does provide a clean place to hide transitions between the current state of the database schema and the desired schema (i.e. the objects). I’ll discuss strategies for that next.

The final major tenet in Nova’s upgrade strategy is decoupling the actual database schema changes from the process of upgrading the nodes that access that schema directly (i.e conductor, api, etc). That is a critical part of achieving the goal.

]]>Upgrades in Nova: RPC APIshttp://www.danplanet.com/blog/2015/10/05/upgrades-in-nova-rpc-apis/
Mon, 05 Oct 2015 17:15:00 +0000http://www.danplanet.com/blog/?p=497Continue reading →]]>This is a part of a series of posts on the details of how Nova supports live upgrades. It focuses on the very important task of doing proper versioning and handling compatibility in your RPC APIs, which is a baseline requirement for supporting environments with mixed versions. The details below are, of course, focused on Nova and should be applicable to other projects using oslo.messaging for their RPC layer.

If you’re not already familiar with RPC as it exists in many OpenStack projects, you might want to watch this video first.

Why We Need Versioning

It’s important to understand why we need to go to all the trouble that is described below. With a distributed system like Nova, you’ve got services running on many different machines communicating with each other over RPC. That means they’re sending messages with data which end up calling a function on a remote machine that does something and (usually) returns a result. The problem comes when one of those interfaces needs to change, which it inevitably will. Unless you take the entire deployment down, install the new code on everything at the same time, and then bring them back up together, you’re going to have some nodes running different versions of the code than others.

Versioning your RPC interfaces provides a mechanism to teach your newer nodes how to speak to older nodes when necessary, and defines some rules about how newer nodes can continue to honor requests that were valid in previous versions of the code. Both of these apply to some time period or restriction, allowing operators to upgrade from one release to the next, ensuring that everything has been upgraded before dropping compatibility with the old stuff. It’s this robustness that we as a project seek to provide with our RPC versioning strategy to make Nova operations easier.

Versioning the Interfaces

At the beginning of time, your RPC API is at version 1.0. As you evolve or expand it, you need to bump the version number in order to communicate the changes that were made. Whether you bump the major or minor depends on what you’re doing. Minor changes are for small additive revisions, or where the server can accept anything at a certain version back to a base level. The base level is a major version, and you bump to the next one when you need to drop compatibility with a bunch of minor revisions you’ve made. When you do that, of course, you need to support both major versions for a period of time so that deployers can upgrade all their clients to send the new major version before the servers drop support for the old one. I’ll focus on minor revisions below and save major bumps for a later post.

In Nova (and other projects), APIs are scoped to a given topic and each of those has a separate version number and implementation. The client side (typically rpcapi.py) and the server side (typically manager.py) both need to be concerned with the current and previous versions of the API and thus any change to the API will end up modifying both. Examples of named APIs from Nova are “compute”, “conductor”, and “scheduler”. Each has a client and a server piece, connected over the message bus by a topic.

First, an example of a minor version change from Nova’s compute API during the Juno cycle. We have an RPC call named rescue_instance() that needed to take a new parameter called rescue_image_ref. This change is described in detail below.

Server Side

The server side of the RPC API (usually manager.py) needs to accept the new parameter, but also tolerate the fact that older clients won’t be passing it. Before the change, our server code looked like this:

What you see here is that we’re currently at version 3.23 and rescue_instance() takes two parameters: instance and rescue_password (self and context are implied). In order to make the change, we bump the minor version of the API and add the parameter as an optional keyword argument:

Now, we have the new parameter, but if it’s not passed by an older client, it will have a default value (just like Python’s own method call semantics). If you change nothing else, this code will continue to work as it did before.

It’s important to note here that the target version that we changed doesn’t do anything other than tell the oslo.messaging code to allow calls that claim to be at version 3.24. It isn’t tied to the rescue_instance() method directly, nor do we get to know what version a client uses when they make a call. Our only indication is that rescue_image_ref could be non-None, but that’s all we should care about anyway. If we need to be able to pass None as a valid value for the parameter, we should use a different sentinel to indicate that the client didn’t pass any value.

Now comes the (potentially) tricky part. The server code needs to tolerate calls made with and without the rescue_image_ref parameter. In this case, it’s not very complicated: we just check to see if the parameter is None, and if so, we look up a default image and carry on. The actual code in nova has a little more indirection, but it’s basically this:

def rescue_instance(self, context, instance, rescue_password,
rescue_image_ref=None):
if rescue_image_ref is None:
# NOTE(danms): Client is old, so mimic the old behavior
# and use the default image
# FIXME(danms): Remove this in v4.0 of the RPC API
rescue_image_ref = get_default_rescue_image()
....

Now, the rest of the code below can assume the presence of rescue_image_ref and we’ll be tolerant of older clients that expected the default image, as well as newer clients that provided a different one. We made a NOTE indicating why we’re doing this, and left a FIXME to remove the check in v4.0. Since we can’t remove or change parameters in a minor version, we have to wait to actually make rescue_image_ref mandatory until v4.0. More about that later.

Client Side

There is more work to do before this change is useful: we need to make the client actually pass the parameter. The client part is typically in rpcapi.py and is where we also (conventionally) document each change that we make. Before this change, the client code for this call looked like this (with some irrelevant details removed for clarity):

While the actual method is a little more complicated because it has changed multiple times in the 3.x API, this is basically what it looks like ignoring that other change. We take just the instance and rescue_password parameters, declare that we’re using version 3.0 and make the cast which sends a message over the bus to the server side.

In order to make the change, we add the parameter to the method, but we only include it in the actual RPC call if we’re “allowed” to send the newer version. If we’re not, then we drop that parameter and make the call at the 3.0 level, compatible with what it was at that time. Again, with distractions removed, the new implementation looks like this:

As you can see, we now check to see if version 3.24 is allowed. If so, we include the new parameter in the dict of parameters we’re going to use for the call. If not, we don’t. In either case, we send the version number that lines up with the call as we’re making it. Of course, if we were to make multiple changes to this call in a single major version, we would have to support more than two possible outbound versions (like this). The details of how client_can_send_version() knows what versions are okay will be explained later.

Another important part of this change is documenting what we did for later. The convention is that we do so in a big docstring at the top of the client class. Including as much detail as possible will definitely be appreciated later, so don’t be too terse. This change added a new line like this:

* 3.24 - Update rescue_instance() to take optional
rescue_image_ref

In this case, this is enough information to determine later what was changed. If multiple things were changed (multiple new arguments, changes to multiple calls, etc) they should all be listed here for posterity.

So, with this change, we have a server that can tolerate calls from older clients that don’t provide the new parameter, and a client that can make the older version of the call, if necessary. This was a pretty simple case, of course, and so there may be other changes required on either side to properly handle the fact that a parameter can’t be passed, or that some piece of data isn’t received. Here it was easy for the server to look up a suitable value for the missing parameter, but it may not always be that easy.

Gotchas and special cases

There are many categories of changes that may need to be made to an RPC API, and of course I cheated by choosing the easiest to illustrate above. In reality, the corner cases are most likely to break upgrades, so they deserve careful handling.

The first and most important is a change that alters the format of a parameter. Since the server side doesn’t receive the client’s version, it may have a very hard time determining which format something is in. Even worse, such a change may occur deep in the DB layer and not be reflected in the RPC API at all, which could result in a client sending a complex structure in a too-old or too-new format for the server to understand, and no version bump was made at all to indicate to either side that something has changed. This case is the reason we started working on what is now oslo.versionedobjects — more on that later.

Another change that must be handled carefully is the renaming or removal of a parameter. When a call is dispatched on the server side as a result of a received message, it is done so by keyword, even if the method’s arguments are positional. This means that if you change the name of a positional parameter, the server will fail to make the call to your method as if you passed a keyword argument to a python method that it wasn’t expecting. The same goes for a removed parameter of course.

In Nova, we typically handle these by not renaming things unless it’s absolutely necessary, and never removing any parameters until major version bumps. If we do rename a parameter, we continue to accept both and honor them in order in the actual implementation, the newer taking precedence if both are provided.

Version Pins

Above, I waved my hands over the can_send_version() call, which magically knew whether we could send the newer version or not. In Nova, we (currently) handle this by allowing versions for each service to be pinned in the config file. We honor that pin on the client side in the initialization of the RPC API class like this:

What this does is initialize our base version to 3.0, and then calculate the version_cap, if necessary that our client should obey. To make it easier on the operators, we define some aliases, allowing them to use release names in the config file instead of actual version numbers. So, we get the version_cap, which is either the alias based on the config, or the actual value from the config if there is no alias, or None if they didn’t set it. When we initialize the client, it gets the version that matches their alias, the version they specified, or None (i.e. no limit) if not. This is what makes the can_send_version() method able to tell us whether a given version is okay to use (i.e. if it’s below the version_cap, if one is set).

What services/APIs should be pinned, when, and to what value will depend on the architecture of the project. In Nova, during an upgrade, we require the operators to upgrade the control services before the compute nodes. This means that when they’ve upgraded from, say Juno to Kilo, the control nodes running Kilo will have their compute versions pinned to the Juno level until all the computes are upgraded. Once that happens, we know that it’s okay to send the newer version of all the calls, so the version pin is removed.

Aside from the process of bumping the major version of the RPC API to drop compatibility with older nodes, this is pretty much all you have to do in order to make your RPC API tolerate mixed versions in a single deployment. However, as described above, there is a lot more work required to make these interfaces really clean, and not leak version-specific structures over the network to nodes that potentially can’t handle them.

]]>Upgrades in Nova: The Detailshttp://www.danplanet.com/blog/2015/10/05/upgrades-in-nova-the-details/
Mon, 05 Oct 2015 17:05:20 +0000http://www.danplanet.com/blog/?p=507Continue reading →]]>Over the last few years, the Nova project has spent a lot of time on improving its story around live upgrades. We’ve made a lot of progress and done so with minimal disruption as we were figuring out how to make it work. As some of this work starts to spread in the wind and pollinate other projects, the details of how all the pieces fit together is hard to communicate. The details of how Nova pulls off some of its tricks in this area live primarily in the heads of a few people.

In OpenStack, projects are expected to maintain a minimum level of what I would call “offline upgrade-ability”. Historically that has meant things like config file compatibility between releases, such that an operator with a valid Juno config should be able to upgrade to Kilo without changes. Database schema migrations (performed offline) have generally been something we know how to do to avoid preventing someone with a large deployment from having to rebuild their data after upgrading. Careful handling of things like deprecations will soon be table stakes.

The goal of our work on live upgrades is to avoid having to take down any component of Nova for an extended period of time. Upgrades of a large cloud take time, hit roadblocks, and uncover bugs. Any phase that requires some service to be down to make a change means that if you get stuck in that phase, something isn’t working and customers are perusing your competitor’s website while they wait.

In a series of posts to follow, I hope to detail and document some of the mechanics with examples from the Nova code and provide insight to why things work the way they do. The target audience of these posts are developers in other OpenStack projects looking to follow in Nova’s footsteps. As always, not all projects will want to do things the way Nova did, and that’s fine. The details are offered here for people that are interested, but are not intended to define the way all projects should do things.

The approach taken to make live upgrades work on something as complicated as Nova is actually composed of many more specific strategies across many subsystems. It is often claimed that adopting one library or protocol will magically make upgrades work. However, in my experience, there is no silver bullet for this problem. Pulling it off requires substantial changes, most of which are in the culture. Some mechanical things need to be done at the database and RPC layers, but in the end, it requires the people writing and reviewing the code to understand the problem space and defend the project against changes that will break upgrades.

What follows this post is hopefully enough of a roadmap for other projects to wrap their heads around the complexity of the problem and start making progress.

]]>Upgrading Nova to Kilo with minimal downtimehttp://www.danplanet.com/blog/2015/06/26/upgrading-nova-to-kilo-with-minimal-downtime/
http://www.danplanet.com/blog/2015/06/26/upgrading-nova-to-kilo-with-minimal-downtime/#commentsFri, 26 Jun 2015 16:12:15 +0000http://www.danplanet.com/blog/?p=475Continue reading →]]>Starting in Icehouse, Nova gained the ability to do partial live upgrades. This first step meant that control services (which are mostly stateless) could be upgraded along with database schema before any of the compute nodes. After that step was done, individual compute nodes could be upgraded one-by-one, even migrating workloads off to newer compute nodes in order to facilitate hardware or platform upgrades in the process.

In the Kilo cycle, Nova made a concerted effort to break that initial atomic chunk of work into two pieces: the database schema upgrades and the code upgrades of the control services. It’s our first stab at this, so it’s not guaranteed to be perfect, but initial testing shows that it worked.

What follows is a high-level guide for doing a rolling Nova upgrade, using Juno-to-Kilo as the example. It’s not detailed enough to blindly follow, but is more intended to give an overview of the steps involved. It’s also untested and not something you should do on a production machine — test this procedure in your environment first and prove (to yourself) that it works.

The following steps also make some assumptions:

You’re using nova-network. If you’re using neutron, you are probably okay to do this, but you will want to use care around the compute-resident neutron agent(s) if you’re running them. If you’re installing system-level packages and dependencies, it may be difficult to upgrade Nova or Neutron packages without upgrading both.

You’re running non-local conductor (i.e. you have nova-conductor services running and [conductor]/use_local=False in your config). The conductor is a major part of insulating the newer and older services in a meaningful way. Without it, none of this will work.

Step 0: Prepare for what is coming

In order to have multiple versions of nova code running, there is an additional price in the form of extra RPC traffic between the compute nodes and the conductors. Compute nodes will start receiving data they don’t understand and they will start kicking that data back to conductor for help translating it into a format they understand. That may mean you want to start up some extra conductor workers to handle this load. How many additional workers you will need depends on the characteristics of your workload and there is really no rule of thumb to go by here. Also, if you plan to convert your compute nodes fairly quickly, you may need only a little extra overhead. If you have some stubborn compute nodes that will continue to run older code for a long time, they will be a constant source of additional traffic until they’re upgraded.

Further, as soon as you start running Kilo code, the upgraded services will be doing some online data migrations. That will generate some additional load on your database. As with the additional conductor load, the amount and impact depends on how active your cloud is and how much data needs to be migrated.

Step 1: Upgrade the schema

For this, you’ll need to get a copy of Kilo code installed somewhere. This should be a mostly temporary location that has access to the database and won’t affect any other running things. Once you’ve done that, you should be able to apply the schema updates:

$ nova-manage db sync

This should complete rather quickly as it does no invasive data migration or examination.

You should grab the code of whatever you’re going to deploy and run the database sync from that. If you’re installing from pip, use the same package to do this process. If you’re deploying distro packages, use those. Just be careful, regardless of where you do this, to avoid service disruption. It’s probably best to spin up a VM or other sandbox environment from which to perform this action.

Step 2: Pin the compute RPC version

This step ensures that everyone in the cloud will speak the same version of the compute RPC API. Right now, it won’t change anything, but once you start upgrading services, it will ensure that newer services will send messages that are compatible with the old ones.

In nova.conf, set the following pin:

[upgrade_levels]
compute = juno

You should do this on any node that could possibly talk to a compute node. That includes the compute nodes themselves, as they do talk to other compute nodes as well. If you’re not sure which services talk to compute nodes, just be safe and do this everywhere.

You don’t technically need to restart all your services after you’ve made this change, since it’s really mostly important for the newer code. However, it wouldn’t hurt to make sure that everything is happy with this version pin in place before you proceed.

I’ll also point out here that juno is an alias for 3.35. We try to make sure the aliases are there for the given releases, but this doesn’t always happen and it sometimes becomes invalid after changes are backported. This obviously is not a nice user experience, but it is what it is at this point. You can see the aliases, and history, defined in the compute/rpcapi.py file.

Step 3: Upgrade the control services

This is the first step where you actually deploy new code. Make sure that you don’t accidentally overwrite the changes you made in step 2 to your nova.conf, or that your new one includes the version pin. Nova, by convention, supports running a new release with the old release’s config file so you should be able to leave that in place for now.

In this step, you will upgrade everything but the compute nodes. This means nova-api, nova-scheduler, nova-conductor, nova-consoleauth, nova-network, and nova-cert. In reality, this needs to be done fairly atomically. So, shut down all of the affected services, roll the new code, and start them back up. This will result in some downtime for your API, but in reality, it should be easy to quickly perform the swap. In later releases, we’ll reduce the pain felt here by eliminating the need for the control services to go together.

Step 4: Watch and wait

At this point, you’ve got control services running on newer code with compute nodes running old stuff. Hopefully everything is working, and your compute nodes are slamming your conductors with requests for help with the newer versions of things.

Things to be on the lookout for are messages in the compute logs about receiving messages for an unsupported version, as well as version-related failures in the nova-api or nova-conductor logs. This example from the compute log is what you would see, along with some matching messages on the sending-side of calls that expect to receive a response:

Exception during message handling: Endpoint does not support RPC version 4.0. Attempted method: build_and_run_instance

If you see these messages, it means that either you set the pin to an incorrect value, or you missed restarting one of the services to pick up the change. In general, it’s the sender who sent the bad message, so if you see this on a compute node, suspect a conductor or api service as the culprit. Not all messages that the senders send expect a response, so trying to find the bad sender by matching up a compute error with an api error, for example, will not always be possible.

If everything looks good at this point, then you can proceed to the next step.

Step 5: Upgrade computes

This step may take an hour or a month, depending on your requirements. Each compute node can be upgraded independently to the new code at this point. When you do, it will just stop needing to ask conductor to translate things.

Don’t unpin the compute version just yet, even on upgraded nodes. If you do any resize/migrate/etc operations, a newer compute will have to talk to an older one, and the version pin needs to remain in place in order for that to work.

When you upgrade your last compute node, you’re technically done. However, the steps after 5 include some cleanup and homework before you can really declare completion and have that beer you’re waiting for.

Step 6: Drop the version pins

Once all the services are running the new code, you can remove (or comment out) the compute line in the upgrade_levels section and restart your services. This will cause all the services to start sending kilo-level messages. You could set this to “kilo” instead of commenting it out, but it’s better to leave it unset so that the newest version is always sent. If we were to backport something that was compatible with all the rest of kilo, but you had a pin set, you might be excluded from an important bug fix.

Because all of your services are new enough to accept old and new messages, you can stage the restarts of your services however you like in order to apply this change. It does not need to be atomic.

Step 7: Perform online data migrations

This step is your homework. There is a due date, but it’s a long way off. So, it’s more like a term project. You don’t have to do it now, but you will have to do it before you graduate to Liberty. If you’re responsible and mindful, you’ll get this out of the way early.

If you’re a seasoned stacker, you probably remember previous upgrades where the “db sync” phase was long, painful, and intense on the database. In Kilo, we’ve moved to making those schema updates (hopefully) lightweight, and have moved the heavy lifting to code that can execute at runtime. In fact, when you completed Step 3, you already had some data migrations happening in the background as part of normal operation. As instances are loaded from and saved to the database, those conversions will happen automatically. However, not everything will be migrated this way.

Before you will be able to move to Liberty, you will have to finish all your homework. That means getting all your data migrated to the newer formats. In Kilo, there is only one such migration to be performed and there is a new nova-manage command to help you do it. The best way to do this is to run small chunks of the upgrade over time until all of the work is done. The size of the chunks you should use depend on your infrastructure and your tolerance for the work being done. If you want to do ten instances at a time, you’d do this over and over:

$ nova-manage migrate_flavor_data --max-number 10

If you have lots of un-migrated instances, you should see something like this:

10 instances matched query, 10 completed

Once you run the command enough times, you should get to the point where it matches zero instances, at which point you know you’re done. If you start getting to the point where you have something like this:

7 instances matched query, 0 completed

…then you still have work to do. Instances that are in a transitional state (such as in the middle of being resized, or in ERROR state) are normally not migrated. Let these instances complete their transition and re-run the migration. Eventually you should be able to get to zero.

NOTE: The invocation of this migration function is actually broken in the Kilo release. There are a couple of backport patches proposed that will fix it, but it’s likely not fixed in your packages if you’re reading this soon after the release. Until then, you have a pass to not work on your homework until your distro pulls in the fixes[1][2].

Summary and Next Steps

If you’ve gotten this far, then you’ve upgraded yourself from Juno to Kilo with the minimal amount of downtime allowed by the current technology. It’s not perfect yet, but it’s a lot better than having to schedule the migration at a time where you can tolerate a significant outage window for database upgrades, and where you can take every node in your cluster offline for an atomic code deployment.

Going forward, you can expect this process to continue to get easier. Ideally we will continue to reduce the number of services that need to be upgraded together, including even partial upgrades of individual services. For example, right now you can’t really upgrade your API nodes separate from your conductors, and certainly not half of your conductors before the other half. However, that reality does exist in the future, and will allow a much less impactful transition.

As I said at the beginning, this is new stuff. It should work, and it does in our gate testing. However, be diligent about testing it on non-production systems and file bugs against the project if you find gaps and issues.

]]>http://www.danplanet.com/blog/2015/06/26/upgrading-nova-to-kilo-with-minimal-downtime/feed/5A brief overview of Nova’s new object model (Part 3)http://www.danplanet.com/blog/2013/07/20/a-brief-overview-of-novas-new-object-model-part-3/
Sat, 20 Jul 2013 01:04:42 +0000http://www.danplanet.com/blog/?p=339Continue reading →]]>In parts one and two, I talked about the reasoning for developing an object model inside of Nova, as well as showed a sample implementation for a toy object. In this part, I will examine parts of some “real” objects that are currently under development in the Nova tree.

The biggest object in Nova is (and probably always will be) Instance. It’s fairly complicated though, so let’s start with something a little simpler, such as the SecurityGroup object. Here is the field definition:

Both of these methods use a common pattern as many of the other objects, which is to query the database for the SQLAlchemy model, and then pass that to a generic function (not shown here) that constructs the new object. Both of these are decorated with remotable_classmethod, which makes them callable from across RPC and at a class level. Querying for a security group would look something like this:

Unlike the fictitious example in Part 2, there is another way to query for security group objects, which is by a collection based on some common attribute. This is often done by project ID, for example. The objects framework provides a way to easily define an object that is a list of objects, such that the list can be queried directly or over RPC in the same way, and so that the list itself contains inbuilt serialization, which handles the serialization of the objects contained within. See the SecurityGroupList object:

The first line shows that this special object is not only a NovaObject, but also an ObjectListBase, which provides the special list behavior. Note that the order of inheritance is important, so they must be in the order shown.

The ObjectListBase definition assumes a single field of “objects” and handles typical list-like behaviors like iteration of the things in the objects field, as well as membership (i.e. contains) operations. Thus, all you need to do is fill out the “foo.objects” list, like the _make_secgroup_list() helper function does:

This method simply populates the “objects” list of the SecurityGroupList object with SecurityGroup objects it constructs from the raw database models provided. It uses the same _from_db_object() helper method as the SecurityGroup object itself. You can use the result of this just like a real list:

The massive Instance object is similar to what we’ve seen in previous examples, and the SecurityGroup example above. There is a base Instance object, and an InstanceList object to provide an implementation for all the ways we can query for multiple instances at once. It’s too big to show here, but here is a subset of the field definition:

Finally, an object with some interesting fields! We see the usual integral ID field at the top, but notice that most of the other fields use “or none” helpers from the utils module. Since many of the fields in the instance can be empty (nullable=True in the database definition), we need to handle “either a string or None” in cases such as user_id. The utils module provides some helpers for datetime and ip address functions, which return datetime.datetime and netaddr.IPAddress objects respectively. Just like the int and str type functions, these take a string and convert it into the complex type when someone does something like this:

These fields with complicated data types bring us to our first concrete example of something needing special handling during serialization and deserialization. The Instance object contains methods like _attr_scheduled_at_to_primitive() and _attr_scheduled_at_from_primitive() that handle converting the datetime objects to and from strings properly. Handlers (and handler-builders) for these types are provided in the utils module. The IP address fields provide a useful example for illustration, such as this serialization method for the IPv4 address:

This gets called by the object’s serialization method when it encounters the complex IPv4 address field. Although not obvious to the layer above us, the netaddr.IPAddress object can serialize itself through simple string coercion, so we do just that. However, since the field could be None, we want to be sure not to convert that to a string resulting with the string “None” instead of None itself. Luckily, we need no special deserialization because the result of the above string coercion is sufficient to pass to the field’s type function itself, which the deserialization routine will try if no special handler is provided.

In a subsequent part, I will talk about advanced topics like lazy-loading, versioning, and object nesting.

]]>A brief overview of Nova’s new object model (Part 2)http://www.danplanet.com/blog/2013/07/12/a-brief-overview-of-novas-new-object-model-part-2/
http://www.danplanet.com/blog/2013/07/12/a-brief-overview-of-novas-new-object-model-part-2/#commentsFri, 12 Jul 2013 18:23:08 +0000http://www.danplanet.com/blog/?p=319Continue reading →]]>In Part 1, I described the problems that the Unified Object Model aims to solve within Nova. Next, I’ll describe how the infrastructure behind the scenes achieves some of the magic of making things easier for developers to implement their objects.

The first concept to understand is the registry of objects. This registry contains a database of objects that we know about, and for each, what data is contained within and what methods are implemented. In Nova, simply inheriting from the NovaObject base class registers your object through some metaclass magic:

class NovaObject(object):
"""Base class and object factory.
This forms the base of all objects that can be remoted or instantiated
via RPC. Simply defining a class that inherits from this base class
will make it remotely instantiatable. Objects should implement the
necessary "get" classmethod routines as well as "save" object methods
as appropriate.
"""
__metaclass__ = NovaObjectMetaclass

In order to make your object useful, you need to do a few other things in most cases:

Declare the data fields and their types

Provide serialization and de-serialization routines for any non-primitive fields

Provide classmethods to query for your object

Provide a save() method to write your changes back to the database

Notice that nowhere in the list is “provide an RPC API”. That’s one of the many magical powers that you get for free, simply by inheriting from NovaObject and registering your object.

To declare your fields, you need something like the following in your class:

fields = {'foo': int,
'bar': str,
'baz': my_other_type_fn,
}

This magic description of your fields describes the names and data types they should have. The key of each pair is, of course, the field name, and the value is a function that can coerce data into the proper format and/or raise an exception if that is not possible. Thus, if I set the “foo” attribute to a string of “1” the integer 1 will be actually stored. If I try to store the string “abc” into the same attribute, I’ll get a ValueError, as you would expect.

The next step is (de-)serialization routines for our attributes. Our “foo” and “bar” attributes are primitives, so we can ignore those, but our “baz” attribute is presumably something more complex, which requires a little more careful handling. So, we define a couple of specially-named methods in our object, which will be called when serialization or de-serialization of that attribute is required:

Now that our object has a data format and the ability to (de-)serialize itself, we probably need some methods to query the object. Assuming our “foo” attribute is a unique key that we can query by, we will define the following query method:

The above example papers over the part about querying the database. Right now, the objects implementations in Nova use the old DB API to do this part, but eventually, the dirty work could reside here in the object methods themselves.

Now, there is some magic here. If I am inside of nova-api (or some other part of nova with direct access to the database) and I call the above classmethod, the decorator is a no-op and the code within the method runs as you would expect, queries the database, and returns the resulting object. If, however, I am in nova-compute and I call the above method, the decorator actually remotes the call through conductor, executes the method there, and returns the result to me over RPC. Either way, the use of the object is exactly the same in both cases:

Now, before we’re done, we need to make sure that changes to our object can be put back into the database. Since a “save” happens on an instance, we define a regular instance method, but decorate it as “remotable” like this:

@remotable
def save(self, context):
# Iterate through items that have changed
updates = {}
for field in self.obj_what_changed():
updates[field] = getattr(self, field)
# Actually save them to the database
save_things_to_db_by_foo(self.foo, updates)
# Reset the changes so that the object is clean now
self.obj_reset_changes()

This implementation checks to see which of the attributes of the object have been modified, constructs a dictionary of changes, and calls a database method to update those values. This pattern is very common in Nova and should be recognizable by people used to using DB API methods.

Now that we have all of these things built into our object, we can use it from anywhere in nova like this:

One more bit of magic to note is the “sticky context”. Since you queried the object with a context, the object hides the context within itself so that you don’t have to provide it to the save() method (or any other instance methods) for the lifetime of the object. You can, of course, pass a different context to save if you need to for some reason, but if you don’t it will use the one you queried it with.

Nifty, huh? In Part 3, I will break from the world of fictitious objects and examine a real one that is already in the Nova tree, as well as fill out some of the other implementation details required.

]]>http://www.danplanet.com/blog/2013/07/12/a-brief-overview-of-novas-new-object-model-part-2/feed/2A brief overview of Nova’s new object model (Part 1)http://www.danplanet.com/blog/2013/07/12/a-brief-overview-of-novas-new-object-model-part-1/
Fri, 12 Jul 2013 17:19:27 +0000http://www.danplanet.com/blog/?p=315Continue reading →]]>As discussed at the Havana summit, I have been working with Chris Behrens (and others) on the unified-object-model blueprint for Nova. The core bits of it made their way into the tree a while ago and work is underway to implement the Instance object and convert existing code to use it. This unifies the direct-to-database query methods, as well as the mirrored conductor RPC interfaces into a single versioned object-oriented API. It aims to address a few problems for us:

Letting SQLAlchemy objects escape the DB API layer has caused us a lot of problems because they can’t be sent over RPC efficiently. The new object model is self-serializing.

Objects in the database aren’t versioned (although the schema itself is). This means that sending a primitive representation of it over RPC runs the risk of old code breaking on new schema, or vice versa. The new object model is versioned for both interface methods and data format.

Database isolation (no-db-compute) results in mirroring a bunch of non-OO interfaces in nova-conductor for use by isolated services like nova-compute. The new object model entirely hides the fact that object operations may be going direct or over RPC to achieve the desired result.

Hopefully the first two items above are fairly obvious, but the third may deserve a little explanation. Currently, we have things in the nova/db/sqlalchemy/api.py like the following:

This is more complicated than it needs to be, but basically we’ve got the instance_get_by_uuid() method at the top, which calls a couple of helpers below to build the SQLAlchemy query that actually hits the database. This is the interface that is used all over nova-api to fetch an instance object from the database by UUID, and used to be used in nova-compute to do the same. In Grizzly, we introduced a new service called nova-conductor, which took on the job of proxying access to these database interfaces over RPC so that services like nova-compute could be isolated from the database. That means we got a new set of versioned RPC interfaces such as the one mirroring the above in nova/conductor/api.py:

I’ll spare you the details, but this turns into an RPC call to the nova-conductor service, which in turn makes the DB API call above, serializes and returns the result. This was a big win in terms of security in that the least-trusted nova-compute services weren’t able to talk directly to the database, and potentially also brought scalability benefits of not having every compute node hold a connection to the database server. However, it meant that we had to add a new API to conductor for every database API, and while those were versioned, it didn’t really solve our problem with versioning the actual data format of what gets returned from those calls.

What we really want is everything using the same interface to talk to the database, whether it can go direct or is required to make an RPC trip. Ideally, services that can talk to the database and those that can’t should be able to pass objects they retrieved from the database to each other over RPC without a lot of fuss. When nova-api pulls an object with the first interface above and wants to pass it to nova-compute which is required to use the second, a horrific serialization process must take place to enable that to happen.

Enter the Unified Object Model. It does all of the above and more. It even makes coffee. (okay, it doesn’t make coffee — yet).

]]>A tool for watching Zuul and Jenkinshttp://www.danplanet.com/blog/2013/07/10/a-tool-for-watching-zuul-and-jenkins/
Wed, 10 Jul 2013 23:14:35 +0000http://www.danplanet.com/blog/?p=309Continue reading →]]>In my work on OpenStack Nova, I often have multiple patches in flight somewhere on the CI system. When patches are first submitted (or resubmitted) they go into Zuul’s “check” queue for a first pass of the tests. After a patch is approved, it goes into the “gate” queue, which is a serialized merge process across all the projects. Keeping track of one’s patches as they flow through the system can be done simply by waiting for Jenkins to report the job results back into Gerrit and/or for the resulting email notification that will occur as a result.

I like to keep close watch of my patches, both to know when they’re close to merging, as well as to know early when they’re failing a test. Catching something early and pushing a fix will kill the job currently in progress and start over with the new patch. This is a more efficient use of resources and lowers the total amount of time before Jenkins will vote on the patch in such a case.

Since Zuul provides information about what’s going on, you can go to the status page and see all the queues, jobs, etc. The problem with this is that the information from gerrit (specifically owner and commit title) isn’t merged with the view, making it hard to find your patch in a sea of competing ones.

To make this a little easier on the eyes, I wrote a very hacky text “dashboard” that merges the information from Gerrit and Zuul, and provides a periodically-refreshed view of what is going on. After contributions and ideas from several other folks, it now supports things like watching an entire project, as well as your own contributions, your own starred reviews, etc. Here is what it looked like at one point on the day of this writing:

The above was generated with the following command:

python dash.py -u danms -p openstack/nova -r 30 -s -O OR -o danms

Basically, the above says: “Show me any patches owned by danms, or in the project openstack/nova, or starred by danms, refreshed every 30 seconds”. This provides me a nice dashboard of everything going on in Nova, with my own patches highlighted for easier viewing.

Patches of my own are highlighted in green, unless they’re already failing some tests, in which case they’re red. If they are in the gate queue and dependent on something that is also failing tests, they will be yellow (meaning: maybe failing, depending on where the failure was introduced).

You can see the gate queue at the top, which has fifteen items in it, seven of which are matching the current set of view filters, as well as the jobs and their queue positions. Below that is the (unordered) check queue, which has 58 items in it. Each job shows the review number, revision number, title, time-in-queue, and the percentage of test jobs that are finished running. Note that since some jobs take much longer than others, the completion percentage doesn’t climb linearly throughout the life of the job.

The dashboard will also provide a little bit of information about Zuul’s status, when appropriate, such as when it enters queue-only mode prior to a restart, or is getting behind on processing events. This helps quickly identify why a patch might have been waiting for a long time without a vote.

If you’re interested in using the dashboard, you can get the code on github.