Artom Lifshitz's perso-fessional writings, probably about OpenStack

Primary Menu

Now that Nova’s device role tagging feature talked about in a previous blog post is getting some real world usage, I’m starting to realise that it’s woefully under-documented and folks are having some misconceptions about what it is and how to use it.

Let’s start with an example. You boot a VM with 3 network interfaces, each for a different purpose, each connected to a different virtual network:

Great, your 4 network interfaces are there. The second one has an IP address. Therefore, eth1 must be your management interface because that’s the only network you have DHCP running on. However, you can’t tell eth0, eth2, and eth3 apart and you don’t know which one is for public data, which one is for private data, and which one is for the robot apocalypse.

That’s an important point to reiterate: if you have a VM with multiple network interfaces, the order in which they appear in the guest OS does not necessarily reflect the order in which they were given in the server boot request. In our example, the management interface was given second to last, but ends up as the second interface in the guest OS, eth1.

So we’re back to our problem: how do we tell eth0, eth2 and eth3 apart – or more generally, how does the guest OS know which network interface is which if DHCP is not enabled for some of them? This is solved with device role tags.

Let’s go back to our example, and boot the same VM with device role tags applied to the network interfaces:

Each element in the devices array corresponds to one of the network interfaces that we’ve tagged when we booted the VM. Each device element contains our tag, but also other information about the device, such as PCI and MAC addresses. Using those, we can cross-reference with the output of ifconfig and figure out which network interface is which. For instance, we know that the network interface tagged with public has the 00:00:00:00:00:5e MAC address. Therefore, eth3 is the public data interface. Similarly, the interface tagged with pvt has the 00:00:00:00:00:af MAC address. Therefore, eth0 is the private data interface.

You’ve noticed by now that there are only 3 devices in the metadata we’ve received from the metadata API. Remember how we didn’t tag the Skynet uplink interface? Devices that aren’t tagged don’t appear in the metadata. The array is called devices but it should have been more accurately called tagged_devices. It makes sense to only include tagged devices since every other piece of information is already known to the guest OS. Let’s pretend we include the Skynet uplink interface in the devices array:

There is nothing here that we can’t already find out with lspci or ifconfig, and it doesn’t help us in any way.

If the metadata API is not available, the config drive can be used. The JSON document returned by our previous curl command can also be found at openstack/latest/meta_data.json on the config drive.

Volumes (the –block-device parameter to the nova boot command) can be tagged in much the same way as network interfaces. The device tagging metadata is slightly different than for network interfaces, but the idea is the same: show the guest OS the device tag, along with other information it can use to figure out which disk the tag applies to. For example, we can boot a VM with two volumes:

Device tagging in OpenStack Nova is a mechanism to communicate to the guest OS the intended usage of virtual network interfaces and disks. For example, if an instance has two virtual network interfaces, one connected to a public network and the other to a private management network, the interfaces can be tagged with ‘pub’ and ‘pvt’ respectively. An application in the guest OS can fetch the tags and provision each interface accordingly.

With python-novaclient, in order to boot an instance with tagged devices, use the tag key in the –nic and –block-device arguments to the nova boot command. For example:

As a reminder, an instance can access metadata in two ways. The first is to curl http://169.254.169.254/openstack/latest/meta_data.json. The second is to look on the configdrive under openstack/latest/meta_data.json (if the configdrive is enabled). In the above example, a devices section will appear in the metadata, looking something like this: