Lorin Hochstein » openstackRamblings about software, research and other things2015-03-03T22:34:18Zhttp://lorinhochstein.wordpress.com/feed/atom/WordPress.comLorinhttps://lorinhochstein.wordpress.comhttp://lorinhochstein.wordpress.com/?p=3462013-11-30T17:25:20Z2013-11-30T17:23:39Z]]>Here’s a modest proposal: a program to pair up individual OpenStack developers with OpenStack operators to encourage better information flow from ops to devs.

Heres’s how it might work. Operators with production OpenStack deployments would indicate that they would be willing to occasionally host a developer. The participating OpenStack developer would travel to the operator’s site for, say, a day or two, and shadow the operator. The dev would observe things like the kinds of maintenance tasks the op was doing, the kinds of tools they were using to do so, and so on.

After the visit was complete, the dev would write up and publish a report about what they learned, focusing in particular on observed pain points and any surprises that the dev encountered about what the operator did and how they did it. Finally, the dev would submit any relevant usability or other bugs to the relevant projects.

You could call it “Adopt an Op”. Although “Adopt a Dev” is probably more accurate, I think that the emphasis should be on the devs coming to the ops.

]]>1Lorinhttps://lorinhochstein.wordpress.comhttp://lorinhochstein.wordpress.com/?p=3402013-10-17T19:52:44Z2013-09-05T02:37:02Z]]>Rackspace is now running a developer discount, so I thought I’d give them a try. Once I signed up for the account and got my credentials, here’s how I got up and running with the command-line tools. I got this info from Rackspace’s Getting Started guide.

Second, I had to ssh as root to the ubuntu instance, not as the ubuntu user. In fact, the Ubuntu 13.04 image I booted doesn’t seem to have cloud-init installed, which surprised me. I’m not sure how the image is pulling my public key from the metadata service.

EDIT: I can’t reach the metadata service from the instance, so I assume that there is no metadata service running, and that they are injecting the key directly into the filesystem.

]]>3Lorinhttps://lorinhochstein.wordpress.comhttp://lorinhochstein.wordpress.com/?p=3292013-08-23T22:23:20Z2013-08-18T02:03:19Z]]>If you’re interested in trying out DevStack, I wrote up some scripts for automatically deploying DevStack inside of a VirtualBox virtual machine using Vagrant: devstack-vm.

In a few minutes, you’ll have a running version of DevStack, configured with Neutron. You can even reach your instances with floating IPs without having to ssh to the VirtualBox VM first. If you want to automatically boot a Cirros instance and attach a floating IP, just run the included Python script which uses the OpenStack Python bindings:

$ ./boot-cirros.py

Edit: Added a line to chmod the private key

]]>3Lorinhttps://lorinhochstein.wordpress.comhttp://lorinhochstein.wordpress.com/?p=3222013-06-26T03:00:38Z2013-06-26T03:00:38Z]]>I have an article about the OpenStack Python APIs up on developerWorks.]]>0Lorinhttps://lorinhochstein.wordpress.comhttp://lorinhochstein.wordpress.com/?p=2962013-05-30T02:36:49Z2013-05-30T02:36:49Z]]>I can’t connect to the instance.

The Ubuntu precise and raring images I had downloaded from the Ubuntu website worked just fine. I was able to boot them cleanly inside of our new OpenStack deployment and ssh into them. But when I tried to launch a CentOS 6.4 instance, something went wrong. It booted up just fine, but the networking wasn’t working properly.

The guest’s console log revealed that the instance was not able to obtain an IP address via DHCP. Doing a tcpdump on the vif of the compute host confirmed that the DHCP reply packet was reaching it.

I saw the DHCP request packets go out, but I didn’t see the DHCP reply packet. Somehow, that packet wasn’t making it across from the host to the guest.

Some Googling revealed that a DHCP failure may occur because of a missing checksum in one of the DHCP packets, and adding a rule to the iptables mangle table can resolve the issue:
iptables -A POSTROUTING -t mangle -p udp --dport bootpc -j CHECKSUM --checksum-fill

I added this rule to the compute host and to the network controller, but the problem remained.

If the DHCP reply packet was getting all of the way to the vif, it seemed like the overall OpenStack setup was ok. Brian had created this CentOS image manually, maybe there was something wrong with it? (It’s always the other guy’s fault, right?) I tried downloading another CentOS image from the links in the rackerjoe/oz-image-build github repository: same problem. I tried creating CentOS a CentOS 6.4 image from scratch, both manually and using Oz. All had the same issue: they booted just fine, but they weren’t able to get IP addresses via DHCP.

There was a suspicious message in the instance boot log.
eth0: IPv6 duplicate address fe80::f816:3eff:fe72:f86d detected!

I tried disabling IPv6 inside of the image, and turning off hairpin mode on the compute host, but that didn’t do it either. I tried turning virtio on and off, and loading and unloading the vhost_net module in the compute host. No effect.

It’s working, I’m getting replies. Next, statically configure the CentOS instance with an IP address of 10.40.0.5 and try to arping it. The compute host is c220-2, we’ll do a tcpdump on the vif at the same time.

It works! This is peculiar. I can arping from the host to the guest, but I can’t arping from the network controller to the guest. I know that the packets are getting across the switch, because I see them when using tcpdump in the compute host.

Is it because of VLAN tagging? I’m running flat mode, there shouldn’t be any VLAN tags added to the packets. Maybe it’s a switch configuration issue?

I check the configuration of the switch, a Cisco Nexus 3000. From what I can tell, it’s not configured to do VLAN tagging. There’s only one VLAN configured, and that’s the native VLAN. I’ve got vlan dto1q tag native disabled. I ask on Server Fault, and it turns out that 46 bytes is the minimum user data permitted in an Ethernet packet, so it’s normal that padding is added before the packet gets sent over the network.

But, it seems like the problem only occurs when that DHCP packet crosses the switch. What happens if I spoof the ARP request packet, but send it from the compute host instead of the network controller? More googling reveals a command-line tool called “packit” that can spoof packets like this.

When I use packit to spoof the network controller ARP request, the CentOS guest replies properly.

The real and spoofed packets appear to be identical. The only difference is that one originated from the network controller, and the other originated from the compute host. And, yet, different responses. Maybe the packets are somehow different, but aren’t being revealed by tcpdump?

Brian’s convinced that it’s a VLAN issue, and that tcpdump isn’t telling the whole story. I try loading the 8021q module inside the guest. When I do this, and I configure a static IP address, then networking works! I can ping from the network controller to the guest. But DHCP still isn’t working.

Next, I try scapy for doing the spoofing. I write a quick scapy script that runs on the compute host, listens for the DHCP packets sent by the network controller, and re-transmits them. I know there are two DHCP packets that will be sent by the controller (DHCPOFFER, DHCPACK), so I do this twice:

When the script runs, the instance revceives an IP address! I’m just sniffing the packet and transmitting it again, without modifying it. Why would this work?

On the openstack-operators, Joe suggests doing:

tcpdump -i eth0 -XX -vv -e

on the CentOS guest instance, with 8021q module not loaded and loaded.

This is the first time I’ve run tcpdump inside of the instance without restricting the output by, say, port, or protocol. I don’t see any difference between when the module is loaded or not, but I do see this:

It says “vlan 0, p 0″. What’s VLAN 0? If I do the same tcpdump on the compute host, it makes no mention of “vlan 0, p0″.

It turns out that these are 802.1p packets, which have priority information in them: something was adding these priority tags when the packets were moving from the network controller to the compute hosts. When using 802.1p, if there’s no VLAN tag, then the convention is to put 0 there. The Linux kernel running in the host (3.2.0) handles these properly, but apparently the Linux kernel in the guest (2.6.32) doesn’t handle it.

In the end, I switched the OpenStack configuration VLAN manager so that the compute host would explicitly remove VLAN tags before passing the packet before passing them in to the guest. It resolved the issue.

This was a rare case, a problem that arose because of the combination of the host kernel, guest kernel, and the NIC configuration. But it illustrates how difficult it is to track down OpenStack networking problems, and how hard it can be to assist someone who cries out, “Help, I can’t connect to my instance!”

]]>4Lorinhttps://lorinhochstein.wordpress.comhttp://lorinhochstein.wordpress.com/?p=2792013-03-21T19:22:53Z2013-03-21T19:22:53Z]]>Because “cloud” has become such a buzzword, it’s tempting to dismiss cloud computing as nothing new. But one genuine change is the rise in software designed to work in an environment where hardware failures are expected. The classic example of this trend is the Netflix Chaos Monkey, which tests a software system by initiating random failures. The IT community calls this sort of system “highly available”, whereas the academic community prefers the term “fault tolerant”.

If you plan to deploy a system like an OpenStack cloud, you need to be aware of the failure modes of the system components (disk failures, power failures, networking issues), and ensure that your system can stay functional when these failures occur. However, when you actually deploy OpenStack on real hardware, you quickly discover that the component that is most likely to generate a fault is you, the operator. Because every installation is different, and because OpenStack has so many options, the probability of forgetting an option or specifying the incorrect value in a config file on the initial deployment is approximately one.

And while developers now design software to minimize the impact due to a hardware failure, there is no such notion of minimizing the impact due to an operator failure. This would require asking questions at development time such as: “What will happen if somebody puts ‘eth1′ instead of ‘eth0′ for public_interface in nova.conf? How would they determine what has gone wrong?”

Designing for operator fault tolerance would be a significant shift in thinking, but I would wager that the additional development effort would translate into enormous reductions in operations effort.

]]>2Lorinhttps://lorinhochstein.wordpress.comhttp://lorinhochstein.wordpress.com/?p=2712013-03-06T03:00:26Z2013-03-06T03:00:26Z]]>We wrote a book in a week. If you do anything that involves OpenStack, check it out.]]>0Lorinhttps://lorinhochstein.wordpress.comhttp://lorinhochstein.wordpress.com/?p=2342012-08-13T00:53:28Z2012-08-13T00:53:28Z]]>I did a talk last week at the OpenStack DC user group on using Ansible to write OpenStack deployment scripts, and using Vagrant for testing. The talk is called Vagrant, Ansible and OpenStack on your laptop. The Ansible scripts and Vagrant config file are on github.]]>0