OpenStack: Adding External Networks to Neutron with GRE

Hello All – today’s post is on Neutron using GRE Networking. The all-in-one OpenStack gives you GRE by default but only a single external subnet. Read on to learn how to add more external networks!

Use this section on just your Neutron Controller(s) to add more external networks. “External networks” allow you to hook up functions running for a specific tenant to the outside world (like a Web server). Typically, this external network will be your DMZ; for example, if you have a small shop with a single after-market router connecting you to the network you probably have 192.168.1.0/24 for your NAT’ed network connected to the Internet. In that case, if you run an OpenStack then any tenant functions that you want to get to from the outside world, will need to have a dedicated IP address on that 192.168.1.0/24 network. Thus, your “external network” would be 192.168.1.0/24 and you would allocate IP addresses on that subnet; one for each public function. Then you would either do port redirection (PAT) or Layer 7 name redirection from your router to funnel inbound requests from the Internet to your backend VMs.

The problem with this? In an OpenStack hosted model, you really want to keep your tenant networks completely separated from your infrastructure / management networks. (So you don’t just want to have a single “external network” that everything connects to; you want to have multiple external networks that you can isolate by function. For example, you may want to segment tenants such that their “public” external IP addresses are truly isolated from each other just like the tenant-specific networks if we want to have VMs run inside OpenStack that connect only to a protected subnet rather than to the default DMZ network. In my case, I have a number of internal functions such as Active Directory, DNS forwarders, SMTP servers, and so on that are truly separate and distinct from the Guest VMs I host for my customers. Thus, it makes sense to keep those IP addresses (and traffic!) completely segmented away from any DMZ used by my customers.

I’ve chosen to give my VM tenants a nice big 172.20.64.0/18 subnet (16,382 IP addresses); that is the “external network” I registered with Neutron. However, I also want to have a 172.24.4.0/22 subnet for my management functions. (These networks are already VLAN’ed off from each other.) And – I may want to add more subnets in the future. So this article was a great way for me to solve these problems and come up with a scalable, extensible, and secure solution that works fine with Neutron and GRE networking.

I found some great articles on this topic and you are wise to check them out (and do your own research):

The Problem: A Single External Subnet

The problem is that Neutron GRE by default only gives you a single external subnet. Consider my example lab: I setup my Neutron Controller with four (4) network interfaces. However…I used only three of those interfaces:

one for internal Management (VLAN OpenStack-104) communication over 172.28.0.0/22

one for internal GRE traffic (VLAN Guest-VMs-010-120) over 172.20.64.130/22

one for external Tenant VM access (VLAN DMZ-106) over the big 172.20.128.0/18 network

I didn’t use the fourth interface, which is the “Infrastructure-103” network 172.24.0.0/22 (VLAN 103). So let’s configure that fourth interface as a second external network for our Neutron environment. That way, I can run my DNS, AD controllers, SMTP servers, Nagios monitors, etc. within my virtualized environment without having any data leaks between my internal network and the Tenant VMs.

The Process

Keep in mind that the basic network setup we already performed leverages the br-ex interface, which is bridged to DMZ-106. This is perfect for hosted tenants but not so good if we want to host internal VMs that we manage ourselves (such as an SMTP server). We want to keep our traffic separate from our hosted customers’ traffic. To do this, we need to specify additional L3 agents. Follow these steps:

When you are adding the First Additional External Subnet (First-Time Only), do these initial steps to make the existing L3 agent context aware of its mapped external network:

Add the new interface to the Neutron system. For this paper, we already did this with our fourth interface we created from the Infrastructure-103 VLAN. But for additional networks (for example, if we wanted to host a VM on the Management-102 VLAN), we’d need to add another network. I’m using the KVM hypervisor to run my OpenStack VMs, so this was simply using virsh edit to add another NIC.

Determine the Neutron network ID of the existing external network e.g. ext-net for this paper:

Modify networking scripts to drive the openvswitch configuration during boot. Note I’m using jumbo frames here but that is probably not necessary (jumbo frames are beyond the scope of this article, search on my site for other articles on when to use them):

Note: Above, I’m giving an IP address not to the physical interface (eth3 on the Neutron Controller VM) but to the bridge (switch) I’m creating on that physical interface. The only reason I’m even doing this is so that I can easily verify whether the bridge is up from a trusted remote controller (to enable Nagios monitoring). Otherwise, you don’t even need an IP address on the bridge any more than you “need” an IP address on any type of switch device.

Restart networking:

service network restart

Get the OpenStack Neutron network UUID; we do this by creating a new Neutron external network. Remember that simply creating the network does not allocate any IPs, now setup any subnets, nor run any routers. It simply creates a record in the Neutron database and we need the UUID of that record.

That being said, you want to spend a few minutes and create the network the way you want it. Consider that it’s not typical for our tenants (hosted customer VMs) to get their own DMZ network space; instead they get their own private tenant networks and they share DMZ space with other tenants. This model isn’t good enough if you truly want all tenant traffic isolated from other tenants. In our case, where we have an “Infrastructure” set of VMs, we want to have an “Infrastructure” tenant that will be hooked up to our infrastructure-103 VLAN on the outside (the tenant-specific DMZ) and to the standard private tenant network on the inside (which is the way that GRE networks operate in Neutron). So follow these general sub-steps:

Create the tenant for which you want isolated DMZ networking. In our case, this is the “infrastructure” tenant:

I ran into problems with this step and had to review my steps carefully (including resorting to invoking strace within the /etc/init.d/neutron-l3-agent-103 script I created) so start the service now, review logs carefully, and *reboot the system* to prove that everything comes up as you expect. Then continue to the next step.

Create your new subnet and router for your tenant as per usual. I’ll cover those basic steps in another article, but there are *plenty* of resources available for this on the Internet.

Verify the Setup

Let’s take a look at output from what I expect with our paper’s use case as the template:

First, I verify that all interfaces up and running. In my case, this is the eth3 and the br-ex-103 interfaces:

That last output is important; the OpenStack Neutron router ID is what the Neutron L3 agent uses to create a Network Namespace to permit separate NAT rules and external routing for the Infrastructure tenant. Let’s verify that the appropriate namespace was created:

That looks good, we have our namespace so all of our hooks and maps are working thus far. Let’s now issue a command to verify that the Neutron L3 agent actually created the IP interfaces within that namespace for our router:

Hi Andrew. Did you experience some issue when a user’s request arrives (e.g: request of floating Ip?

In my case, the two L3 agents are listening qpid messages, and only one message is created in qpid when a user does a request (for example, assigning a floating IP). In this case, that message(request) should be processed by the specific L3 agent which deals that floating IP range. But it could not happen. Indeed the issue is that the request could be treated by the other L3 agent and obviously does not apply the request over the router correctly.

I’m temporally solving this restarting the L3 agents, but it’s not a good choice.