In an OVN deployment (with OpenStack or not), I have faced issues related to external (North/South) connectivity to/from the VMs and most of the time it is misconfiguration in the OVN databases. So I thought of writing this post.

I assume that the reader is familiar with the basic OVN architecture. Please see at the end of the post which has links to some of the tutorials and blog posts on OVN.

This can be again configured as HA or non HA. If HA is enabled then the gateway router port is scheduled on multiple chassis with one acting as master. If it fails for some reason then the other chassis will take over.

In this blog post I will concentrate on the logical gateway router port with no HA. In the next blog post, I intend to cover the logical gateway router port with HA scenario.

What does scheduling mean here ? It means the chassis which is selected to host the gateway router port provides the centralized external connectivity. The north-south tenant traffic will first be redirected to this chassis and it acts as a gateway.

I will take OpenStack as an example here. Let’s say you have a private network “private” with subnet – 172.168.0.0/24 and a VM port is created with IP – 172.168.0.12. The private network is attache to a neutron router “r1” and a gateway is added to it.

Step 1. Get the list of chassis in your deployment

In OVN terminology, chassis is nothing but a node where ovn-controller service is running. ovn-controller service running on each chassis connects to the south bound database and an entry is created for each chassis in the southbound db.

Step 2: Verify ovn-bridge-mappings on all your chassis’s.

Make sure that ovn-bridge-mappings are configured in your chassis.

In order for a chassis to provide external connectivity, ovn-controller expects “ovn-bridge-mappings” to be configured. You can verify ovn-bridge-mappings settings by running the below command in the chassis.

#ovs-vsctl get open . external_ids:ovn-bridge-mappings
"datacentre:br-ex"

In my case it returns “datacentre:br-ex”. Please see http://openvswitch.org/support/dist-docs/ovn-controller.8.html and grep for ovn-bridge-mappings for more information about it. In case the above command returns error and you want that chassis to provide external connectivity, then configure ovn-bridge-mappings by running

#ovs-vsctl set open . external_ids:ovn-bridge-mappings=”datacentre:br-ex”

“datacentre:br-ex” is just an example. Also create the ovs bridge “br-ex” if not present.

Step 3: Get the scheduled chassis of the gateway router port

Next step is to figure out where the gateway router port is scheduled. The chassis on which the gateway router port is scheduled acts as the gateway for the tenant traffic.

First get the name of the logical router gateway port by running the below command. 10.0.0.102 happens to be gateway ip attached to the router in my case. You can figure it out by running “openstack router show r1”.

If you look into the options column, you will see that the gateway port is scheduled on the chassis “116e3e4f-3ae1-4788-a300-b902b019530b” which is “overcloud-controller-0.novalocal” in my case. You will see another option “gateway_chassis”. If that is set, then the gateway port is scheduled on multiple chassis with HA configured. Let’s assume “gateway_chassis” column is empty for now. In case “options” column is empty it means the gateway router port is not scheduled. In the case of OpenStack this should not happen. In the case of other CMS’s (cloud management system) it is expected that this column is set by CMS. You can schedule it manually. See step 4

Step 4: Schedule the gateway router port if required

This step will be required either if “options” column was empty in step 3 or the gateway router port was scheduled on a chassis which doesn’t provide external connectivity. So you want to reschedule it to another chassis which provides external connectivity. Select a chassis where you want to schedule. Make sure that it has ovn-bridge-mappings configured. If you are facing the external connectivity issue with your tenant traffic, then this is most likely the cause and you need to fix it here.

Let’s say you want to select the chassis 58e05e13-bc58-4afc-b975-88b13c9b38cf (overcloud-controller-1.novalocal).

Following the above steps should provide external connectivity to your tenant traffic. If it still doesn’t work, most likely it is a bug in OVN. Please report it to the OVS mailing list <dev@openvswitch.org>.

Conclusion

In this blog post we saw how to inspect the OVN databases to figure out the issue if your external connectivity is broken for your tenant traffic. In the next blog post we will see how to fix issues for HA scenario.

Recently native DHCP support has been added to OVN. In this post we will see how native DHCP is supported in OVN and how it is used by OpenStack Neutron OVN ML2 driver. The code which supports native DHCP can be found here and here.

Please see this to understand the architecture of OVN and the services of the OVN. To brief, OVN has a service called ovn-northd which generates the logical flows based on the OVN northbound database state. OVN northbound database is populated by the OVN ML2 neutron driver. OVN has another service called ovn-controller which is run on each compute host. ovn-controller translates the logical flows generated by ovn-northd into OpenFlow flows and adds these flows into the integration bridge (br-int) managed by the local ovs-vswitchd instance.

I recommend reading this blog and this as I found them to be very useful along with the ovn-architecture man page if you are curious to know more about OVN.

OpenStack Neutron supports DHCP and provides the IP addresses to the VMs using the Neutron dhcp agent. The dhcp agent can be configured to run on multiple nodes. When a VM boots up it sends DHCP discover broadcast packets which is received by the “dnsmasq” (spawned and configured by dhcp agent) which looks its configuration and send the DHCP reply packet with the appropriate IP address. In the OpenStack world, the IPv4 addresses are assigned when the neutron port is created. So it becomes easier to send the DHCP reply packet with the appropriate IPv4 address.

Until the native DHCP support was added into OVN, we were relying on the dhcp agent to support DHCP in the OpenStack environment.

Advantages of having native DHCP support are

We don’t need to rely on dhcp agent. So no namespace and ‘dnsmasq’ instance for each virtual network is needed.

With the dhcp agent approach, it is not completely distributed. In cases where the dhcp agent is down, the VMs might not get the DHCP replies.

It is completely distributed. ovn-controller running in each compute node handles the DHCP requests from the VMs hosted locally which makes the DHCP support in OVN distributed.

A little about Continuations feature of OVS

Native DHCP is supported using an OVS feature called “Continuations”. This feature which will be available in OVS 2.6. release. Please see here for detailed information.

“Continuations” provides an OpenFlow action called “NXT_PACKET_IN2” which has a flag called “pause” and a field called “userdata”. With the “pause” flag set, the controller when receives a packet as packet-in, can inspect the packet and modify it if required and can send the packet back to the switch which will resume the pipeline from the point where it was interrupted.

Native DHCP details

Lets see some details on how native DHCP is supported in OVN.

When ovn-controller receives a DHCP request packet, in order to send a DHCP reply

It needs to know the IPv4 address to be offered

The DHCP options to be added in the DHCP reply packet.

OVN Northbound database has a new table called “DHCP_Options” which is used to define the set of DHCP options. In the Logical_Switch_Port table a new column called “dhcpv4_options” is added which refers to the DHCP_Options rows. In order to make use of native DHCPv4 feature, the CMS (Cloud management system) is expected to define DHCP options for each of the logical ports.

ovn-northd then adds logical flows to send the DHCP replies for each logical port which has an IPv4 address and DHCP options defined. ovn-northd adds two new stages in the ingress logical pipeline – “ls_in_dhcp_options” followed by “ls_in_dhcp_response” where these logical flows are added.

Let’s say we have a logical port with name “port1” configured with IPv4 address – “10.0.0.4” and the following DHCP options defined – lease_time=”43200″, mtu=”1442″, router=”10.0.0.1″, server_id=”10.0.0.1″ and server_mac=”fa:16:3e:96:22:da”.

The OVN action “put_dhcp_opts” transforms the DHCP request packet into a reply packet, adds the DHCP options defined and stores 1 in the ovs register reg0 bit 3. If the packet is invalid, it leaves the packet unchanged and stores 0 in the ovs register reg0 bit 3.

In order to understand how this action transforms the DHCP request packet into the reply packet, lets see the corresponding OF flow.

As you see above, the action “put_dhcp_opts” translates into controller action with “pause” flag set and the DHCP options stored in the “userdata” field.

When a DHCP request packet is received, ovs-vswitchd sends the packet to “ovn-controller”. “ovn-controller” receives this packet, extracts the offer ip and the DHCP options from the “userdata”, frames a DHCP reply packet and adds these DHCP options, stores 1 in the ovs register field bit and sends the packet back to the switch. How would ovn-controller know which register to use ? It is also stored in the “userdata” field.

On receiving the packet back, the ovs-vswitchd resumes the packet and executes the next pipeline “ls_in_dhcp_response”.

The OVN actions “outport = inport; flags.loopback = 1; output;” in the “ls_in_dhcp_response” pipeline causes the reply DHCP packet to be delivered to the VM port(which sent the DHCP request packet).

Below diagram depicts the sequence of actions when the VM sends a DHCP request packet.

OpenStack Neutron OVN ML2 driver

The neutron OVN ML2 driver makes use of this feature. In order to use native DHCP, the configuration option “ovn_native_dhcp” should be set to True in the ML2 plugin configuration file.

OVN ML2 driver creates the DHCP_Options row for every subnet and defines the DHCP options. All the ports of the subnet will refer to the DHCP_Options row. It also supports the extra DHCP options if defined for any port. Please see here to get an overview of how native DHCPv4 is used in neutron.

Native DHCPv6 support in OVN

Patches to support DHCPv6 are submitted for review in the OVS dev mailing list. Once they are reviewed and accepted, OVN will have native DHCPv6 support. This feature will be really useful once OVN supports IPv6 Router Advertisements. OVN already supports IPv6 routing. Patches are up for review to support IPv6 RAs.

Limitation of using native DHCP support

OVN still doesn’t have native DNS support for internal DNS queries. If support for internal DNS is a requirement in the OpenStack deployments, then the dhcp agent needs to be used.

To have metadata support, dhcp agent would be still required. There is a patch in networking-ovn which supports VM metadata access using native DHCP which can be found here.

We know that all software components and services generate log files. These log files are vital in troubleshooting and debugging problems. If the log files are not managed properly then it can be extremely difficult to get a good look into them.

Although system administrators cannot control the generation of logs, they can achieve some level of log management by

having log rotators to get rid of the old log files.

using syslog to catch alerts.

archiving logs etc.

OpenContrail has several components, many of which can generate logs as well as store them in the log files. OpenContrail also provides the mechanism to configure the logging, so that the system administrators / DevOps can define the logging parameters to suite their own requirements.

In this blog post we will see logging support in OpenContrail components and what are the logging configuration mechanisms supported by it.

OpenContrail uses Sandesh protocol which provides the mechanism to exchange messages between various OpenContrail components. It also provides the functionality of logging those messages and the logs into the log files. You can read more about Sandesh in this great article

Logging can be configured by :

choosing the log file

selecting the log file size

defining custom formatters/loggers

using syslog etc.

OpenContrail has mainly Python components and C++ components.

Python components of OpenContrail are :

contrail API server

schema transformer

SVC monitor

discovery server

analytics Op server

C++ components of OpenContrail are :

contrail vrouter

contrail controller

Query engine

contrail analytics server

contrail DNS

C++ components of OpenContrail use log4cplus for logging and python components use python logging.

OpenContrail versions

The configuration mechanisms defined in this post are supported by the master version of OpenContrail.

You need to cherry pick the below patches if you are using R2.2 or R2.1 version as these patches are still not merged yet.

Similar to the python logging configuration file, you can define a log configuration file for the C++ components and give the path of the configuration file in the ‘log_property_file’ configuration parameter. When ‘log_property_file’ is defined, other logging parameters are ignored by the OpenContrail C++ components. log4cplus uses the term property file for the log configuration file.

Conclusion

You’ve now hopefully seen how logging is supported in OpenContrail and how you can define your own custom logging configuration files. With this knowledge, it should be possible for system admins/DevOps to manage the log files properly and help them quickly and efficiently troubleshoot problems.