Announcing EVPN for scalable virtualized networks

This post was updated on 2/22/17 to reflect the official launch of EVPN. You can now access EVPN in general availability. Read the white paper to learn more about this exciting new feature.

When we set out to build new features for Cumulus Linux, we ask ourselves two questions: 1) How can we make network operators’ jobs easier? And 2) How can we help businesses use web-scale IT principles to build powerful, efficient and highly-scalable data centers? With EVPN, we believe we nailed both.

Why EVPN?

Many data centers today rely on layer 2 connectivity for specific applications and IP address mobility. However, an entire layer 2 data center can bring challenges such as large failure domains, spanning tree complexities, difficulty troubleshooting, and scale challenges as only 4094 VLANS are supported.

Therefore, modern data centers are moving to a layer 3 fabric, which means running a routing protocol, such as BGP or OSPF between the leaf and spine switches. In order to provide layer 2 connectivity, between hosts and VMs on different racks as well as maintain multi-tenant separation, layer 2 overlay solution is deployed such as VXLAN. However, VXLAN does not define a control plane to learn and exchange MAC addresses. Therefore, VXLAN cannot, by itself, deal efficiently with host mobility and fast convergence. Practical deployments of VXLAN without a control plane often use a controller.

Traditional bridging and L2VPN solutions like VPLS and native VXLAN retrieve host location via the data plane, meaning it learns the location of the hosts based on the data frames that pass through the switch. When the bridge receives a frame on a switchport, it will take note of its source MAC address and add it, along with the received port, to the switch’s MAC table. This MAC address table directs traffic to the correct destination. See below.

The same data plane learning happens over the MPLS (or other) tunnel also as there is no control plane protocol to exchange the MAC addresses learned on the local switch ports with the remote PEs or VXLAN Virtual Tunnel End Points (VTEPs). Learning solely via the data plane has its own set of limitations, such as limited redundancy, no per-flow load balancing, lack of traffic engineering, and slow convergence.

How does EVPN help me?

Cumulus Networks EVPN is a next-generation control-plane solution for VXLAN tunnels that uses the BGP routing protocol to provide high scale, redundancy, traffic engineering, multi-tenant separation, and fast convergence for host and VM mobility — all while interoperating between vendors. BGP is a well known, mature routing protocol that powers the Internet. It is very robust and scalable. The Multiprotocol BGP (MP-BGP) extensions allow BGP to carry all kinds of information by introducing support for address-families and sub-address-families. All of the above characteristics make BGP very attractive to use for a VXLAN control plane.

In early 2017, Cumulus Quagga will natively contain the new BGP address-family that was created by the IETF for EVPN — the EVPN address-family, as seen below.

The MP-BGP EVPN address-family advertises MAC addresses learned on a switch to the remote VTEPs while providing for multi-tenant separation and overlapping addresses. Advertising MAC addresses via BGP allow remote switches to dynamically learn the location of hosts or VMs without requiring data packets to traverse the VXLAN tunnel first. After the MAC address is learned on one switch, then all peered switches will learn it as well, which can reduce broadcast traffic in the network and allow fast convergence when a host or VM moves to a different rack.

For example, in a setup with three leafs and three hosts, Host 1’s MAC address would be learned by leaf01, host 2’s address by leaf02, and host 3’s address by leaf03. This often happens as soon as the link between the leaf and the host comes up, through Gratuitous ARP (GARP). At this point, each leaf would advertise its locally learned MAC address to the other leafs via BGP. Now, all of the leafs have all of the hosts MAC addresses and their locations without requiring a data packet to traverse the switch first.

Using the above setup as an example, the command sh evpn vni 10100 mac outputs all the MAC addresses and their locations associated within that VNI, or VXLAN tunnel. For example, EVPN Table for leaf01 shows:

Whereas the table for leaf02 similarly shows:

If host 1 then moves to rack 3, leaf03 will learn of the new location and immediately update the other two leafs of the move. It is not necessary to wait for MAC table timeouts or a new data packet sent by the host for data center wide convergence.

Can you summarize the benefits of deploying EVPN?

Cumulus EVPN provides many benefits to a data center, including:

Controller-less VXLAN: No controller is needed with EVPN, as it enables VTEP peer discovery through BGP.

Scale and Robustness: EVPN uses the standard BGP routing protocol for the control plane. BGP is a mature well-known protocol that powers the internet. For data centers that already run BGP, this involves just adding another address-family.

Fast convergence/mobility: The BGP EVPN address family includes features to track host moves across the datacenter, allowing for very fast convergence.

Multi-vendor interoperable: Since EVPN is a standard, it will be interoperable with other vendors that adhere to the standard.

Support for Active/Active VxLAN: Cumulus EVPN supports host redundancy to switch pairs with an MLAG configuration.

Share this blog post!

Diane is a Technical Marketing Engineer with Cumulus, working with product management and sales to help influence and drive new technologies and solutions into Cumulus Linux. She also evangelizes new networking technologies and solutions in Cumulus Linux by writing white papers, presenting at webinars, and creating videos. Diane has 20 years experience in the networking industry (optical DWDM, layer 2 and layer 3), holds CCIE #2537 Emeritus, and has a B.S and M.S in Electrical Engineering.

[…] and love EVPN as a control plane for VXLAN tunnels over a layer 3 infrastructure (Need a refresher? Check out our blog post on the topic). EVPN gives us the ability to deploy VXLAN tunnels without controllers. Plus, it offers a range of […]

the only thing which I couldn’t figure it out was the method of assigning/ mapping the VNID to the learned MAC addresses in the switch. If a MAC is learnt over VLAN 100, how the switch maps it to VNID?

With regards to the mapping, look at the white paper (link provided at top of the blog) page 20. This is a partial output of net show configuration. The bridge-access 100 on switchport2 puts that port in VLAN 100. VLAN 100 is also assigned to VXLAN-ID 10100 (under interface vxlan_1). To walk the mac address through the network on 2 VNIs (VXLAN-IDs), please look at the paper starting page 10.

Nice to hear from you. Yes, this EVPN release will provide a control plane for VXLAN tunnels only. Having said that, draft-ietf-bess-evpn-overlay describes using capabilities in RFC7432 but as applied to overlay tunnels instead of MPLS – so RFC7432 is still applicable

Many network operators would love to see EVPN-over-MPLS support as well, even if that depends on RFC3107 BGP-labeled IPv4 unicast (to avoid the requirement of an LDP or RSVP implementation, although OpenBSD’s LDPd has been ported to Quagga).