NSX L2 Bridging

Overview

This next overview of L2 Bridging was taken from great work of Max Ardica and Nimish Desai in the official NSX Design Guide:

There are several circumstances where it may be required to establish L2 communication between virtual and physical workloads. Some typical scenarios are (not exhaustive list):

Deployment of multi-tier applications: in some cases, the Web, Application and Database tiers can be deployed as part of the same IP subnet. Web and Application tiers are typically leveraging virtual workloads, but that is not the case for the Database tier where bare-metal servers are commonly deployed. As a consequence, it may then be required to establish intra-subnet (intra-L2 domain) communication between the Application and the Database tiers.

Physical to virtual (P-to-V) migration: many customers are virtualizing applications running on bare metal servers and during this P-to-V migration it is required to support a mix of virtual and physical nodes on the same IP subnet.

Leveraging external physical devices as default gateway: in such scenarios, a physical network device may be deployed to function as default gateway for the virtual workloads connected to a logical switch and a L2 gateway function is required to establish connectivity to that gateway.

Deployment of physical appliances (firewalls, load balancers, etc.).

To fulfill the specific requirements listed above, it is possible to deploy devices performing a “bridging” functionality that enables communication between the “virtual world” (logical switches) and the “physical world” (non virtualized workloads and network devices connected to traditional VLANs).

NSX offers this functionality in software through the deployment of NSX L2 Bridging allowing VMs to be connected at layer 2 to a physical network (VXLAN to VLAN ID mapping), even if the hypervisor running the VM is not physically connected to that L2 physical network.

Figure above shows an example of L2 bridging, where a VM connected in logical space to the VXLAN segment 5001 needs to communicate with a physical device deployed in the same IP subnet but connected to a physical network infrastructure (in VLAN 100). In the current NSX-v implementation, the VXLAN-VLAN bridging configuration is part of the distributed router configuration; the specific ESXi hosts performing the L2 bridging functionality is hence the one where the control VM for that distributed router is running. In case of failure of that ESXi host, the ESXi hosting the standby Control VM (which gets activated once it detects the failure of the Active one) would take the L2 bridging function.

Independently from the specific implementation details, below are some important deployment considerations for the NSX L2 bridging functionality:

The VXLAN-VLAN mapping is always performed in 1:1 fashion. This means traffic for a given VXLAN can only be bridged to a specific VLAN, and vice versa.

A given bridge instance (for a specific VXLAN-VLAN pair) is always active only on a specific ESXi host.

However, through configuration it is possible to create multiple bridges instances (for different VXLAN-VLAN pairs) and ensure they are spread across separate ESXi hosts. This improves the overall scalability of the L2 bridging function.

The NSX Layer 2 bridging data path is entirely performed in the ESXi kernel, and not in user space. Once again, the Control VM is only used to determine the ESXi host where a given bridging instance is active, and not to perform the bridging function.

Configure L2 Bridge

In this scenario we would like to Bridge Between App VM connected to VXLAN 5002 to virtual machine connected to VLAN 100.

My current Logical Switch configuration:

We have pre-configured a VLAN-backed port group for VLAN 100:

Bridging configuration is done at the DLR level. In this specific example, the DLR name is Distributed-Router:

Double Click on the edge-1:

Click on the Bridging and then green + button:

Type Bridge Name, Logical Switch ID and Port-Group name:

Click OK and Publish:

Now VM on Logical Switch App-Tier-01 can communicate with Physical or virtual machine on VLAN 100.

Design Consideration

Currently in NSX-V 6.1 we can’t enable routing on the VXLAN logical switch that is bridged to a VLAN.

In other words, the default gateway for devices connected to the VLAN can’t be configured on the distributed logical router:

None working L2 Bridge Topology

So how can VM in VXLAN 5002 communicate with VXLAN 5001?

The big difference is VXLAN 5002 is no longer connected to the DLR LIF, but it is connected instead to the NSX Edge.

Redundancy

DLR Control VM can work in high availability mode, if the Active DLR control VM fails, the standby Control VM takes over, which means the Bridge instance will move to a new ESXi host location.

Bridge Troubleshooting:

Most issues I ran into was that the bridged VLAN was missing on the trunk interface configured on the physical switch.

Active DLR control VM is located at esx-02a, so the bridging function will be active in this ESXi host.

Both ESXi hosts have two physical nics: vmnic2 and vmnic3.

Transport VLAN carries all VNI (VXLAN’s) traffic and is forwarded on the physical switch in VLAN 20.

On physical switch-2 port E1/1 we must configure trunk port and allow both VLAN 100 and VLAN 20.

Note: Port E1/1 will carry both VXLAN and VLAN traffic.

Find Where Bridge is Active:

We need to know where the Active DLR Control VM is located (if we have HA). Inside this ESXi host the Bridging happens in kernel space. The easy way to find it is to look at “Configuration” section in the “Manage” tab.

Note: When we powered off the DLR Control VM (if HA is not enabled), the bridging function on this ESXi host will stop to prevent loop.

Categories

Roie Ben Haim is a Senior Member of Technical Staff who specializes in Networking and Security at VMware and who is currently focused on implementing solutions, which incorporate VMware’s NSX platform as well as integrating with various Cloud platforms on VMware’s infrastructure.
Roie works in VMware’s Consulting (PSO) team whose focus is on the delivery of Networking Virtualization and Security solutions. In this role Roie provides technical leadership in all aspects, including the installation, configuration, and implementation of VMware’s products and services. This is also includes being involved from the inception of these project, through requirements assessment, design and deployment phases and then into production which ensures continuity for VMware’s customers.
Roie has over a 15 years of experience working on data center technologies, and providing solutions for global enterprises, which primarily focus on Network and Security.
A highly motivated and enthusiastic MSc graduate Roie holds a wide range of industry leading certificates, including his most recent Network Virtualization (VCDX-NV). Cisco CCIE x2 (DC/SEC) and Juniper JNCIE-SP.
Roie is not only a strong team member, but is also able to demonstrate his skills and experience working in various fields.
As a well known and respected blogger, Roie maintains an impressive blog at:
http://routetocloud.com