Nested ESXi on vCloud Air

One of the benefits of having vSphere as the virtual infrastructure for vCloud Air is ability to run a wide variety of guest operating systems. This includes hypervisors such as ESXi, which can be installed within vCloud Air as a virtual machine. The solution, commonly referred to as Nested ESXi, opens the door to a lot of interesting use cases. Coupled with vCloud Air OnDemand, VMware’s pay by the minute billing, VMware customers can easily deploy virtual vSphere infrastructure for test/dev, labs or classrooms for educational purposes. vCloud Air can also expose hardware features like Intel-VT/AMD-V directly to the virtual machine’s guest OS. This is an important feature to deliver performance to your virtual hypervisor and to the nested VMs. Other companies offer binary translation or emulation of these hardware features which can impact performance. While vCloud Air supports many different operating systems we do not recommend running your production workloads on Nested ESXi.

Nested ESXi isn’t a new concept. VMware Hands-On Labs has delivered over 300,000 Labs since November 2012, all of which have been Nested ESXi. There are also dozens of personal blogs that illustrate how to accomplish Nested ESXi within a vSphere environment or within vCloud Air. Cloud services such as vCloud Air do not expose the option for promiscuous mode. Mainly because of the network performance degradation of turning your port group to an Ethernet hub. So how do you configure Nested ESXi in vCloud Air with the same network performance and no promiscuous mode? In this blog I will briefly discuss the common use cases for Nested ESXi, show why most people tell you promiscuous mode is needed, and finally provide a work-around demonstrating how it can be done without promiscuous mode.

Nested ESXi Use Cases

Test/Development: Suppose you or your company is building a new solution that integrates with vSphere and you need virtual infrastructure for developers to write code against. You can deploy Nested ESXi as a test/development platform in vCloud Air without spending money on hardware. By creating Nested ESXi templates you can easily deploy and tear down vSphere instances as needed during development cycles.

Classroom: Once your developers have finished building the product, you will need a way to educate marketing, sales, and support staff on its features. You can provide hands on training using the same Nested ESXi template used during development. The template can be used for classroom activities or labs and deployed on a per student basis. Using vCloud Air OnDemand you can deploy the classroom only when needed and pay only for what you use. If you convert this training for external use, you can utilize public IP addresses to enable your customers to access the course anywhere in the world.

Demo and Labs: What better way of selling your vSphere solution than showing a live demo of your product? Your sales staff can deploy a Nested ESXi demo template while on site with the customer and show case its features. For a better support experience, your support staff can deploy a Nested ESXi lab environment to reproduce customer issues or resolve bugs within your software.

Promiscuous Mode and Nested ESXi

If you are not familiar with what Promiscuous Mode in virtual switching, I suggest reading the following VMware KB; How promiscuous mode works at the virtual switch and portgroup levels. In essence, by turning on Promiscuous Mode on a port group of a vSwitch, or vSphere Distributed Switch (vDS), vSphere allows all Ethernet frames to pass to all VMs that are attached to the port group even if it is not intended for that particular VM. I like to think that Promiscuous Mode is similar to treating the port group as an Ethernet hub instead of a switch. Let me explain why people say you need promiscuous mode for Nested ESXi.

In the below illustration we can see the components for Nested ESXi. The large light blue box represents our physical host which, in this example, resides in the vCloud Air data center. The physical host has a few virtual machines running on it including our Nested ESXi host, represented as a dark blue box. Virtual machines on the physical host connect to a port group on the virtual distributed switch (vDS). The physical NIC of the host, which has a MAC address of ‘EF’, connects to a physical switch upstream. The Nested ESXi host has two nested VMs running in it. The nested VM called Nested 1 with MAC address AB, connects to a port group on the virtual switch of the Nested ESXi host. The Nested ESXi vmnic with a MAC address of ‘CD’ connects to the physical host’s vDS port group.

Without promiscuous mode enabled the ingress network traffic from the physical switch is dropped. The reason is the physical host’s vDS is not aware of the MAC address of the nested VM, ‘AB’. The vDS only sees directly attached vmnics on the port group which in our case is the Nested ESXi host’s MAC address of ‘CD’.

With promiscuous enabled the network traffic is sent to every virtual machine on the port group regardless of the MAC address. In the example below, the Nested VM 1 receives the network traffic and also does the Normal VM with the MAC address of “1A” but it is dropped at the vmnic.

Nested VM vmnic to Virtual Host vmnic Mapping or 1:1

We know that without promiscuous mode ingress network traffic is dropped because the vDS is unaware of the MAC address. However by editing the MAC address of the nested VM network adapter to match the Nested ESXi host network adapter we are now finally able to reach the nested VM. In the illustration we can see I changed the MAC from AB to CD on the nested virtual machine, Nested VM 1. ESXi doesn’t see a duplicate MAC address because we can enable forge transmit on the Nested ESXi host.

1:1 Walk-through

I decided to show details on the work-around rather than discuss how to upload and configure Nested ESXi in vCloud Air. I mentioned before there are dozens of blogs showing how to do Nested ESXi but I wanted to point one out in particular. William Lam, a Product Integration Architect with VMware has done a great write-up of how to get started on Nested ESXi on vCloud Air. Here is the link to the article on his blog: http://www.virtuallyghetto.com/2014/05/how-to-run-nested-esxi-on-the-vcloud-hybrid-service.html.

Once you have deployed the Nested ESXi host(s) in vCloud Air I would recommend the following for this particular walk-through:

Add a minimum of two additional vmnics aside from the management NIC on the Nested ESXi host. These additional NICs will be used for the VM network traffic. When adding the vmnics, ensure you select the network adapter type “E1000”.

Either create a jump host within vCloud Air so you can use the vSphere Client to manage the Nested ESXi hosts or you can configure a NAT and Firewall rule on the vCloud Air Edge Gateway to allow vSphere Client or vSphere Web Client access to the Nested ESXi hosts.

Create two Routed Organization Networks within vCloud Air. Once called “Management” for the jump host and vmk0 management NIC, and the other “VM Network” for your nested virtual machine network. Attach the primary nic of the Nested ESXi host to the Management network along with your jump host. Add the two other nics of the Nested ESXi host to the VM Network. You may opt to create additional networks such as one for VMotion or one Storage traffic if you have a storage virtual appliance inside vCloud Air.

Connect to the Nested ESXi host using the vSphere Client or vSphere Web Client. From here we will want to create two virtual machines within the Nested ESXi host. I downloaded Centos 7 as an ISO on my jump box and transferred the ISO to the Nested ESXi host datastore. I called my VMs Centos1 and Centos2

Next create two new port groups called “VM Network 1” and “VM Network 2”.

We will now need to add our two other network adapters. In the vSwitch Properties, select the Network Adapters tab and select “Add”. Select both NICs to be added but specify that each NIC will be in Standby Adapters for the default vSwitch failover policy. This is shown below. You would also ensure that the newly added network adapters are also set to standby for the management network to avoid management communication issues.

We will then need to modify both port groups to only have one network adapter. This is important because we need to ensure that the nested vm can only talk over a specific network adapter of the Nested ESXi host. Edit the port group called “VM Network 1”. Select the NIC Teaming tab and select “Override switch failover order”. We will now move vmnic0 and vmnic2 to the Unused Adapters and move vmnic1 to the Active Adapters. Do the same with “VM Network 2” but select vmnic 2 as the only Active Adapter and vmnic0 and vmnic1 as unused.

Navigate to Network Adapters on the Nested ESXi host. You will need to copy the MAC address for vmnic 1 and vmnic 2 for the next step.vmnic1 MAC = 00:50:56:1d:52:50vmnic2 MAC = 00:50:56:1d:52:51

We will now edit the nested virtual machine to match the Nested ESXi Network Adapter. Right click on Centos1 and select “Edit Settings”. Select the Network adapter and under “MAC Address” select Manual. Enter the MAC address of vmnic1. In my example I entered 00:50:56:1d:52:50. Then under “Network Connection”, select VM Network 1.For Centos2 repeat the above but use vmnic2 MAC address and VM Network 2 for the Network Connection.

Now that we have edited the nested VM with the MAC address of the Nested ESXi network adapter, configured the port group to only use one vmnic, and only one VM is in that port group we’re are all set!
I added firewall and SNAT rules on the vCloud Air Edge Gateway to allow my Routed Organization network called “VM Network” to be able to communicate out. To save some time you can capture this Nested ESXi host to the vCloud Air catalog and deploy it again when needed. The only configuration would be an additional firewall and NAT rule.

Other Considerations

To increase the capacity of the amount of nested VMs per virtual host and reduce the amount of 1:1 vmnic-to-Nested ESXi network adapters, you could always deploy a virtual router in the Nested ESXi host. You would still configure a 1:1 with the external nic of the virtual router but you would create an isolated port group to which your VMs would be attached. The internal interface of the virtual router would act as the default gateway for your nested VMs to communicate through. Below is an example.

If you really want to get crazy, you could deploy vCenter Server with NSX and create a VXLAN virtual wire that would span multiple hosts. This way you can vmotion VMs to and from the Nested ESXi host while maintaining inter nested virtual machine communication. You would configure the virtual wire and map the 1:1 Nested ESXi host adapter to the VXLAN vmkernel interface shown below.

Conclusion

We at VMware vCloud Air look forward to finding more streamlined and robust ways of delivering Nested ESXi so stay tuned! vCloud Air is a versatile cloud platform that can not only run your existing applications and VMs but offer many other interesting use cases. Take this opportunity to think creatively about what you can do with virtual vSphere infrastructure on-demand with vCloud Air.

I would like the thank Rob Nafus, Massimo Re Ferre, William Lam and Eoin Hardwicke for contributing to this Nested ESXi solution on vCloud Air.

Thank you! I hope you found this blog informative. For future content like this, you can follow George on Twitter @GeorgeKobar.

For more information about VMware vCloud Air, visit vcloud.vmware.com, and keep an eye on the blog for upcoming tips and best practices for using vCloud Air.

Comments

bob rapp

Robert

November 11th, 2015

Currently working on similiar setup but using the standard switch, I need to add a iSCSI target using a secondary NIC on the ESXi host, however it seems that the vkernel port cannot reach the exterior unless it has the same MAC as the vmnic (same principle as with the VMs inside the host). I do not why the system correctly asigns matching MACs for VMkernel port 0 and vmnic 0 but with additional cards this does not occur, I’ve been trying to change the MAC on the vkernel port but no luck, also tried editing the ESX.conf to change the vmnic MAC, but it seems that the MAC gets rewritten by the vcloud air interface.