The virtualization blog by Joep Piscaer

Virtualizing vCenter with vDS: Another Catch-22

You’ll know when you’ve come up with a good title when other people start copying it. Thanks Jason!

I’m setting up a VMware vSphere environment for a demo and presentation I’m giving later this week on VMware vCenter Lab Manager 4 at the Dutch VMUG Event. I prepared a couple of VM’s on my laptop running VMware Workstation, among wich a vCenter Server installation. I installed two physical servers with ESX4, attached them to vCenter and configured a dvSwitch on both. As these hosts are really simple desktops with just a single NIC, I had to configure the dvUplinks group with one uplink per host. I migrated the Service Console Port (vswif0), created a couple of port groups for Virtual Machine networking and created a VMKernel port for NFS and VMotion. Since there was no need for the standard vSwitch anymore, I simply deleted the associated Port Groups and the vSwitch itself:

Now I had all ESX services (vswif0 and vmk0) and virtual machine port groups running on the dvSwitch. The end result was pretty:

So now for the migration of those local VM’s to the newly set up ESX-hosts using VMware Converter Standalone. I shut down the four VM’s and created the Convert task to v2v them. As I reviewed the options, I noticed that Converter picks up Port Groups from a dvSwitch, but does not advertise them as such:

Oh well, as long as they’re there, right? I started the v2v tasks and prepared some home-made pizza in the mean time. After an hour or so, I came back to the computer (the pizza was great!) and found all tasks successfully completed. However, the VM’s, now running on vSphere, were not present on the network. I could not ping them or find their MAC-adresses in the switch. They were simply not online. They booted correctly, had VMware Tools running, had the correct TCP/IP settings, but their virtual network adapters weren’t connected to the dvSwitch:

To make matters worse: I could not select the correct network label (Port Group) in the drop down list. After some long and hard thinking, I figured out why: ESX couldn’t communicate with vCenter to update the dvSwitch’s status. This is simply because the vCenter VM was one of the migrated VM’s, and thus suffered from the same problem: it wasn’t connected to the network. How’s that for a catch-22!

As I said earlier, the physical hosts run on a single vmnic. No easy fix here then, I cannot create a standard vSwitch, create a port group on it, add a vmnic and migrate the vCenter VM to this port group to get the VM online and thus be able to get the other VM’s attached to the right (dvSwitch) Port Group, after which I can migrate the vCenter VM to the right PG.

I had to break the dvSwitch on the physical host by removing vmnic0 and vswif0 from it and attach it to a standard vSwitch. As you cannot migrate a vmnic and a virtual adapter like the Service Console Port at the same time, I had a major problem. Migrating either would mean that I would lose all connectivity to the host. Getting them both attached to a standard vSwitch was my only way out of this conundrum. I need to fall back to the CLI for this one!

First, I removed vswif0 and vmnic0 from the dvSwitch:

esxcfg-vswif -d vswif0
esxcfg-vswitch -Q vmnic0 -V 969 dvSwitch

I created a new standard vSwitch and Port Group and attached vmnic0 as an uplink. I also recreated vswif0:

I had restored the original setup: a simple standard vSwitch with a Service Console Port and a Virtual Machine Port Group. VMnic0 provided the link to the outside world. I moved the vCenter VM to the VMTMP Port Group. vCenter and ESX could finally negotiate details on the dvSwitch, so I could get on with it, attaching all other VM’s to the correct (dvSwitch) Port Group. The last VM to go was obviously the vCenter VM, as doing so would result in the VM losing connectivity to the network again. I quickly dove back into the CLI and reversed the commands I had just given to attach the Service Console to the dvSwitch again.

Wow, what a way to go to get a couple of VMs to the correct Port Group, phew! Of course, having just a single NIC in a dvSwitch environment is BAD. I was just too lazy to arrange some extra NICs for these desktops because I’ll only be using them temporarily. That being said, I learned first hand why a hybrid solution (which requires multiple physical NICs) is a great idea. For further reading on hybrid environments, please do check out VCDX 007’s post on it!