Elsewhere

Was migrating one of our offices to a new IP scheme the other day and vMotion started failing. I had a good idea what the problem could be (coz I encountered something similar a few days ago in another context) so here’s a blog post detailing what I did.

For simplicity let’s say the hosts have two VMkernel NICs – vmk0 and vmk1. vmk0 is connected to the Management Network. vmk1 is for vMotion. Both are on separate VLANs.

When our Network admins gave out the new IPs they gave IPs from the same range for both functions. That is, for example, vmk0 had an IP 10.20.1.2/24 (and 10.20.1.3/24 and 10.20.4/24 on the other hosts) and vmk1 had an IP of 10.20.12/24 (and 10.20.1.13/24 and 10.20.1.14/24 on the other hosts).

Since both interfaces are on separate VLANs (basically separate LANs) the above setup won’t work. That’s because as far as the hosts are concerned both interfaces are on the same network yet physically they are on separate networks. Here’s the routing table on the hosts:

1

2

3

4

5

~# esxcli network ip route ipv4 list

Network Netmask Gateway Interface Source

--------------------------------------------------

default0.0.0.010.20.1.254vmk0 MANUAL

10.20.1.0255.255.255.00.0.0.0vmk0 MANUAL

Notice that any traffic to the 10.20.1.0/24 network goes via vmk0. And that includes the vMotion traffic because that too is in the same network! And since the network that vmk0 is on is physically a separate network (because it is a VLAN) this traffic will never reach the vMotion interfaces of the other hosts because they don’t know of it.

So even though you have specific vmk1 as your vMotion traffic NIC, it never gets used because of the default routes.

If you could force the outgoing traffic to specifically use vmk1 it will work. Below are the results of vmkping using the default route vs explicitly using vmk1:

1

2

3

4

5

6

7

8

9

10

11

12

13

~# vmkping 10.20.1.13

PING10.20.1.13(10.20.1.13):56data bytes

---10.20.1.13ping statistics---

2packets transmitted,0packets received,100%packet loss

~# vmkping -I vmk1 10.20.1.13

PING10.20.1.13(10.20.1.13):56data bytes

64bytes from10.20.1.13:icmp_seq=0ttl=64time=0.433ms

64bytes from10.20.1.13:icmp_seq=1ttl=64time=0.198ms

---10.20.1.13ping statistics---

2packets transmitted,2packets received,0%packet loss

round-tripmin/avg/max=0.198/0.316/0.433ms

The solution here is to either remove the VLANs and continue with the existing IP scheme, or to keep using VLANs but assign a different IP network for the vMotion interfaces.

Update: Came across the following from this blog post while searching for something else:

If the management network (actually the first VMkernel NIC) and the vMotion network share the same subnet (same IP-range) vMotion sends traffic across the network attached to first VMkernel NIC. It does not matter if you create a vMotion network on a different standard switch or distributed switch or assign different NICs to it, vMotion will default to the first VMkernel NIC if same IP-range/subnet is detected.

Please be aware that this behavior is only applicable to traffic that is sent by the source host. The destination host receives incoming vMotion traffic on the vMotion network!

That answered another question I had but didn’t blog about in my post above. You see, my network admins had also set the iSCSI networks to be in the same subnet as the management network – but separate VLANs – yet the iSCSI traffic was correctly flowing over that VLAN instead of defaulting to the management VMkernel NIC. Now I understand why! It’s only vMotion that defaults to the first VMkernel NIC in the same IP range/ subnet as vMotion.

On a newly built host today vMotion was failing while migrating VMs to this host. vMotion would get stuck at 14% and then fail with the above error. Found this VMware KB article – it was informative but didn’t help. From this article I learnt though that I can use vmkping with the -I switch to specify an interface to use while pinging. This is handy when you want to ping a remote address via a specific interface – say, the vMotion IP address of a remote host, via the vMotion VMKernel of this host. Usually vmkping automatically selects an interface on the network you are trying to ping but it’s possible you are using the same subnet for vMotion and many other services.

Anyhow, in my case I noticed that if I removed one of the underlying physical adapters I am able to vmkping. So add that to the list of things to try if you too are in a similar situation. Odd though that it failed though! I would have thought a failed physical adapter means it will just try a different one? Clearly in my case the other adapter was working.

I don’t know more details but it could be that the physical NIC was up but the switch was blocking? Not sure.

Was creating / migrating some ESXi hosts during the week and came across the above error “Number of IPv4 routes did not match” when checking for host profile compliance of one of the hosts. All network settings of this host appeared to be same as the rest so I was stumped as to what could be wrong. Via a VMware KB article I came across the esxcfg-route command that helped identify the problem. To run this command SSH into the host:

1

2

~# esxcfg-route

VMkernel defaultgateway is10.50.0.252

By default the command only outputs the default gateway but you can pass it the -l switch to list all routes:

1

2

3

4

5

6

~# esxcfg-route -l

VMkernel Routes:

Network Netmask Gateway Interface

10.50.0.0255.255.255.0Local Subnet vmk0

1.1.1.0255.255.255.0Local Subnet vmk2

default0.0.0.010.50.0.252vmk1

In my case the above output was from one of the hosts, while the following was from the non-compliant host:

1

2

3

4

5

6

~# esxcfg-route -l

VMkernel Routes:

Network Netmask Gateway Interface

10.50.0.0255.255.255.0Local Subnet vmk0

10.50.0.0255.255.0.0Local Subnet vmk2

default0.0.0.010.50.0.252vmk1

Notice the vmk2 interface has the wrong network. Not sure how that happened. Oddly the GUI didn’t show this incorrect network but obviously something was corrupt somewhere.

To fix that I thought I’ll remove the vmk2 interface and re-add it. Big mistake! Possibly because its network was same as that of the management network (10.50.0.0/24) removing this interface caused the host to lose connectivity from vCenter. I could ping it but couldn’t connect to it via SSH, vSphere Client, or vCenter. Finally I had to reset the network via the DCUI – it’s under “Network Restore Options”. I tried “Restore vDS” first, didn’t help, so did a “Restore Standard Switch”. This is a very useful – it creates a new standard switch and moves the Management Network onto that so you get connectivity to the host. This way I was able to reconnect to the host, but now I stumbled upon a new problem.

The host didn’t have the vmk2 interface any more but when I tried to recreate it I got an error that the interface already exists. But no, it does not – the GUI has no trace of it! Some forum posts suggested restarting the vCenter service as that clears its cache and puts it in sync with the hosts but that didn’t help either. Then I came across this post which showed me that it is possible for the host to still have the VMkernel port but vCenter to not know of it. For this the esxcli command is your friend. To list all VMkernel ports on a host do the following:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

~# esxcli network ip interface list

vmk2

Name:vmk2

MAC Address:00:50:52:24:2d:e3

Enabled:true

Portset:DvsPortset-1

Portgroup:N/A

Netstack Instance:defaultTcpipStack

VDS Name:dvSwitch1(ISCSI)

VDS UUID:92c785403e7f5a5a-fd4ee59d2f208dd2

VDS Port:132

VDS Connection:39130298

MTU:1500

TSO MSS:65535

Port ID:100663300

...

After that, removing the VMkernel interface can be done by a variant of same command:

1

~# esxcli network ip interface remove --interface-name vmk2

Now I could add the re-add the interface via vSphere and get the hosts into compliance.

Before I conclude this post though, a few notes on the commands above.

If you have PowerCLI installed you can run all the esxcli commands via the Get-EsxCli cmdlet. For example:

I would have written more on the esxcli command itself but this excellent blog post covers it all. It’s an all powerful command that can be used to manage many aspects of the ESXi host, even set it in maintenance mode!

If that VM were stuck for some reason and cannot be stopped or restarted via vSphere it’s very useful to know the esxcli command can be used to kill the VM (has happened a couple of times to me in the past):

1

~# esxcli vm process kill -t force -w 155802

Regarding the type of killing you can do:

There are three types of VM kills that can be attempted: [soft, hard, force]. Users should always attempt ‘soft’ kills first, which will give the VMX process a chance to shutdown cleanly (like kill or kill -SIGTERM). If that does not work move to ‘hard’ kills which will shutdown the process immediately (like kill -9 or kill -SIGKILL). ‘force’ should be used as a last resort attempt to kill the VM. If all three fail then a reboot is required. (required)

Another command line option is vim-cmd which I stumbled upon from one of the links above. I haven’t used it much so as a reference to myself here’s a blog post explaining it in detail.

Lastly there’s also a bunch of esxcfg-* commands, one of whom we came across above.

I haven’t used these much. They seem to be present for compatibility reasons with ESXi 3.x and prior. Back then you had commands with a vicfg- prefix, now you have the same but with a esxcfg- prefix. For instance, esxcfg-vmknic is now replaced with esxcli network interface as we saw above.

That’s all for now!

Update: Thought I’d use this post to keep track of other useful commands.

Remotely enable SSH

The esxcli commands are cool but you need to enable SSH each time you want to connect to the host and run these (unless you install the CLI tools on your machine). If you have PowerCLI though you can enable SSH remotely.