Investigation

NetScaler was a separate physical box but the same problem should apply to any form-factor.

At first I was blaming the network. But further analysis showed it only seems to affect a small subset of services – those hosted on Hyper-V. So I started blaming Hyper-V but turned out it wasn’t that simple.

Drilling deeper into the issue I could see a significant high number of TCP re-transmissions on the mentioned sessions. It took me some time but then I’ve noticed the different MAC addresses involved in one session. One Intel and one Microsoft-tagged. Microsoft being the VMs MAC, Intel being the current host’s physical NIC.

Trace showing the flapping MACs

Interesting enough most issues seemed to follow a pattern:

Microsoft MAC speaks to NetScaler happily

Intel MAC comes into picture

Session struggles

Root cause

So it seems Hyper-V bounces MACs in certain scenarios. While this is not necessarily a problem on its own, it can become one. Meet MAC-Based Forwarding.

Ok, blaming MBF is only one part of the problem. The other resides in the fact that Hyper-V obviously seems to strictly expect responses on the virtual MAC while sending out traffic sourcing from both MACs.

Hyper-V virtual MAC behavior

Combining this with MBF will cause Hyper-V to drop packets the moment it decides to use the physical MAC and that’s exactly what happened in the above trace.

I’ve seen the same behavior with some routers using VRRP in the past. They did send out all traffic on the physical interface but expected all responses on the virtual interface.

VRRP virtual MAC behavior

However, this was only a small subset of routers I’ve encountered. Not all that use VRRP or other HA mechanisms. But still, this shows that it seems to be a sporadically reoccurring issue between NetScaler, MBF and some vendors.

Solutions

In the case of Hyper-V, the problem can be solved on both ends of the connection.

Another fellow blogger, Dale Scriven, pointed out in his article on the vHorizon blog that this can also be solved changing the Hyper-V NIC teaming load balancing settings from “Dynamic” to “Hyper-V Port”. Read the full story here: NetScaler VPX and Hyper-V flapping

Now I don’t know who’s right or wrong here. NetScaler or Hyper-V. I couldn’t find sufficient evidence in the RFCs I’ve skimmed to blame either of them. But I know that for the last years I’ve avoided MBF and my PBR concept never faced me with any of the nasty network-issue surprises that I occasionally face with MBF.

4 Replies to “NetScaler services flapping with Hyper-V”

The fundamental issue with the NS’er is that if there are numerous NIC’s enabled (2-arm mode), all interfaces will be in the native VLAN by default. Having numerous interfaces in the same broadcast domain without LAGing them will induce MAC moves. This can easily be seen in the shell of the NS’er by leveraging the ‘nsconmsg’ command with the counter ‘bdg_mac’. Leverage the guidance in the following post to see if mac moves are occuring – https://www.citrix.com/blogs/2014/05/02/netscaler-counters-grab-bag/

Simple solution is to create access VLANs for each of the interfaces other than the DG interface so remove them from the native VLAN. Being that tagging is not in play, you’ll see no issues, but will see the mac moves cease. When too many mac moves occur, interface ‘mutes’ occur, which fundamentally means that zero packets are flowing from that interface to avoid the bridge loop.

Yes, the VLANs is another point that gets misconfigured waaay too often!

The MAC flapping happened on Hyper-V, not on NS. So they can’t be solved on NS.
But MBF didn’t dealt with the flapping well, PBR did, hence that was my solution.

Beside of that, the need for PBR/MBF applies regardless of whether you have VLANs configured or not – as soon as you have multi-arm deployments.
NetScaler would always route to the default gateway or closest route if you don’t do PBR or MBF. Therefore it potentially creates asymmetric routing if you need to access two different interfaces from the same client subnet.
The VLANs aren’t taken into consideration in routing decisions as NS behaves like a router and will always choose the shortest path he knows.