Great Article on Hyper-V and NUMA

Any software designer/engineer needs to be aware of how Non-Uniform Memory Access (NUMA) impacts the performance of services that will run on that hardware. This goes double for virtualisation administrators, and here’s why.

NUMA is a hardware design feature that divides CPUs and memory in a physical server into NUMA nodes. You get the best performance when a process uses memory and CPU from within the same NUMA node. When a process requires more memory, but the current NUMA node is full, then it’ll get memory from another NUMA node and that comes at a performance cost to that process, and possibly all other processes on that physical server.

And that’s why virtualisation engineers need to be aware of this. In Hyper-V we have Dynamic Memory. In VMware, there are other techs that do similar (but work differently) things to add memory to a VM under the covers. When there’s contention in a NUMA node, a VM will be given additional memory from a different NUMA node – and then performance will drop.

When present this topic, NUMA causes a lot of confusion. Microsoft gave us a rather badly out-dated formula for calculating NUMA node sizes. It’s actually a hardware specification (that all OS’s and hypervisors have to deal with) so the only really accurate way to determine NUMA node layouts is via PerfMon or chatting to the hardware vendor. In the meantime, I stumbled across this fantastic article by Benjamin Athawes. Benjamin explains NUMA superbly and talks about how to determine what’s in your hardware.

In the Hyper-V world, we can disable NUMA node spanning in the host settings. That’s thanks to how Dynamic Memory works – there isn’t an over-commitment that must be lived up to by the hypervisor. If we see lots of spanning that impacts performance, then we have choices:

Reconsider hardware specs to increase the size of NUMA nodes: if there is a lot of consistent NUMA node spanning that is required to supply badly needed memory to VMs

In Windows 8 Hyper-V, guests will have will get a new feature where the guest OS can be NUMA node aware. That’s really requires because we’re jumping to 32 vCPU support which will likely span many NUMA nodes. With this feature, guest OS processes/memory can be scheduled to take NUMA node placement into account.

Archives

About this Blog

This blog serves 2 purposes. Firstly, I want to share information with other IT pros about the technologies we work with and how to solve problems we often face. I've worked with technologies from the desktop to the server, Active Directory, System Center, security and virtualisation.

Secondly, I use my blog as a notebook. There's so much to learn and remember in our jobs that it's impossible to keep up. By blogging, I have a notebook that I can access from anywhere. It has saved my proverbial many times in the past.

Waiver

Anything you do to your IT infrastructure, applications, services, computer or anything else is 100% down to your own responsibility and liability. Aidan Finn bears no responsibility or liability for anything you do. Please independently confirm anything you read on this blog before doing whatever you decide to do.