Mirantis OpenStack 7.0: NFVI Deployment Guide — NUMA/CPU pinning

As we hinted in the section about Huge pages, NUMA means that memory is broken up into pools with each vCPU having its own pool of “local” memory. Best performance comes from making sure a process and its memory are running in the same NUMA cell. Unfortunately, the nature of virtualization means that processes typically use whatever vCPU is available, whether it’s local to the memory or not. This is partially because when deciding how to allocate VMs to CPUs, the default OpenStack scheduler attempts to optimise for heavily contended systems rather than guarantee the performance of individual VMs. You can override this behavior using CPU Pinning.

CPU Pinning enables you to pin, or establish a mapping between virtual CPU to the physical core so that a virtual CPU will always run on the same physical one. By exposing NUMA topology to the VM and pinning vCPU to specific core it’s possible to improve VM performance by ensuring that access to memory will always be local in terms of NUMA topology.

Compute hosts configuration

To enable CPU Pinning, perform the following steps on every compute host where you want CPU pinning to be enabled.

You should see that each vCPU is pinned to a dedicated CPU core, which is not used by the host operating system, and that these cores are inside the same host NUMA cell (in our example it’s cores 4 and 16 in NUMA cell 1).

You should see that each vCPU is pinned to a dedicated CPU core, which is not used by the host operating system, and that these cores are inside another host NUMA cell. In our example it’s core 2 in NUMA cell 1 and core 10 in NUMA cell 2. As you may remember in our configuration, cores 1-5 and 12-17 from cell 1 and cores 7-11 and 18-23 from cell 2 are available to virtual machines.

Troubleshooting

You might run into the following errors:

internal error: No PCI buses available in /etc/nova/nova.conf

In this case, you’ve specified the wrong hw_machine_type in /etc/nova/nova.conf

libvirtError: unsupported configuration

Per-node memory binding is not supported with this version of QEMU. You may have an older version of qemu, or a stale libvirt cache.