Posey's Tips & Tricks
Getting Real About NUMA and Hyper-V, Part 2
While Hyper-V largely handles NUMA placement automatically, admins can still hurt VM performance by overallocating virtual CPUs past a single physical CPU's core count, making it important to size VMs to stay within one NUMA node when possible.
In the first part of this article series, I illustrated the significance of NUMA memory use by comparing it to, of all things, a trip to the refrigerator. Now, I want to take a step back and talk about NUMA use with Hyper-V.
I started my previous article by confessing that even though I have been using Hyper-V since the days of Windows Server 2008, I never really paid any attention to NUMA placement. That raises a big question… How on Earth did I manage to get away with using Hyper-V for the better part of two decades without worrying about NUMA?
One possible answer is that I didn't start using running Hyper-V on multi-socket servers until four or five years ago, so NUMA didn't even come into play. However, the real answer about how I managed to use Hyper-V for years without worrying about NUMA is simply that Hyper-V does all of the heavy lifting for you. Hyper-V uses a few different tricks behind the scenes to make sure that it is making optimal use of NUMA.
So that's the end of the story, right? Not quite. Even though Hyper-V tries to do the right thing, it is a virtual machine's configuration that will ultimately determine just how well the virtual machine will perform. Unfortunately, there is at least one Hyper-V configuration myth that often does more harm than good. That myth is the idea that you can improve a virtual machine's performance by allocating additional virtual processors to the virtual machine.
Like all good myths, this one does have an element of truth to it. if you have two virtual CPUs allocated to a virtual machine and you increase the virtual CPU count to four, then you are probably going to see a performance improvement. However, when it comes to increasing the virtual CPU count, there are a few things to keep in mind.
First, a virtual machine won't benefit from additional virtual CPUs if its performance is being constrained by a non-CPU bottleneck. For example, if lack of memory is limiting a virtual machine's performance, then adding some extra CPU resources isn't going to help.
Second, adding extra virtual CPUs won't help if the workload running on the virtual machine doesn't need those resources. Suppose for instance that a virtual machine is idle and it's CPU usage is hovering around 2%. Such a VM is only using a small fraction of its existing CPU resources, so there is no advantage to adding more.
Third, adding virtual CPU resources can do more harm than good if the system's CPU resources are already maxed out at the hardware level. After all, you can't squeeze more CPU cores out of thin air.
Finally, adding additional virtual CPU resources usually won't help with performance if the virtual CPU count exceeds a single physical CPU's core count.
In the previous article, I showed a screen capture from one of my servers. That particular machine was equipped with two physical CPUs with 18 cores each. In a situation like that, a virtual machine with 18 or fewer virtual processors will usually map to a single physical processor, meaning that the virtual machine's memory resides in a single NUMA node (assuming that you have not provisioned more memory than what exists in the NUMA node). If however, I were to allocate 20 virtual processors to that virtual machine, it would then be using cores on two different CPUs, meaning that memory access is now being spread across two different NUMA nodes. In this particular case, the virtual machine would likely perform better with 16 virtual processors than with 20.
This raises the question of why Hyper-V doesn't actively stop you from spanning NUMA nodes. The reason is that sometimes, spanning NUMA nodes is justified. Suppose that you need to create a really big virtual machine for a workload that requires numerous virtual processors and lots of memory. It would be better for such a machine to run a bit sluggishly than for Hyper-V to tell you that you are not allowed to create such a virtual machine, even though your server has the hardware to support it.
So when it comes to creating Hyper-V virtual machines, my advice would be to try to create virtual machines that can fit within the confines of a single NUMA node if you can, but to not be afraid of crossing NUMA boundaries when there is a justifiable need for doing so.
About the Author
Brien Posey is a 22-time Microsoft MVP with decades of IT experience. As a freelance writer, Posey has written thousands of articles and contributed to several dozen books on a wide variety of IT topics. Prior to going freelance, Posey was a CIO for a national chain of hospitals and health care facilities. He has also served as a network administrator for some of the country's largest insurance companies and for the Department of Defense at Fort Knox. In addition to his continued work in IT, Posey has spent the last several years actively training as a commercial scientist-astronaut candidate in preparation to fly on a mission to study polar mesospheric clouds from space. You can follow his spaceflight training on his Web site.