Posey's Tips & Tricks
Getting Real About NUMA and Hyper-V, Part 1
NUMA in Hyper-V isn't something you have to master--it's something you need to avoid accidentally breaking, and understanding a host's NUMA layout helps you do that.
I have a confession to make. Even though I have been using Hyper-V since it was first released with Windows Server 2008, I have never paid much attention to NUMA. I always thought of NUMA as being one of those things that only huge corporations had to worry about. Besides, if I am to be totally honest with you, the concept of NUMA never really clicked with me. Most of the NUMA-related material that I have read immediately caused my eyes to glaze over and so I just never had an interest in finding out more.
More recently, a few architectural changes that I have been making in my own organization have forced me to take a fresh look at NUMA, especially as it related to Hyper-V. What I have discovered is that NUMA isn't something that you have to worry about mastering. Instead, it's something that you need to avoid accidentally breaking. Notice that I used the word accidentally. There are times when it absolutely makes sense to break NUMA, but I am getting ahead of myself.
OK, so before I tell you what you actually need to know about NUMA when working with Hyper-V, I want to show you a couple of quick PowerShell commands that will hopefully help the explanation to make more sense. The first command (which should be run on the Hyper-V host) is: Get-VMHostNUMANode. The second command is: Get-CIMInstance Win32_Processor | Select-Object Name, NumberOfCores, NumberOfLogicalProcessors. You can see both of these commands shown in Figure 1.
[Click on image for larger view.]
Figure 1. These Are the Commands That You Can Use to Get a Better Understanding of NUMA.
The first of the two commands causes the machine to display the server's NUMA nodes. As you can see by the NodeID, there are two NUMA nodes in this particular machine. Another thing that the first command shows you is how much memory is allocated to each NUMA node. This particular computer has 384 GB of memory installed and that memory is divided between the two NUMA nodes, with each receiving 192 GB.
The second command shows the machine's physical CPUs, the number of virtual processors, and the number of logical processors. This machine has two physical CPUs. The fact that there are two CPUs and two NUMA nodes is not a coincidence. Each CPU has its own NUMA node.
Some people have described a multi-CPU server as being like multiple PCs that have been glued together. Like individual PCs, each CPU has its own memory. However, the real difference between multiple discrete PCs that have been jammed together in a single case and a modern server is that just because a CPU has its own memory does not mean that it can't use memory belonging to another CPU. A CPU can use as much memory as it needs, all the way up to the server's total capacity, regardless of any NUMA nodes that might exist. The catch however, is that CPUs have much faster access to their own memory than to the memory associated with neighboring CPUs. Let me further illustrate the point through an analogy that is sometimes used.
Imagine for a moment that you are cooking dinner in your own kitchen. Over the course of the meal-preparation, you constantly have to get ingredients out of the refrigerator and put back the items that you are done with. So in this analogy, your kitchen is the CPU and the refrigerator is the memory that is allocated to the CPU.
So now, imagine that you are going to be having a large dinner party. Think of this large dinner party as being like running a large workload on your server. Because you are going to be preparing so much food, the refrigerator in your kitchen just isn't large enough to hold all of the ingredients. This would be like having a workload that can't fully fit within a CPU's own memory.
Since you can't just skip the refrigeration process, you have to refrigerate some of the ingredients elsewhere. Maybe this means using a second refrigerator that's in the garage, or maybe you have to put a few items in your neighbor's refrigerator. In any case, this secondary refrigerator gets the job done, but efficiency goes out the window. After all, it's way faster to get an item out of the refrigerator residing in the same kitchen where you are preparing the meal than to have to go to your garage or to your neighbor's place every time you need an ingredient.
This simple illustration is the very essence of NUMA. Your workloads can use any memory that they want, but they will perform a lot better if they can use the memory that belongs to the CPU that is hosting the workload. It really is that simple.
So now that I have talked about NUMA at a really high level, I want to go just a little bit further and talk about what this means for your Hyper-V workloads.
About the Author
Brien Posey is a 22-time Microsoft MVP with decades of IT experience. As a freelance writer, Posey has written thousands of articles and contributed to several dozen books on a wide variety of IT topics. Prior to going freelance, Posey was a CIO for a national chain of hospitals and health care facilities. He has also served as a network administrator for some of the country's largest insurance companies and for the Department of Defense at Fort Knox. In addition to his continued work in IT, Posey has spent the last several years actively training as a commercial scientist-astronaut candidate in preparation to fly on a mission to study polar mesospheric clouds from space. You can follow his spaceflight training on his Web site.