Posey's Tips & Tricks
Diagnosing Virtual Server Performance Problems
I recently encountered a situation involving virtual server memory allocation that could help others coming to grips with network performance issues.
Initially, the problem wasn't so clear. Over the past couple of months, I had noticed that my Internet connection was running more slowly, but I didn't think a lot about it. Some of my neighbors tend to do a lot of video streaming. Between that streaming and the fact that I am operating a continuous data protection solution that constantly writes data to the cloud, I just assumed that all of the available bandwidth was being used.
It took a little more time for the problem to unravel. At the beginning of February, I left for a month-long expedition to Antarctica. When I returned, I discovered that my Internet connection was so slow that it was practically useless. I knew then and there that I had a bigger problem than just bandwidth saturation.
Slow DNS Resolution
After analyzing my network traffic, I discovered that my Internet connection was actually OK. The problem was that DNS queries were being resolved extremely slowly, if at all.
My DNS server exists as a virtual machine on a Hyper-V Server. The Hyper-V Server also hosts an Exchange 2007 Client Access Server/Hub Transport Server and another virtual machine that acts as a domain controller and as a file server. These virtual machines have been in place for about two years and I have never had any performance problems with them. I can also tell you that I have not made any configuration changes, aside from applying patches, for quite some time.
Still, my DNS server was running at a snail's pace even though the DNS server should have been placing almost no load on the host server. After all, how many resources does a Windows Server 2003 DNS server really consume on a network with only two users? When I set the virtual machine up a couple of years ago, I gave it 2 GB of RAM and one virtual processor and that had always been more than sufficient.
My first instinct was that maybe the DNS server was suffering from some sort of malware infection or that perhaps I had deployed a bad patch. Once I had ruled out those possibilities, however, I knew that the problem had to be that one of the other virtual machines was starving the DNS server for resources.
Even though Hyper-V allows you to overcommit CPU resources, I had not done so. I had allocated fewer virtual processors than the total number of physical CPU cores that were installed in the server. I wanted to make sure that there was no CPU overcommitment and that there were a couple of cores left over to service the host operating system.
I also knew that I hadn't overcommitted the server's memory. The server has 8 GB of physical RAM and I had dedicated 2 GB to each virtual server and left 2 GB for the host OS. Even though 2 GB per VM may not sound like enough, all of the VMs are running Windows Server 2003 and there are only two users on the network.
I was also able to rule out network congestion as a possible cause. I had dedicated a separate gigabit NIC to each virtual machine, and there was no way that there were enough DNS name resolution requests to saturate a gigabit connection.
The only hardware resource that I was unable to eliminate as a possible cause of the problem was the storage array. The host server boots off a dedicated hard drive, but all of the virtual hard disks that are used by the virtual machines reside on a common storage array. The array is fast enough that it had never previously had any trouble servicing all three virtual machines, but I was now left wondering if something had changed that caused a significant increase in disk I/O.
Exchange Overload
After spending some time using the Performance Monitor on each of my virtual machines, I discovered that my Exchange Server was low on memory. As a result, the operating system was making heavy use of virtual memory. Even though the Exchange Server itself did not outwardly show any signs of performance problems, the constant paging operations were causing a tremendous amount of I/O on my storage array. The array was managing to keep pace with Exchange, but just didn't have the ability to also keep up with DNS requests.
To make a long story short, I shut down all of the virtual machines and reallocated the memory. Rather than dedicating 2 GB to each operating system, I stole a gigabyte from the DNS server and another gigabyte from the host partition. That allowed me to assign 4 GB to the Exchange Server.
When I rebooted all of the servers, I discovered that they all performed very well. It seems strange to think that I got better performance from my DNS server by giving it less memory to work with, but such is the sometimes strange world of server virtualization.
About the Author
Brien Posey is a 22-time Microsoft MVP with decades of IT experience. As a freelance writer, Posey has written thousands of articles and contributed to several dozen books on a wide variety of IT topics. Prior to going freelance, Posey was a CIO for a national chain of hospitals and health care facilities. He has also served as a network administrator for some of the country's largest insurance companies and for the Department of Defense at Fort Knox. In addition to his continued work in IT, Posey has spent the last several years actively training as a commercial scientist-astronaut candidate in preparation to fly on a mission to study polar mesospheric clouds from space. You can follow his spaceflight training on his Web site.