Posey's Tips & Tricks
Hyper-V Replication Lessons Learned
Avoiding this configuration blunder should save you some considerable time.
I recently found myself in a situation in which I was forced to replace all of my Hyper-V servers (it’s a long story). In my haste to get back online, I made a simple, yet consequential configuration error that I wanted to show you.
Before I tell you what happened and how I fixed the problem, I need to tell you a little bit about my Hyper-V environment. I have two production host servers, each of which are connected to dedicated external storage arrays by way of a 10 Gbe connection. This external storage array contains a folder called F:\VMs, where my virtual machines reside.
In this particular case, I ended up having to install a temporary server and set up my production VMs on it. My storage arrays were unaffected by the disaster, so I simply connected the new VMs to existing virtual hard disks.
When my replacement hardware arrived, I installed one of the new servers, connected it to a storage array, and then used Hyper-V replication to replicate my virtual machines to the new server.
When the replication process completed, I performed a planned failover so that my VMs were running on the new host. At that point, I disabled replication, removed my temporary server, and replaced it with another new server. I then enabled replication and began replicating my VMs to the new server. This is where the problems began.
A few hours into the replication process, the replication failed. I didn’t receive an error message on the screen, but the Windows event logs contained hundreds of errors. The primary host simply indicated that replication could not be performed, but didn’t really give a reason why. You can see an example of such an error in Figure 1.
[Click on image for larger view.] Figure 1. Replication failed for no apparent reason.
The recipient host (the Hyper-V machine that I was trying to replicate to) also contained error messages in its event log. Most of these error messages didn't provide any hints as to the cause of the problem. However, some mentioned that a network error might have been to blame.
After wasting time looking for a non-existent network error, I decided to remove and reconfigure Hyper-V replication. It was then that I spotted the problem. When you enable a Hyper-V host to act as a replica server, there is an option to specify the location within which you want to store the replica files, as shown in Figure 2. For whatever reason, I had forgotten to modify the default path so that it pointed at F:\VMs.
[Click on image for larger view.] Figure 2. It's important to define the default location for replica files.
Because I had forgotten to change the default location for replica files, Windows tried to store them all on the C:\ drive. This meant that the replication process worked normally until Hyper-V depleted all the space on the server's C: drive. At that point, replication failed and my server slowed to a crawl.
Obviously, the fix for this problem was to correct the path that is to be used by replicated virtual machines. However, changing the path alone did not cause Hyper-V to return the hard disk space that it had claimed.
To reclaim my lost disk space, I deleted the replica virtual machine from the replica host. I then went to C:\ProgramData\Microsoft\Windows\Hyper-V and deleted everything within that folder (although, there were some small files that I couldn't delete). I also tried to delete the files from C:\ProgramData\Microsoft\Windows\Virtual Hard Disks, but found the folder to be empty.
After a bit of looking around, I discovered that the replicated virtual hard disk files were being stored in C:\Users\Public\Public Documents\Hyper-V\Hyper-V\Virtual Hard Disks\Hyper-V Replica\Virtual Hard Disks\<VM identifier>. You can see what this location looks like in Figure 3.
[Click on image for larger view.] Figure 3. Hyper-V placed replication data in the Public Documents folder.
I deleted the Hyper-V folder and all subfolders from beneath the Public Documents folder. This had the desired effect of returning all of my hard disk space, as shown in Figure 4. Now that I once again had plenty of free space on my C: drive, I was able to move forward with replicating my virtual machine.
[Click on image for larger view.]
Figure 4. The C: drive now has plenty of free space.
About the Author
Brien Posey is a 22-time Microsoft MVP with decades of IT experience. As a freelance writer, Posey has written thousands of articles and contributed to several dozen books on a wide variety of IT topics. Prior to going freelance, Posey was a CIO for a national chain of hospitals and health care facilities. He has also served as a network administrator for some of the country's largest insurance companies and for the Department of Defense at Fort Knox. In addition to his continued work in IT, Posey has spent the last several years actively training as a commercial scientist-astronaut candidate in preparation to fly on a mission to study polar mesospheric clouds from space. You can follow his spaceflight training on his Web site.