Posey's Tips & Tricks
Taking Hyper-V Health Monitoring to the Next Level, Part 2
Let's launch a PowerShell script that goes beyond replication health to evaluate Hyper-V failover readiness by checking storage, networking, memory, VM configuration and other key conditions that could determine whether a virtual machine can successfully fail over.
In the first part of this series, I explained that while I recently created a tool designed to assess Hyper-V replication health, that tool did little to determine whether or not it would actually be possible to fail a virtual machine over to another host. As such, I have created a new script that tests various conditions to assess the possibility of a successful failover. This script will eventually become part of a more comprehensive tool, but for right now, I wanted to share my health checks with you.
I outlined the first three health checks in the previous article. The fourth health check is designed to make sure that all of the virtual machine’s virtual hard disks actually exist. A score is calculated by dividing the number of virtual hard disks that have been confirmed to exist by the total number of virtual hard disks that should exist and then multiplying the answer by 10.
The next health check tests the virtual machine version number. This test is really simple. If the virtual machine version number is 9 or higher, then 10 points are awarded. Otherwise, no points are awarded and a message is displayed indicating that the virtual machine version is outdated.
The sixth test checks to see if the virtual machine contains any checkpoints. The existence of a checkpoint won’t normally cause a virtual machine failover problem, but checkpoints can slow things down and in extreme cases, lengthy checkpoint trees might be problematic. As such, I simply tested for checkpoints. If no checkpoints are found then I award 10 points.
Test number 7 is effectively a ping test that is designed to determine whether the replica host is reachable. If the test succeeds, I award 10 points.
I seriously considered triggering a critical failure if the connectivity test fails. However, I didn’t do that because if the replica host’s firewall is configured to block ICMP traffic, then a ping test will fail even when nothing is wrong.
The final test is a memory test that is designed to make sure that the replica host has enough free memory to run the virtual machine. The script does trigger a critical failure if there is not enough memory available.
One thing to keep in mind is that a successful test does not absolutely guarantee that sufficient memory will be available. If multiple virtual machines were to failover at the same time, then they could potentially deplete the replica host’s memory.
Once the various tests are complete, a final status is determined. The script will indicate that the status will be either Ready, Warning, At Risk or Failover is Impossible. The final status is determined by the overall score. A score of at least 80 is required for a status of Ready. The reason why I picked this number is because if the replication health were to be in a warning state, then the only way that the VM would be considered ready would be if every single test was successful.
So with that said, I want to share with you the code that I wrote. Here it is:
$Global:HyperVContext = @{
Credential = Import-Clixml -Path C:\Healthcheck\Cred.xml
}
Function Invoke-HyperVCommand {
Param (
[String]$ComputerName,
[ScriptBlock]$ScriptBlock,
[Object[]]$ArgumentList
)
If (-Not $Global:HyperVContext.Credential) {
Throw "Credential not initialized"
}
Invoke-Command -ComputerName $ComputerName `
-Credential $Global:HyperVContext.Credential `
-ScriptBlock $ScriptBlock `
-ArgumentList $ArgumentList `
-ErrorAction Stop
}
Function Get-HyperVRemoteData {
Param (
[String]$VMName,
[String]$ComputerName
)
Invoke-HyperVCommand -ComputerName $ComputerName -ScriptBlock {
Param ($VMName)
$VM = Get-VM -Name $VMName
$Replication = Get-VMReplication -VMName $VMName -ErrorAction SilentlyContinue
$NICs = Get-VMNetworkAdapter -VMName $VMName
$Disks = Get-VMHardDiskDrive -VMName $VMName
$Checkpoints = Get-VMSnapshot -VMName $VMName -ErrorAction SilentlyContinue
$VMSwitches = Get-VMSwitch | Select-Object -ExpandProperty Name
Return [PSCustomObject]@{
VM = $VM
Replication = $Replication
NICs = $NICs
Disks = $Disks
Checkpoints = $Checkpoints
VMSwitches = $VMSwitches
HostName = $env:COMPUTERNAME
}
} -ArgumentList $VMName
}
Function Get-FailoverReadinessScore {
Param (
[String]$VMName,
[String]$ComputerName
)
$CriticalFailure = $False
$Data = Get-HyperVRemoteData -VMName $VMName -ComputerName $ComputerName
$VM = $Data.VM
$Replication = $Data.Replication
$NICs = $Data.NICs
$Disks = $Data.Disks
$Checkpoints = Get-VMSnapshot -VMName $VMName -ErrorAction SilentlyContinue
$ReplicaHost = $Replication.ReplicaServer
$Score = 0
$MaxScore = 100
$Issues = @()
# ---------------------------
# Replication Health (30 Possible Points)
# ---------------------------
Switch ($Replication.Health) {
"Normal" { $HealthScore = 30 }
"Warning" { $HealthScore = 10; $Issues += "Replication warning" }
"Critical" { $HealthScore = 0; $Issues += "Replication in critical state"; $CriticalFailure = $True }
Default { $Issues += "Replication not configured" }
}
Write-Host ("Replication Health: {0}" -f $Replication.Health)
Write-Host "The replication health score is $HealthScore out of 30"
$Score = $Score + $HealthScore
# ---------------------------
# Replication Freshness (10 Possible Points)
# ---------------------------
If ($Replication.LastReplicationTime) {
$Age = (Get-Date) - $Replication.LastReplicationTime
$AgeMinutes = [Math]::Round($Age.TotalMinutes, 2)
If ($Age.TotalMinutes -lt 5) {
$LagScore = 10
$Score = $Score + $LagScore
}
ElseIf ($Age.TotalMinutes -lt 10) {
$LagScore = 5
$Score = $Score + $LagScore
$Issues += "Replication lag of under 10 minutes"
Write-Host "Replication lag of under 10 minutes" -ForegroundColor Orange
}
Else {
$Issues += "Stale replication ($([Math]::Round($Age.TotalMinutes)) minutes)"
Write-Host "Stale replication ($([Math]::Round($Age.TotalMinutes)) minutes)" -ForegroundColor Red
$LagScore = 0
}
Write-Host "The most recent replication occured $AgeMinutes minutes ago"
Write-Host "The replication freshness score was $LagScore out of 10"
}
# ---------------------------
# Network Mapping (10 Possible Points)
# ---------------------------
$ReplicaSwitches = Invoke-HyperVCommand -ComputerName $ReplicaHost -ScriptBlock {
Get-VMSwitch | Select-Object Name, SwitchType
}
$ValidNICs = 0
$TotalNICs = 0
foreach ($NIC in $NICs) {
$TotalNICs++
# Skip disconnected NICs (they are irrelevant for failover boot networking)
if (-not $NIC.Connected) {
$Issues += "NIC '$($NIC.Name)' is disconnected"
continue
}
$SourceSwitch = $NIC.SwitchName
# Check to see if the NIC is bound to a switch
If ([string]::IsNullOrWhiteSpace($SourceSwitch)) {
# Require at least one External switch on replica host
$HasExternal = $ReplicaSwitches | Where-Object SwitchType -eq "External"
If ($HasExternal) {
$ValidNICs++
}
else {
$Issues += "No External virtual switch available on replica host for NIC '$($NIC.Name)'"
Write-Host "No External virtual switch available on replica host for NIC '$($NIC.Name)'" -ForegroundColor Red
}
continue
}
# Case 2: NIC is bound to a specific switch
$Match = $ReplicaSwitches | Where-Object Name -eq $SourceSwitch
If ($Match) {
$ValidNICs++
}
else {
$Issues += "Required switch '$SourceSwitch' for NIC '$($NIC.Name)' does not exist on replica host"
Write-Host "Required switch '$SourceSwitch' for NIC '$($NIC.Name)' does not exist on replica host" -ForegroundColor Red
}
}
# Score calculation
If ($TotalNICs -gt 0) {
$NICScore = [Math]::Round(($ValidNICs / $TotalNICs) * 10)
$Score += $NICScore
Write-Host "Network mapping score: $NICScore out of 10"
}
else {
Write-Host "No NICs found to evaluate"
}
# ---------------------------
# Storage
# ---------------------------
$ValidDisks = 0
ForEach ($Disk in $Disks) {
If ($Disk.Path) {
$ValidDisks++
}
Else {
$Issues += "Invalid disk path detected at $Disk.path"
Write-Host "$Disk.path was invalid" -ForegroundColor Red
}
}
If ($Disks.Count -gt 0) {
$DiskScore = [Math]::Round(($ValidDisks / $Disks.Count) * 10)
Write-Host "The disk score was $DiskScore out of 10"
$Score = $Score + $DiskScore
}
# ---------------------------
# VM Version (10 Possible Points)
# ---------------------------
If ($VM.Version -ge 9) {
$Score = $Score + 10
Write-Host "The VM Version score was 10 out of 10"
}
Else {
$Issues += "Outdated VM configuration version"
Write-Host "The VM version is outdated" -ForegroundColor Red
Write-Host "The version score is 0 out of 10" -ForegroundColor Red
}
# ---------------------------
# Checkpoints (10 Possible Points)
# ---------------------------
If (-Not $Checkpoints) {
$Score = $Score + 10
Write-Host "No checkpoints present"
Write-Host "The checkpoint score is 10 out of 10"
}
Else {
$Issues += "Checkpoints present"
Write-Host "Checkpoints present" -ForegroundColor Red
Write-Host "The checkpoint score is 0 out of 10" -ForegroundColor Red
}
# ---------------------------
# Replica Host Reachability (10 Possible Points)
# ---------------------------
$Reachable = Test-Connection -ComputerName $ReplicaHost -Count 1 -Quiet
If ($Reachable) {
Write-Host "The replica host is reachable"
Write-Host "The reachability score is 10 of 10"
$Score = $Score + 10
}
Else {
$Issues += "Replica host not reachable"
Write-Host "The replica host is not reachable" -ForegroundColor Red
Write-Host "The reachability score is 0 out of 10" -ForegroundColor Red
}
# ---------------------------
# Memory Capacity (10 Possible Points)
# ---------------------------
$VMMemoryMB = [Math]::Round($VM.MemoryStartup / 1MB, 0)
$OS = Invoke-HyperVCommand -ComputerName $ReplicaHost -ScriptBlock {
Get-CimInstance Win32_OperatingSystem
}
$FreeMemoryMB = [Math]::Round($OS.FreePhysicalMemory / 1024, 0)
Write-Host "Free Memory on Replica Host: $FreeMemoryMB MB"
Write-Host "Memory Required by the Virtual Machine: $VMMemoryMB MB"
If ($FreeMemoryMB -ge $VMMemoryMB) {
Write-Host "The replica host has enough memory to run the VM"
Write-Host "The memory score is 10 out of 10"
$Score = $Score + 10
}
Else {
$Issues += "Insufficient memory on replica host"
$Score = 0
$CriticalFailure = $True
Write-Host "The replica host has insufficient memory" -ForegroundColor Red
Write-Host "The memory score is 0 out of 10" -ForegroundColor Red
Write-Host "This is a critical condition" -ForegroundColor Red
}
If ($Score -ge 80) { $FinalStatus = "Ready" }
ElseIf ($Score -ge 50) { $FinalStatus = "Warning" }
Else { $FinalStatus = "At Risk" }
If ($CriticalFailure -eq $True) {$FinalStatus = "Failover is Impossible"}
Return [PSCustomObject]@{
VMName = $VMName
Score = $Score
Status = $FinalStatus
Issues = ($Issues -join "; ")
TimeStamp = Get-Date
ReplicaHost = $ReplicaHost
}
}
################################
### Main Body ###
################################
Write-Host "Using credential: $($Global:HyperVContext.Credential.UserName)"
# ----------------------------
# Define test targets
# ----------------------------
### Be sure to update the $ComputerName and $VMList to include your own host name and virtual machine names.
$ComputerName = "<Your Hyper-V Host>"
$VMList = @("VM1", "VM2")
# ----------------------------
# Run evaluation loop
# ----------------------------
$Results = @()
ForEach ($VM in $VMList) {
Write-Host "Evaluating $VM on $ComputerName..." -ForegroundColor Cyan
Try {
$Result = Get-FailoverReadinessScore -VMName $VM -ComputerName $ComputerName
$Results += $Result
Write-Host "$($Result.VMName): $($Result.Score) ($($Result.Status))" -ForegroundColor Green
}
Catch {
Write-Host "Error processing $VM : $_" -ForegroundColor Red
}
}
# ----------------------------
# Output summary
# ----------------------------
$Results | Format-List
About the Author
Brien Posey is a 22-time Microsoft MVP with decades of IT experience. As a freelance writer, Posey has written thousands of articles and contributed to several dozen books on a wide variety of IT topics. Prior to going freelance, Posey was a CIO for a national chain of hospitals and health care facilities. He has also served as a network administrator for some of the country's largest insurance companies and for the Department of Defense at Fort Knox. In addition to his continued work in IT, Posey has spent the last several years actively training as a commercial scientist-astronaut candidate in preparation to fly on a mission to study polar mesospheric clouds from space. You can follow his spaceflight training on his Web site.