Posey's Tips & Tricks
Why Most Backup Success Metrics Are Meaningless
Traditional backup metrics can show perfect health while failing to reveal whether critical workloads can actually be restored.
Imagine a situation in which a backup dashboard is all green, indicating that all is well. Shortly thereafter, a critical failure prompts the organization to restore a workload from backup. In spite of the backup dashboard indicating that the backups are good, the restoration process fails. The question is, why?
In this particular case, the backup dashboard presented the backup as being healthy, when clearly it was not. The problem isn't so much that the backup dashboard lied about the backup health, but rather that the backup dashboard is tracking the wrong health metrics. Over the course of decades, IT pros have been conditioned to track certain KPIs as a way of assessing backup health. In recent years however, both workloads and the backups that protect them have evolved and so metrics that could once be relied upon have become far less relevant than they once were.
The problem with backup success metrics is that you and the backup application may have very different definitions of what constitutes a successful backup. While it's true that backup applications tend to be more sophisticated than they once were, there are still some backup applications that define success as having copied the protected data and then verified that the data was copied accurately.
But what about the various metrics that are tracked by the backup application as a way of assuring backup admins of backup health? Although every backup application tracks its own unique set of metrics, some of the more commonly tracked metrics include: the backup job success rate, deduplication ratios, storage capacity utilization, throughput, the number of protected workloads, and backup retention compliance. While these metrics can undeniably be useful, they do little to answer the question of whether a workload can be reliably restored.
So how can a backup be unusable when all indications are that backups have been completing successfully? There are actually quite a few possible reasons. As an example, a recovery might fail as a result of a corrupted restore chain. Imagine for instance, that an organization performs “incremental forever” backups. If even one of the incremental backups happens to be damaged, missing, or even just stored in the wrong location, it can cause the entire restoration to fail.
I have also seen something similar happen in Hyper-V environments. Hyper-V allows you to create checkpoints that serve as a tool for rolling a virtual machine back to an earlier state. When you create a checkpoint, what actually happens behind the scenes is that Hyper-V creates a differencing disk for the virtual machine. At that point, the virtual machine's virtual hard disk becomes read only (preserving its previous state) and all future write operations are directed to the differencing disk. I have seen situations in which an organization does not take checkpoints into account when backing up Hyper-V virtual machines. They might continue backing up the original virtual hard disk, but neglect to backup the differencing disk. This would result in all of the information stored within the checkpoint being lost in the event of a restoration.
The opposite can also happen. If an organization backs up the differencing disk, but not the original virtual hard disk, then the backup will be useless since the differencing disk is meaningless without the corresponding virtual hard disk.
Another reason why restorations can fail is because of the distributed nature of modern workloads. Let's pretend for a moment that a particular workload depends on a number of different components that are scattered among any number of virtual machines or containers. Now let's suppose that the workload fails and that IT decides that the failed VM or container needs to be restored from backup.
Even if the restoration succeeds, the restore operation can break the workload. If you were to restore only a single component of a larger workload, you could cause any number of problems ranging from version inconsistencies to expired encryption keys or credential failures.
Another reason why recovery operations might fail stems from missing dependencies. Modern applications depend on any number of external components such as identity systems, cloud control planes, certificates, APIs, orchestration layers, and more. If any of these dependencies were accidentally excluded from the backup then it may become impossible to recover a workload that depends on the missing dependency.
This actually points to yet another reason why backup metrics can be misleading. Suppose that an organization's backup software reports a success rate of 99.5 percent. That sounds excellent. The problem is that not all workloads are equally important. What if that missing 0.5 percent happened to include the Active Directory, the DNS, an ERP database, or your database transaction logs? How many other workloads could that situation potentially impact?
Although the backup metrics that we have all relied upon for years have their place, a more useful metric might be a recovery confidence score (some backup applications do provide something similar). This score could be calculated based on things like testing frequency, past recovery success, dependency mapping and backup job maturity.
About the Author
Brien Posey is a 22-time Microsoft MVP with decades of IT experience. As a freelance writer, Posey has written thousands of articles and contributed to several dozen books on a wide variety of IT topics. Prior to going freelance, Posey was a CIO for a national chain of hospitals and health care facilities. He has also served as a network administrator for some of the country's largest insurance companies and for the Department of Defense at Fort Knox. In addition to his continued work in IT, Posey has spent the last several years actively training as a commercial scientist-astronaut candidate in preparation to fly on a mission to study polar mesospheric clouds from space. You can follow his spaceflight training on his Web site.