Inside a Domain Controller Nightmare
Enterprise Windows administrators worldwide can relate to cold-sweat-down-the-back moment that's detailed in Wired's new chronicle of the NotPetya attack last summer.
"The Untold Story of NotPetya, the Most Devastating Cyberattack in History," by Andy Greenberg, focuses on the apparently collateral damage to the world's largest shipping conglomerate, A.P. Møller-Maersk, when NotPetya hit last summer.
Posing as a piece of ransomware, NotPetya was actually spreading extremely quickly and encrypting systems' master boot records, rendering them unusable and unrecoverable. Conventional wisdom is that Russia designed the malware to attack Ukraine, but NotPetya brought Maersk's global operations to a halt and cost the giant $250 million to $300 million or more.
Deep in the piece, Greenberg reports on Maersk's NotPetya-related trouble with domain controllers:
Early in the operation, the IT staffers rebuilding Maersk's network came to a sickening realization. They had located backups of almost all of Maersk's individual servers, dating from between three and seven days prior to NotPetya's onset. But no one could find a backup for one crucial layer of the company's network: its domain controllers, the servers that function as a detailed map of Maersk's network and set the basic rules that determine which users are allowed access to which systems.
Maersk's 150 or so domain controllers were programmed to sync their data with one another, so that, in theory, any of them could function as a backup for all the others. But that decentralized backup strategy hadn't accounted for one scenario: where every domain controller is wiped simultaneously. "If we can't recover our domain controllers," a Maersk IT staffer remembers thinking, "we can't recover anything."
Salvation came in the form of a power outage. Frantic calls went out from the recovery operations center near London to hundreds of IT admins in datacenters worldwide.
Maersk's desperate administrators finally found one lone surviving domain controller in a remote office -- in Ghana. At some point before NotPetya struck, a blackout had knocked the Ghanaian machine offline, and the computer remained disconnected from the network. It thus contained the singular known copy of the company's domain controller data left untouched by the malware -- all thanks to a power outage.
Sometimes what seems like bad luck -- say, a power outage knocking down your domain controller -- turns out to be the luckiest thing in the world.
Posted by Scott Bekker on 08/27/2018 at 10:03 AM