Never Again
Sometimes You Have to Say No
One admin realizes too late that he might have taken more than he could handle.
I was hired by a midsize IT department to be the lead Windows engineer. There was no one in this position before. My primary job was to consolidate and unify a decentralized and what I'd call an "everyone-did-what-they-wanted" type of environment. There were multiple Windows NT domains mixed in with Netware and multiple levels of authentication. The goal was to have one internal corporate domain and single sign-on authentication that leveraged Active Directory. I also had to handle data migration and consolidation, mixed in with putting out daily fires and resolving many issues before I could even begin a migration. This project had been underway for two years and run by others with no migration experience, so my timeline was about eight months.
Disaster Strikes
The real horror story began with the testing phase. I was attempting to put all the pieces together to achieve my goals. I had set up parallel Windows-based development environments, but made the error of setting up synchronization to the production Netware platform. This in and of itself was fine, until I decided to break down the development environments.
My blunder was that I forgot that I set up full bidirectional synchronization between my development AD domain and Netware. I deleted the OUs and user accounts in AD before stopping the synchronization. The result was the Netware user accounts started deleting from the tree.
I didn't realize this until we started getting reports from users that they were getting messages regarding their account deletions. At first, I didn't put two and two together. By the time I had, roughly half the 900-plus Netware accounts had been deleted.
To compound the issue, we discovered that our backups had been failing for months with no one noticing. We were forced to restore from months-old data. Needless to say, it was a full 26 hours before we had systems backed up and users logged in and working.
Older and Wiser
The moral of this story is twofold: First, learn to say no and push back when your plate is full and spilling over. I was new to the organization and was trying too hard to prove myself. Consequently, I took on too much and made the rookie mistake of costing myself and others on my team days of work. Second, I should have more regularly monitored the backups. If we had better backups, the impact from this mishap would have been softened.
I finally got through the migrations, and on time, no less. Now, I oversee administration of a single Active Directory domain that provides single sign-on functionality to all corporate resources. Management let me stay on and finish what I started, but having to admit I was the cause of that fiasco was one tough pill to swallow.
About the Author
Brian Mangiaracina is a systems engineer.