Never Again

First Things Should Never Be Done Second

Things fall apart before an admin's first security inspection.

• By Anonymous
• 01/01/2008

It was a Thursday morning and my inspector was arriving in less than 24 hours. I was a bit worried and I wanted everything to go well. I had recently implemented a new tape backup device and software that was pretty difficult to administer. I named my backup tapes the same so the software could easily read and overwrite them each night.

Fear of Embarrassment
On this day I remembered that one of my tapes was named incorrectly and had been giving me problems so I renamed (and reformatted) all six of my backup tapes. I did this just in case I had to load them for my inspector. Why? Because I would rather have blank tapes, than tapes that failed to load up.

Soon afterward, I was thinking about the Primary Windows Domain Controller. I knew that the domain security policies were weak so I feared the inspector's security posture testing would produce embarrassing results.

I decided to take matters into my own hands. On the Primary Domain Controller, I installed one of his reporting tools that evaluated more than 100 patches, domain security settings, file permissions and so on. As you've probably guessed, my results were bleak, so I decided to modify some of the settings. After making the changes, I remembered to make a current backup. It took three hours, but at least I had one on hand.

Unfortunately, when I rebooted the server, it didn't start up. After I finagled it for a few hours it booted, but didn't recognize the domain. I restored it, but the tape had some of the modified settings and was still useless.

By mid-evening I had to face the music. I had crippled the Windows PKI and KRE systems on the Domain Controller. What happened was it had pushed out the settings to the backup Domain Controller, too. Ultimately, I destroyed the Active Directory and the domain security policies, and I couldn't restore or rejoin the domain. Even worse, I needed this system back up the next day.

I ended up spending weeks rejoining workstations, rebuilding domains and restructuring profiles. Eventually, things became normal again. I didn't lose my job, but I lost some credibility.

Lesson Learned
Doing a post mortem, I realized I had spent the prior weeks doing non-critical tasks for management when I knew my backup situation was hanging on by a thread.

Now I know that an SA should tend to the most critical tasks first, and ensure that management understands why.

I realized that I never should have reformatted those tapes.

Well, wouldn't you know, that after this horrible experience and spending all night trying to restore the system, my inspector showed up that Friday. We logged into the backup server and ran his report, found the three or four items that he was interested in, and corrected them on the spot.

When I mentioned the fiasco (later) he laughed and told me that he wanted to assess the system and determine what would be a problem -- he didn't want me to fix the problem. I should have found out what the inspector wanted from me -- a typical newbie mistake.