Never Again

First Things Should Never Be Done Second

Things fall apart before an admin's first security inspection.

A few years ago I was given the responsibility of managing a very small but important network, which had a mix of a few Windows and Unix servers. At around the one-year anniversary of managing this system, I was preparing for my first security inspection.

It was a Thursday morning and my inspector was arriving in less than 24 hours. I was a bit worried and I wanted everything to go well. I had recently implemented a new tape backup device and software that was pretty difficult to administer. I named my backup tapes the same so the software could easily read and overwrite them each night.

Fear of Embarrassment
On this day I remembered that one of my tapes was named incorrectly and had been giving me problems so I renamed (and reformatted) all six of my backup tapes. I did this just in case I had to load them for my inspector. Why? Because I would rather have blank tapes, than tapes that failed to load up.

Soon afterward, I was thinking about the Primary Windows Domain Controller. I knew that the domain security policies were weak so I feared the inspector's security posture testing would produce embarrassing results.

I decided to take matters into my own hands. On the Primary Domain Controller, I installed one of his reporting tools that evaluated more than 100 patches, domain security settings, file permissions and so on. As you've probably guessed, my results were bleak, so I decided to modify some of the settings. After making the changes, I remembered to make a current backup. It took three hours, but at least I had one on hand.

Unfortunately, when I rebooted the server, it didn't start up. After I finagled it for a few hours it booted, but didn't recognize the domain. I restored it, but the tape had some of the modified settings and was still useless.

By mid-evening I had to face the music. I had crippled the Windows PKI and KRE systems on the Domain Controller. What happened was it had pushed out the settings to the backup Domain Controller, too. Ultimately, I destroyed the Active Directory and the domain security policies, and I couldn't restore or rejoin the domain. Even worse, I needed this system back up the next day.

I ended up spending weeks rejoining workstations, rebuilding domains and restructuring profiles. Eventually, things became normal again. I didn't lose my job, but I lost some credibility.

What's Your Worst IT Nightmare?
Write up your story in 300-600 words and e-mail it to Editor Ed Scannell at escannell@redmondmag.com. Use "Never Again" as the subject line and be sure to include your contact information for story verification.

Lesson Learned
Doing a post mortem, I realized I had spent the prior weeks doing non-critical tasks for management when I knew my backup situation was hanging on by a thread.

Now I know that an SA should tend to the most critical tasks first, and ensure that management understands why.

I realized that I never should have reformatted those tapes.

Well, wouldn't you know, that after this horrible experience and spending all night trying to restore the system, my inspector showed up that Friday. We logged into the backup server and ran his report, found the three or four items that he was interested in, and corrected them on the spot.

When I mentioned the fiasco (later) he laughed and told me that he wanted to assess the system and determine what would be a problem -- he didn't want me to fix the problem. I should have found out what the inspector wanted from me -- a typical newbie mistake.

About the Author

The submitter of this "Never Again" story wishes to remain anonymous.

comments powered by Disqus

Reader Comments:

Wed, Jan 16, 2008 Roger Zan Hartsdale, NY USA

I agree with Christopher's comment completely. Before making any changes to a system, whether or not one feels confident of the results, it's imperative to always have a good backup of the data to be changed as well the registry and system state. A common mistake made by newbies in this field as well as some overly confident seasoned pros is that they just simply jump in and make changes on a whim without a proper change control process or means for failing back if things go bad quickly. Unfortunately, Microsoft's certification process does not emphasize this point, but one must use logic and "common" sense when planning on a major system change. Too bad common sense is not very common though and careless mistakes will continue to be made by those with fast fingers who don't think through the consequences of their actions.

Wed, Jan 9, 2008 Christopher Bell Glossop, UK

Me, I blame Microsoft. I have been teaching MOC for years and it never fails to rile (and frequently amuse) me when I read such things as "after making changes to the master database make a backup". Doesn't quite work does it? Take a backup so that your fallback position should things go wrong is exactly where you are! I always thought that you should take a backup before you mess with things so that your fallback is where you were before you started messing. I would be interested to know whether Mr(s) Anonymous had been taught this.

Add Your Comment Now:

Your Name:(optional)
Your Email:(optional)
Your Location:(optional)
Comment:
Please type the letters/numbers you see above

Redmond Tech Watch

Sign up for our newsletter.

I agree to this site's Privacy Policy.