First Things Should Never Be Done Second
Things fall apart before an admin's first security inspection.
A few years ago I was given the responsibility of managing a very small but
important network, which had a mix of a few Windows and Unix servers. At around
the one-year anniversary of managing this system, I was preparing for my first
It was a Thursday morning and my inspector was arriving in less than 24 hours.
I was a bit worried and I wanted everything to go well. I had recently implemented
a new tape backup device and software that was pretty difficult to administer.
I named my backup tapes the same so the software could easily read and overwrite
them each night.
Fear of Embarrassment
On this day I remembered that one of my tapes was named incorrectly and had
been giving me problems so I renamed (and reformatted) all six of my backup
tapes. I did this just in case I had to load them for my inspector. Why? Because
I would rather have blank tapes, than tapes that failed to load up.
Soon afterward, I was thinking about the Primary Windows Domain Controller.
I knew that the domain security policies were weak so I feared the inspector's
security posture testing would produce embarrassing results.
I decided to take matters into my own hands. On the Primary Domain Controller,
I installed one of his reporting tools that evaluated more than 100 patches,
domain security settings, file permissions and so on. As you've probably guessed,
my results were bleak, so I decided to modify some of the settings. After making
the changes, I remembered to make a current backup. It took three hours, but
at least I had one on hand.
Unfortunately, when I rebooted the server, it didn't start up. After I finagled
it for a few hours it booted, but didn't recognize the domain. I restored it,
but the tape had some of the modified settings and was still useless.
By mid-evening I had to face the music. I had crippled the Windows PKI and
KRE systems on the Domain Controller. What happened was it had pushed out the
settings to the backup Domain Controller, too. Ultimately, I destroyed the Active
Directory and the domain security policies, and I couldn't restore or rejoin
the domain. Even worse, I needed this system back up the next day.
I ended up spending weeks rejoining workstations, rebuilding domains and restructuring
profiles. Eventually, things became normal again. I didn't lose my job, but
I lost some credibility.
Your Worst IT Nightmare?
|Write up your story
in 300-600 words and e-mail it to Editor Ed Scannell at firstname.lastname@example.org.
Use "Never Again" as the subject line and be sure
to include your contact information for story verification.
Doing a post mortem, I realized I had spent the prior weeks doing non-critical
tasks for management when I knew my backup situation was hanging on by a thread.
Now I know that an SA should tend to the most critical tasks first,
and ensure that management understands why.
I realized that I never should have reformatted those tapes.
Well, wouldn't you know, that after this horrible experience and spending all
night trying to restore the system, my inspector showed up that Friday. We logged
into the backup server and ran his report, found the three or four items that
he was interested in, and corrected them on the spot.
When I mentioned the fiasco (later) he laughed and told me that he wanted to
assess the system and determine what would be a problem -- he didn't want me
to fix the problem. I should have found out what the inspector wanted from me
-- a typical newbie mistake.
The submitter of this "Never Again" story wishes to remain anonymous.