In-Depth

The Expiration Date That Did Us In

IT people move on; so, who's doing backups?

We run a small Windows NT network and Access database. A different technician set up our server, but I had no information about it. I’d checked it over about 18 months before our disaster and determined that the backups ran fine and everything seemed to work smoothly. After the initial inspection, I checked the server about once a month.

One day, the database was acting strangely, so we checked other files stored on the server. Some wouldn’t open. We tried to reboot the server, for the first time in months. On the reboot we heard grinding, then nothing. The hard drive was fried. Well, that’s OK; it happens, right? So we bought a new hard drive. While we were at it, we got two, since they’re cheap, and it would allow us to set up a mirrored drive.

Also in this issue:

 Get Active Directory Replication Right!
by Andrew Lindley

 Exchange 2000 Upgrade, Times Two
by Cynthia Balusek

 Wireless Meets Mother Nature
by Justin Melot

 Troubleshooting Under Pressure
by James D. Pollock

 Hard Drive Fall Down, Go Boom!
by Christopher M. Roscoe

(Back to introduction.)

We brought the server back up to normal with the new drives. The next step was to restore the data from backup. That’s when we got a nasty surprise: It turned out that the information on the tapes was six months old. This obviously wouldn’t work. We tried to troubleshoot a little more, but with no luck—the new information just wasn’t there. For some reason the backup just quit working after the first year.

We sent the failed drive to a restoration company. A week and a half later the company told us the drive was too badly damaged, and they weren’t able to recover anything. We were out of luck. We lost half a year’s worth of data and were forced to use data that was six months old.

Following this trauma, I was determined to find out what had gone wrong. After checking into it deeper, I found out the tapes we used for backup had an expiration date set when they were initially formatted. Once it reached that date, they quit working.

Lesson learned: Never let somebody else set up your server and backup scheme without finding out exactly what that person did.

The new steps we’ve taken to eliminate data loss include implementing a RAID solution, with a monthly double check of the backups; labeling tapes with their expiration dates; and copying server contents to another computer weekly for additional redundancy.

It may seem less than earth-shaking, but for a small company, it sure was a disaster. We’re still trying to recover from that little incident.

About the Author

Jeremy Dillinger, MCSE+I, does tech support at a local company and for his local school district.

comments powered by Disqus

Reader Comments:

Tue, Oct 1, 2002 Anonymous Anonymous

Great article

Sat, Sep 28, 2002 Scott Spiess Redding, CA

Ahhh but let us not forget about AD and its tombstone dates too. Microsoft by default will not allow you to restore an AD tree that is over a certain time. Unless you want to get really good at typing in user accounts and profile paths, double check your backups and don't rely on that one tape that you might have stashed from the first server backup.

Sat, Sep 21, 2002 Carl Arizona

Wow! We have enough trouble getting the administration of our SQL server straight so that a backup is scheduled. The old administrator quit to start his own software business somewhere else and the new guy is way overtasked, wearing three different hats and working for peanuts. Problems like the one mentioned here never occurred to me (A tape is a tape, right?). We're in a very technical, scientific environment where data rules. My business analysis: may our good luck continue, or I hope I'm not there when the s___ hits the fan.

Fri, Sep 20, 2002 Katie Bellingham

Nice work Jeremy!!

Wed, Sep 18, 2002 Edward Liu San Jose

Good! I don't have similar situation but I heard the other story about tape recovery. When the hard disk failed, the data has backed up on tape successfully. But the tape restore took 5 full days to complete. It was just extremely slow on the data recovery from tape.
Have you ever tried to do data recovery from tape before? You better do it before it happens to you!!!

Wed, Sep 18, 2002 Mark Phillips Seattle

Great lesson for anyone new to our field.
However ALL administrators that are responsible for the organizations data should make it 100% certain that they know they can restore from a disaster, which includes proper test restores and simulated disasters to help identify and mitigate risk. The administrator deserved this hard learned lesson because ultimately they are responsible, and inevitabily a disaster is sure to happen to all of us.

Wed, Sep 18, 2002 Jim Anonymous

I had a similar situation as Ken. I had two drives in a RAID config fail. I tried to restore the backup, but it had been overwritten with bad data from the hard drives before they failed. The users were able to regenerate most of the data, and other data we were able to get from some of our vendors, so it wasn't a complete disaster. However, it's certainly one that I never want to repeat.

Wed, Sep 18, 2002 Ken New York

Excellent lesson learned. I had a similar nightmare with an Exchange Server 6 months ago, only my mistakes were twofold. First, I got too busy to check the detail logs for the backup system and missed that although the backup was marked as a success, the Exchange DBs stopped backing up 4 months prior. The second was letting my boss intimidate me into not buying adding in RAID or extra drives to create addiquate mirrors when I built the Exchange server. The myopic decision that saved maybe $2.5k, cost us around $20k in recovery costs (plus the down time and long nights I put in). Fortunately, Ontrack was able to recover 96% of the data I needed from the drive or there would have been some mighty angry employees. Won't let that happen again!

Wed, Sep 18, 2002 Anonymous Anonymous

excellent

Add Your Comment Now:

Your Name:(optional)
Your Email:(optional)
Your Location:(optional)
Comment:
Please type the letters/numbers you see above

Redmond Tech Watch

Sign up for our newsletter.

I agree to this site's Privacy Policy.