Tales from the Trenches: All Through the Night

Wherein our hero spends the wee hours staring down servers, rustling up disks, reaching out for help, and pondering his own stupidity.

It was one of those straightforward “I’ve-done-this-a-million-times-before/should-only-take-a-couple-of-hours” kinds of projects. The task at hand: Install industry-standard backup software (ARCserveIT) onto industry-standard servers (Compaq ProLiants) running an industry-standard operating system (Windows NT 4.0, SP 5) equipped with industry-standard tape drives (Exabyte DATs). I had it all figured out. At about 8 p.m. on a Friday evening, I’d head on over to the co-location facility where our e-commerce servers were housed, pop in the ARCserveIT CD, and make a few mouse clicks. Next, I’d make a test backup, configure the backup schedules, and bring our production Web site back up. I told my wife I’d be home by 11.

When I finally walked out of the data center the next morning, almost 12 hours later, and got in my car to drive home, I reflected on the night’s events.

8 p.m., Friday Evening

Sitting in my office, I go over the game plan. Six NT servers run the company’s Internet e-commerce site. Three of them have internal SCSI tape drives and are currently running NT Backup. I have three ARCserveIT Enterprise licenses, along with a few open file and SQL Server agents. Management has agreed to having our 24x7 site down for a few hours so I can install and configure ARCserveIT.

I collect the license certificates, grab the ARCserveIT CD, and head out the door for the co-location facility.

8:45 p.m.

After proving to the data center tech that I’m authorized to access our equipment, I’m escorted to our server rack. I’ve been here before, but just to take stock of the servers. Tonight will be the first time I actually get my fingernails dirty. I’m pleased that my predecessor, who was primarily a programmer who also wore the hat of a network administrator, chose to use Compaq servers. But as I struggle to connect the monitor, keyboard, and mouse cables to the keyboard/video/mouse switch (KVM) in the rack, I start to wonder if he really knew what he was doing. After finally getting everything connected, I redirect our dot-com address to point to a maintenance page on another server and get to work.

9:30 p.m.

Because I want to minimize the down time of our primary Web server, I decide to install to it first. I pop in ARCserveIT and launch the setup program. I accept most of the default options, and after just a few minutes the system is restarting. I call my wife and tell her that I’m only 10 minutes behind schedule—and that I’ll see her in a couple of hours.

The file copy portion has started up, so I switch back to the first server, which is still initializing that RAID controller.

Suddenly, it offers up a lovely rendition of the infamous BSOD—the Blue Screen of Death! A fatal stop error, involving Compaq’s SCSI controller driver. As I scramble for a piece of paper and something to write with so I can record the particulars of the error, the system reboots itself. That’s when I remember that the Automatic Server Recovery (ASR) was set to restart on errors like this.

I watch the screen as the system restarts. It gets to the RAID initialization again and sure enough, the BSOD returns. It now starts its memory dump. Thankfully, it takes a good 45 seconds or so to write the contents of the 512M of RAM installed on this system, giving me enough time to jot down the significant error information.

11 p.m.

I pause for a moment to figure out how I’m going to fix this. Let’s see, what’s the textbook way to approach this kind of problem? Oh, yeah, the Emergency Repair Disk (ERD)! Only one slight problem. There is no ERD. But, hey, I’m an MCSE—I can fix this.

That’s when it hits me. One of the other servers I’m installing ARCserveIT onto has the same configuration as the one that’s crashed. I’ll just make an ERD from that machine, then use it on the first one. I track down the data center tech and ask her if I can borrow a floppy disk. We both forage through desk drawers and file cabinets and eventually locate a precious disk.

As I eject the newly created ERD, the bubble of elation surrounding me bursts—I realize that I’m also going to need an NT Server CD so I can launch setup so that I can use the repair utility to access the ERD and fix my server. Another panicked call to the data center tech!

12:15 a.m., Saturday

“Hi, this is Kevin again. You wouldn’t happen to have an NT Server CD I could borrow?”

“Wow, um, I can call the guys upstairs and see if they know where one is. I’ll get back to you in a bit.” The guys upstairs are the CCIE types—router gurus who keep the backbone of this co-location facility up and running. I’m sure they won’t know anything about an NT Server CD. But 20 minutes later the tech comes in with a CD. She says the only reason they had one on hand is because another customer of theirs accidentally left it behind earlier that day after installing some new systems.

I pop the CD in, start up the system, and wait for that RAID controller to initialize. The bubble of elation begins to form again. It gets to the point where it prompts for the ERD. I put it in and restore the Compaq SCSI driver. Just like clockwork.

Another restart, another round of waiting for that RAID controller to initialize, and another BSOD! Another burst bubble.

2 a.m.

I then realize that the best tool available to me at this point is the telephone. Time to call tech support. But which tech support? Computer Associate for ARCserveIT support? Compaq? Microsoft? As I think about this, I call my wife to let her know it’s going to be a late one. Funny—she doesn’t seem the least bit surprised.

I decide to call ARCserveIT support first. I mean, everything worked before I installed that software, right? It takes me at least 30 minutes to find the correct number to reach after-hours tech support. Once I finally have the number, I’m pleasantly surprised at how quickly I get a live voice on the other end. After asking me all the usual stuff about software version, serial number, platform and more, the support rep asks for a callback number. The way it works, she explains, is that an on-call support engineer will be paged with my callback number. I ask her how long it takes them to respond. Her reply: Anywhere from 10 minutes to two hours. I wait about half an hour, then decide to bark up a different tree.

4 a.m.

Now it’s Compaq’s turn. Maybe they can help me bring my server back to life. In less than 10 minutes I’m on the phone with an actual support engineer. Way to go, Compaq! I describe the situation and, after a pregnant pause, I’m put on hold while the techie does some research.

Ten minutes and she’s back. She wants me to run through the startup routine, describing each step as it occurs. Oh, boy! I get to wait while that RAID controller initializes yet another time. Of course, the BSOD appears, and I read off the pertinent information.

More holding time while she researches again. This time it’s only five minutes. She wants me to describe the system configuration again, especially how the various SCSI devices are set up. There’s the built-in SCSI controller, to which the tape drive is connected, and there’s the Smart 2DH RAID controller. Connected to the RAID controller are three 9G hard drives, configured as one logical drive with two partitions. The CD drive is connected to the built-in IDE controller.

Potential good news: She’s fairly confident that if we disable the tape drive, we should be able to get past the BSOD. I remind her that it’s an internal tape drive, and that the server is stuffed into a rack without sliding rack rails. It could take hours to get to the drive. No problem, she says. Just go into the server configuration utility and disable the SCSI controller. Sounds like a plan!

I reboot the server and press F10 to access the system utilities. What’s this? No system partition can be found! Did the programmer-cum-network administrator not use Smart Start to configure these servers? I check the other servers and, sure enough, none of them has a Compaq system partition. And, of course, I don’t happen to have a Smart Start CD with me either. I’m starting to get really mad at the programmer who set these systems up in the first place.

5 a.m.

I ask the Compaq engineer to hold on a moment while I download the system configuration utility from Compaq’s Web site. I switch the KVM to another server, get the file, and then start the process to create the floppy. But it actually needs several floppies (four to be exact). All I have is the one we managed to scrounge earlier in the evening. And from that earlier search I know we won’t find any more.

I come up with a plan. After the first floppy is finished I eject it, then switch back to the dead server. I boot from the floppy, and when it prompts for disk number 2, I switch back to the other server, then use that same floppy over again for the second disk. I continue this process for disks 3 and 4, and finally the utility is up and running on the dead server.

I disable the SCSI controller and reboot the server. There’s that RAID initialization again…and no BSOD! The GUI loads, and shortly I’m greeted with the logon screen. As I log in, the system advises me that at least one service has failed to load. I check the event logs and see that the tape driver hasn’t loaded, along with all the ARCserveIT services. That’s fine; that’s what’s supposed to happen with the disabled SCSI controller.

6:15 a.m.

Now we can attempt to rectify the root problem. Just then, the ARCserveIT tech calls, finally responding to the page of four hours ago. I have him call back on the land line, then conference him in with the Compaq engineer. We bring him up to date and discuss the next steps.

The ARCserveIT guy wants to know if we’ve tried the generic SCSI driver instead of the Compaq version. The Compaq tech reluctantly informs us that the on-board SCSI controller is an LSI Logic Symbios board and, yes, there is a standard driver available, but not from Compaq. I go to the LSI Logic Web site and download the generic driver.

I try installing the driver on another server configured like the one that crashed—same model ProLiant, same OS, same tape drive. Works! Then I install ARCserveIT. Still works! Of course, Compaq Insight Manager can’t get all the information from the controller now; to top it off, the Compaq tech tells me her company doesn’t support this driver.

So we’ve narrowed the problem down to the particular combination of the Compaq SCSI controller (with the Compaq driver), ARCserveIT, and the Exabyte Digital Audio Tape drive—they just don’t work together. I uninstall ARCserveIT on both machines, reinstall the Compaq SCSI driver, and bring the Web site back up, knowing that I’ll have to come back some other time to enable the SCSI controller again (this time with a Smart Start CD!).

7:30 a.m.

I call my wife and tell her I’m on my way home. On the following Monday, I sit in the weekly development meeting explaining what happened over the weekend. After I hold the group spellbound for 20 minutes, the product VP asks, “What have you learned from all this?”

“Well,” I reply, “two basic truths: Don’t leave home without installation software and blank media, and programmers make lousy network admins!”

comments powered by Disqus

Reader Comments:

Tue, May 22, 2007 fran usa

2007 salary survey for ast administrator

Fri, Apr 8, 2005 Anonymous Anonymous


Wed, Jul 9, 2003 Andrew Roe Michigan

I have the following certs MCP, MCSA, MCSE 2K, CCNA, CCIP I also have 9 years experience I have found it is very difficult to find anything over the upper 50K limit ... .people just aren't paying like they use to. Unless you willing to work where other people are not an example would be detroit michigan.

Mon, May 26, 2003 techvet962 USA

This survey is absolute crap!!!
I have been certified MCSE since 1997 and at my peak earned about $90K as a MCT. There are way too many certified people now in the industry and average salary has gone to 45K. There is more interest in Cisco certifications. I am changing from the computer field as I have been unemployed for a year! DO NOT BELIEVE THIS MAGAZINE!!! There are few controls for honesty when people fill out these things. Good luck, but consider something other than an MCSE...

Sun, May 25, 2003 Anonymous Anonymous

I have worked in IT for 3 years and I don't even make close to this range even with my MCSA/MCP, A+, Network+ and living in the Dallas Metroplex

Fri, Apr 11, 2003 Anonymous Anonymous

Since the recession, and to many tech. employee got layoff's i do not think companey will pay this kind of amount. I have person working for me for $45000.00 a year who has MCSE and MCSD and MCSE +I with 8 years experience.

Fri, Apr 4, 2003 Anonymous Anonymous

"Don’t leave home without installation software" rings true even two years later. Thanks for an amusing article.

Thu, Jan 23, 2003 SanDieigoTech San Diego

Reading all this I want to let you know my experience. I have an A+,N+,MCP,MCSE,CNA,CNE,CCNA,CCNP IP and I can tell you no matter how many certs you have you need exp. You will be lucky to make 35k a year starting out if that. At this time I finally landed a job paying 60k a year but it wasn't the certs that got me the job, it was experience and a security clearance. The market is extremely saturated with techs looking for jobs.

Wed, Dec 18, 2002 SHIYAS INDIA


Wed, Dec 18, 2002 MUJEEB INDIA


Thu, Dec 5, 2002 Moosefoot NYC

BTW, MCP magazine is absolutely correct. The salaries are down across the board. Associates of mine that were making 100k+ are down to as low as 60k. The market will take some time to come back, if you really like this business you will stay in it and eventually make the money.

Thu, Dec 5, 2002 Moosefoot NYC

I am a 8 year veteran in the IT field. I primarily do Desktop Support in Manhattan, NY. The certification is what you make it. I am an MCSE, MCP + Internet and am making 70k.

Tue, Oct 29, 2002 Dave Israel

Hi to All,
There is nothing true about certifications ... I'm 37 y.old with more that 17 yers expirence, Master of Electronics, MCSE, MCDBA, CCNP and A+ ... and more that one and half years I'm looking for the job ....

The true is the PROTECTIONS

Fri, Oct 18, 2002 gulfaraz ahmed china

I have passed mcse 1998,after that i got traning microsoft sales specialist,ccna and pentium iii officials traning from intel.
I have got job quickly but after 2 years lost job,till now i am doing marketing not networking job. some time i think for new vendor cenrtificate can i found job in uk then what salary will be? I have contined my studies and get training of win2k.

Tue, Jun 25, 2002 Cytoxin PA

These surveys are such a joke. NO one makes the money they promise. The training centers should be sued for using these things in their ads. I'm MCSE, N+, A+, I+, MCT, CIW with 5 yrs experience, and I'm only getting 39k a year! What a joke! I can't wait to see what kind of crap they come up with the 2k2 survey.

Fri, May 17, 2002 John PA

You have to reallize that if you have no experience(1 Mil/year for the company your salary will be there. And yes the survey reflects big cities.

Fri, May 10, 2002 Carla MN

I am an NT4 & W2K MCP working on my MCSA. I have 4 years of experience in IT....still not making a salary in the range shown above, let alone a BONUS! Where do ya'll who are making that much work??

Fri, Apr 26, 2002 Chris T. NYC

Man, do I wish this salary survey was for real. I see at the top 46K for and MCSE with one year experience. I'm and A+, N+, MCSE, MCDBA, and CCNA with 4 years experience. In this market, I'm making 49K, and people tell me I'm lucky! I think someone is blowing smoke up your you know what!! This salary survey is a bunch of bologna!!! P.S. MCSE with one year experience in NYC = maybe 30K if you know somebody or your dating the CIO's daughter!!

Mon, Apr 8, 2002 Les Little Rock, AR

If I take into consideration my location, number of employees, my experience, my job duties. I make a good salary with benefits. I'm a CNE and MCP almost MCSA working toward MCSE. I have 5 years experience as a systems engineer/network admin. I work as network admin for one the largest IT solution providers in Arkansas. I take these surveys for what they are, just surveys. Experience plus self worth and experience get you better money.

Tue, Mar 12, 2002 Anonymous Anonymous

This is what I am talking about. Ali Mansouri from Fresno, CA 5 Certifications and 1 year experience. You guys are killing the rest of us.

Fri, Mar 1, 2002 Naag Hyderabad,India

The survey doesnt reflect the reality.

Wed, Feb 13, 2002 Anonymous Anonymous

A certification is no help without the experience to back it up. Experience will get you further once your foot is in the door.



Sun, Feb 3, 2002 Mr. G. Abbeville, Louisiana

Looks like Anonymous from Louisville knows what is actually happening in the real world. Many people are ill prepared to do the job, but the are well prepared to pass the tests.

Just passing all those tests is not enough to be gainfully employed in this industry; a real technology foundation is needed to enable a person to grow in their job. My real concern is the number of paper CCNAs being cooked up in countless CISCO academies in this state. This is going to be far worse than the number of paper MCSEs that currently litter this industry.

Looks like we will have more CCNAs than CISCO routers!

Pay shown is about right for a practicing and gainfully employed certified person who has a firm foundation to build upon in a growing economy.

Past graduates of the Network Specialist AAT curriculum, which I wrote and was being correctly taught as of 2000 in Lafayette (population about 100k), were starting from about 24,000 to 36,000. With any certification(s) (A+, MCP, CCNA), they would consistently get more than 30K. Since my departure a little over a year ago that has dropped to less than 20K in starting salary for current graduates. The starting salary is probably much lower due to the economy.

An MCSE with a few years experience who is good at their job gets approximately what the pay survey shows. Newly employed MCPs that are properly prepared with a sound foundation in this industry also got about 30,000 at the start of their first year of employment. The only thing I have seen cause a person to get less is attitude!

Certifications are a "+" to a career not the career itself.


Sat, Feb 2, 2002 Anonymous Louisville KY

These figures are probably accurate, for people with 2 years or more experience. The numbers for one year of experience seem a little inflated to me. Keep in mind that this is a national average though, $50K in some places is more like $25K-$30K in a place with average cost of living.

I wasn't a participant in the survey, but when I was an MCSE with 1 year experience I was making $30,000. I've got about 2 years experience now, and I've got 14 MCP's, 2000 MCSE, 2000 MCDBA, MCSA, CCNA, CCDA, A+, Network+, Server+, and I make mid $50's.

And as for all you people looking at this going "wow, all i have to do is pass some tests and I can make $XX/yr!!" that isn't exactly true. An MCSE with 0 years experience is worth about as much as a McDonald's worker, and it would be very difficult to find any IT related job in this economy with just a piece of paper.

The market is flooded with "paper MCSE's", as they're called. People with the paper, but no real world knowledge and no experience. These people would be lucky to make $25K.

The people in real demand are ones such as myself. The paper, plus the experience to show I really do know what the paper says.

Not to discourage anyone from getting into IT, but it's not as easy as taking some tests and instantly getting a huge salary. These salary surveys give a lot of people false hopes, and training centers don't help with their "The average MCSE made $67,000 last year" advertisements. Yeah, the average MCSE also has 4+ years experience, and more credentials than just the MCSE.

Mon, Dec 17, 2001 Øystein Norway

Where did the international survey go. In August this year You hoped to get the international survey online "soon". Will this survey be populated?

Thu, Dec 13, 2001 HA London

Salaries for MCSEs in London are so low one would think it stands for MacDonald Certified Serveur Employee.

Wed, Dec 12, 2001 Helio Andrade London

I find that the salaries in London are so low compared to the US that you can find vacancies for MCT roles like the on below:

MCSE - Win 2000 Technical Trainer - Train To Cisco
Trainers required for established training house. Spin off from established London college. (ideally MCT) with MCSE with Win 2000 (essential) certification. To deliver official Microsoft curriculum to experienced technical students. Additional qualifications such as MCT or specialisms in particular Back Office products (SQL Server/SMS/ Exchange) an advantage. Immediate training in CCNA. Lively, dynamic team!.
Type: Permanent Location: City, Edgware, Hammersmith, Central London Start: ASAP Salary/Rate: 20k-22k + Benefits Posted: 12/12/2001 09:24:05 Reference: JSBH1263.TRAINER.2000.LON

Helio Andrade

Wed, Nov 28, 2001 Rob New Mexico

We need more New Mexico cities listed in next years survey. It would be really cool if we could get charts about how many certifed people there are in the each state listed by certification. For those of us who live/ work in areas that are not one of the main cities listed, it is hard to impress the HR folks with this survey. They state teh data doesn't pertain to our area,and rely on locally rate charts from the local temp agencies.

Tue, Nov 20, 2001 Mark Gregson UK North West


Do you have any details of UK salaries, as I'm about to commence the MSCE 2000 course.


Mark Gregson.

Thu, Nov 15, 2001 Liz Largo, Florida

I would love to make this much money. The only problem is that i can't afford to go back to school, but i am still going to try to go back to school.

Tue, Nov 13, 2001 hahah Anonymous

Good grief, I just noticed this "article" is put out by a MS certification magazine. OF COURSE you should get certified, and of course, get their magazine.

Tue, Nov 13, 2001 bob Anonymous

This makes no sense, nobody makes this kind of money unless they're in a big city. Good luck getting these salaries anywhere else... And if you ARE in a big city, EVERYONE'S salary is higher because everything just plain costs more to live. So to put this survey out of geographic context and to omit the means, standard deviations, etc, makes this entire survey meaningless. You could have one mcse making 200 grand and nine mcses making 30 grand and the average would be highly inflated.

Fri, Nov 9, 2001 Paul Derby

Kev, check around in nottingham, I've seen a couple of places doing it. Go and ask them. Ones that spring to mind are and

Thu, Nov 1, 2001 Ali Mansouri Fresno, CA

Hi, I Hold MCSE(WinNT4.0), MCDBA (Win2000),MCP Win2000(Accelerated Exam),MCP WinNT4.0 ,MCP SQL Server.
What salary can I look for with one year of experience in California?

Tue, Oct 23, 2001 CHAND INDIA


Tue, Oct 23, 2001 Jeanny Miami, FL

I am a senior at Florida International University persuing a BA in MIS, I am considering a MCSD or MCSE, which one is better interms of demand, salary, and future?

Mon, Oct 22, 2001 Moataz ariny Egypt

you are lucky i am mcse, mcdba, ciw...
i earn at most 3600 dollars annualy...may be there is somthing wrong in egypt!!!!

Fri, Oct 19, 2001 Jose Puerto Rico

I live at Puerto Rico, an american territory, a common wealth in fact. And must say that these salaries does not reflect what companies are paying to her IT professional here.
In the south of the PR island, not the capital (San Juan) and MCP with 1 year of experience earn...with much as $24000 per year.
Could be very interesting to analize or investigate why here there is not so respect to MCP titles.

Tue, Oct 16, 2001 MR Anonymous Oregon

Where are you finding the people for this survey. I have been working in the IT industry for the last eight years. Im a MCSE, MCP+I and a Cisco CCNA. I don't make even close to the numbers shown. Im lucky if I can even make 30,000 a year. Im on my way to changing careers because the computer industry is a joke.

Mon, Oct 15, 2001 dolmundo Oporto

I am a windows 2000 MCSE
What should be my salary , no experience

Thu, Oct 11, 2001 Anonymous Anonymous

The salary charts are highly inflated and are misleading to individuals entering the ISD career field. MCSE cert. in Texas, are lucky to make $30, 000 a year.

Thu, Oct 11, 2001 Kev Nottingham England

I am currently stuck in a dead end job going nowhere fast. Im looking towards investing about £4000 in an MCSE 2k course to try to gain a better future. Does anyone know if this is actually worth doing in the UK or will I need tons of experience too.

Wed, Oct 3, 2001 Eric Columbus Ohio

I am a network administrator with basic duties at a fairly large company. I am currently workig on my MCSE. What salary can I look for with one year of experience?

Thu, Sep 13, 2001 Graham England

I particularly liked the 'swap the floppy dance'!! Been there, done that.

Thu, Sep 13, 2001 rajnish india

Im almost mcse in win2000 . So what would be my salary.

Wed, Sep 12, 2001 Rick Temecula Ca.

I. am also looking to change career. I am going to school for mcse.. what do you think the future looks like. I see a lot of people getting there mcse .. are we looking at a over load in the field,,would like replies from any one from So. Ca.

Fri, Aug 31, 2001 Roo Anonymous

I like your salaries. I am an MCP (70-210) without a year's experience. What would my salary be?

Wed, Aug 29, 2001 Gerard Mass.

I am in the process of changing careers from Lettercarrier to Desktop Support. I am attending BUCEC.I am eventually going into Networking.What do you think the chances are with the way the economy is?

Tue, Aug 28, 2001 Ken Berkowitz USA, MD

Are these salary ranges for REAL? I am thinking of switching careers and want to find out more. I am not a "kid" and have worked with PC's as a hobby since 1982, yes 1982(VIC20). Would like replies from MCP's in my area.

Fri, Aug 24, 2001 Michael Domingo, Web Edit Irvine, CA

Allan, we hope to get the International Salary Survey online soon. It was taken down because of all the bad data that was being entered (no validation or controls, for Pete's sake!).

Thu, Aug 9, 2001 Allan Follack Anonymous

Where is the International Salary Survey? I am Canadian and cannot find any info on this site for Canadian salaries.

Add Your Comment Now:

Your Name:(optional)
Your Email:(optional)
Your Location:(optional)
Please type the letters/numbers you see above

Redmond Tech Watch

Sign up for our newsletter.

I agree to this site's Privacy Policy.