In-Depth

Tales from the Trenches: Critical State

Sometimes you need to troubleshoot the people before the technology.

While at a client site one day, an alpha page came in from a natural gas trading firm we support: “John, our NT server running our Oracle 8.0 database and supporting our trading floor is down. It’s stopped at a blue-screen, and on every reboot, it comes back to the same blue screen. It says something about an inaccessible boot device. Call me.”

I got on the phone immediately, knowing that for every hour down, the company was losing many thousands of dollars. After talking to the customer for a couple of minutes, I determined this wasn’t something we could fix over the phone. I immediately excused myself from the current job and headed over to the trading firm.

The server I was going to work on was a Hewlett-Packard NetServer with dual Pentium Pro processors and 256M of RAM. Additionally, it had four 4G SCSI hard drives, in a RAID 5 configuration, with a hardware RAID controller. A powerful advantage of a hardware RAID controller is the fact that it allows you to have a fault-tolerant system, but also lets you work with system partitions from Window NT’s setup routine. If you use software fault-tolerance, such as the mirroring that comes with NT, you can’t work with system partitions in the setup routine without breaking the fault tolerance.

The Sordid Details
The system partition was formatted with NTFS, so I knew we couldn’t boot from a DOS disk to examine the files on the hard disk. The server also had a 12/24G tape backup running Computer Associates’ ARCserve backup software, so I was hoping we had a good backup to restore with, should that be necessary.

The server’s state was critical when I arrived: it was at the “blue screen of death,” with a “STOP 0x0000007b” “Inaccessible Boot Device.” I decided to do a parallel install of NT Server, allowing me to examine the integrity of the start-up environment. Before beginning the parallel installation, I gathered the proper driver for the RAID controller. The moment the setup started, I began pressing the F6 key. This allowed me to install the RAID driver and have NT recognize the controller on restart. This avoided another instance of the “Inaccessible Boot Device” message. While inspecting the startup environment, I found the boot.ini file to be pointing to the wrong disk. A normal boot.ini file looks like this:

[boot loader]
timeout=10
default=multi(0)disk(0)rdisk(0)partition(1)\WINNT
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINNT=
"Windows NT Workstation Version 4.00"
multi(0)disk(0)rdisk(0)partition(1)\WINNT=
"Windows NT Workstation Version 4.00
[VGA mode]" /basevideo /sos

Ours was like this:

[boot loader]
timeout=10
default=multi(0)disk(4)rdisk(0)partition(1)\WINNT
[operating systems]
multi(0)disk(4)rdisk(0)partition(1)\WINNT=
"Windows NT Workstation Version 4.00"
multi(0)disk(4)rdisk(0)partition(1)\WINNT=
"Windows NT Workstation Version 4.00
[VGA mode]" /basevideo /sos

Note the disk callout difference in the third line? I quickly changed the boot.ini to correct the problem in the disk callouts and restarted the server. The server immediately rebooted into the existing operating system! We celebrated briefly and continued into the OS. We then we had some services fail and saw several shared folders that no longer existed on the server. It turned out the customer had tried an emergency repair on the server, restored a registry from an old emergency repair disk, and didn’t tell me!

Our only hope at this point was to extract an updated catalog from the backup tape and restore our server, including the registries. It turned out we had a good backup from the previous night.

The restore operations stated success, so we restarted the server and held our breath yet again. The server came back up! We rechecked the integrity of the restoration and boot.ini file. Everything looked good so far. We then tested the integrity of the Oracle database. The clients were able to attach to it successfully! We took the clients back off-line, installed the OS service packs, and reapplied the Y2K updates. Again, we had the server restart. Everything started successfully, and clients were able to attach to the database.

Time is of the Essence

Things could have gone differently. If my customer had had an up-to-date emergency repair disk or hadn’t restored a registry during an emergency repair, we could have repaired the boot.ini file and had the system up faster. Every time you make a change to your disks or partitions or upgrade the service pack on your server, you need to be sure to upgrade your emergency repair disk.

Additionally, if you do need to do an emergency repair of a server, you should only restore a registry from your emergency repair disk as a last resort. It turned out the problem with the boot.ini file had been created when the customer had moved some partitions around to better use the RAID for the Oracle server.

Next, be sure to communicate all of the details of the attempts to repair the system with all members of the team attacking the problem. Often some of the attempts to repair a downed server aren’t always directed at the right source of the problem. When I’m working with some of our junior network people, I have to constantly remind them, I need all of the data to formulate a plan.

Last, always have a good backup, and monitor and audit it regularly. Take it from me: There’s no worse feeling than not having a good backup as your last line of defense.

About the Author

John T. Kruizenga, MCSE, has worked with computers and networking since 1988. He has designed and managed networks that incorporate VoIP and QoS, remote management, WAN integration, collaborative software, and Web integration.

comments powered by Disqus

Reader Comments:

Sun, Jun 22, 2008 Sarpreet singh khunger System Engineer nettworking

(1)Manage and maintain physical and logical devices.(2)Configuring and troubleshooting the desktop envirement.Implementing,managing and maintaing,a microsoft windows server 2003 network infrastrutre.(3)Managing,implementing,and maintaining,microsoft windows server tm 2003 Active directory infrastructure.(4)Implementing,managing,maintaing and troubleshooting,hardware devices and divers.

Wed, Nov 2, 2005 Anonymous Anonymous

good

Thu, Jun 16, 2005 steve ca

""While inspecting the startup environment, I found the boot.ini file to be pointing to the wrong disk.""

HOW did you do this?Please e-mail me.

Wed, May 4, 2005 arvind kumar rayeekanti hyderabad

i am basically from electrical background and i want to switch over to system side for MCSE or networking CCNA so i request to kindly e-mail about the entire details of MCSE and CCNA courses consists off. and also i am very much eager to know the details plzzz help me out

Fri, Dec 10, 2004 sajid india

am an mcse, mcdba and ccna. am looki ng for a great career at middle east.. may i get deatailed information about the market situation in gulf countries.

Wed, Nov 10, 2004 Anonymous Anonymous

great article

Sun, Dec 21, 2003 eroz india

iam eroz from india my question is regarding mcse that after having this certification how much i can get job opportunities .
thank you

Sun, Dec 21, 2003 Anonymous Anonymous

my name is eroz and my query is regarding mcse .iam doing job in the marketing field and iam very keen to go abroad to usa or canada.at present iam living in india.i dont have any professional degree for your information iam graduate(b.sc.) my question to u is that how this mcsc certification could help me to secure my future.my e-mail id is bawa7_rajput@rediffmail.com
waiting for ur reply
thank you

Sun, Aug 3, 2003 Sid UK, London

im interested in completing the msce course. i hope, however, to complete in pakistan (iv heard its MUCH cheaper!). can sum1 plz help!! email me. thanx

Tue, Jul 1, 2003 suhel canada

i am mcse and I want job.

Mon, May 26, 2003 Iftikhar Ali Pakistan

I Am a recent MCSE and MCDBA and Have Two to Three Years of Experience.
I am from South East Asia but South East Asia is not mantioned here and specially the salaries of network tec. and relevent to network are too much low.
Most of the MCSE and MCDBA are jobless or working at very low salary.
I think its very sad for microsoft and the IT industry.

Mon, May 26, 2003 Iftikhar Ali Pakistan

I Am a recent MCSE and MCDBA and Have Two to Three Years of Experience.
I am from Far east but far east is not mantioned here and specially the salaries of network tec. and relevent to network are too much low.
Most of the MCSE and MCDBA are jobless or working at very low salary.
I think its very sad for microsoft and the IT industry.

Sun, May 18, 2003 Deepak Jain India

i have done my MSCE in 2000.

Mon, May 20, 2002 Shimal Thakrar London, UK

This is a very good forum to share experiences and knowleadge of true professionals. I recently got my MCSE on the W2K route, glad to be on board. I'm looking to move to the USA, if anyone knows of any positions, please mail me.

Mon, Mar 11, 2002 Manoj Kuwait

MCSE and all other certifications doesn't acually showcase an individual's IT talent...the only benefit of this certifications I can find is that they only gives you a chance to put your foot throu' for any job vacancy...thats it.

Mon, Mar 4, 2002 Albert Ling Malaysia

I am living in Malaysia, I would like to ask if there is any survery data on Malaysia or Asia. Does Sector have any influence in the salary range ? I am currently working In Health Care Sector ( hospital)

Wed, Jan 23, 2002 Ron Middle East

Jobs are available in the middle east for americans and foreingers. Search for Kuwait, Doha Qatar, Saudi Arabia etc. My first job here pays me in the mid 60s. I have MCSE2k, MCSE4.0, CCNA, MCP+I, Wideband GHZ, A+ and an associate in Microsoft Technology.

Some advice: While attaining your MCSE do any type of workstudy program. For veterans check with the VA hospital Lan\Wan shop. Take on volunteer job assignment offered by the college, start your own business whether it's web design using frontpage for a friend or fixing your moms computer. Go to ebay or some other place and buy workstations and slowly build your own network. Better yet do this with a group of friends and each bring their own PC with a hub. I haven't worked on my site since I left the states but you can check it out at gotfame.com

Mon, Jan 14, 2002 Ptochos Minneapolis

I'm also from Minneapolis. This area is very saturated with techs. I've been to several recruters, and they don't offer much - Even with W2k MCSE, A+, and over 14 years computer experience. This site would do justice to it's facts and figures if it also gave some pointers as to "where" to get these fantastic salarys in the Twin Cities.

Fri, Jan 4, 2002 John USA

12/25/01: fuji from Minneaplos says: "These numbers look too high. especially with all these yahoos out there trying to get job and under bidding everyone else. driving the average salary down."

Couldn't have been better said!!!

Tue, Dec 25, 2001 fuji Minneaplos

These numbers look too high. Espcially with all these yahoos out there trying to get job and under bidding everyone else. Driving the average salary down. I am a resent W2K MCSE + 2-3 years experiance. I am a Network Tech but that does not describe my job at all. Jobs are slim and way low salary compared to job description + Yrs of experiance.

Wed, Dec 19, 2001 Adam USA

Mike, do yourself a favor. Go to college, get a job with the university using your certs., get the experience while you get your degree. You'll sit prettier in the end.

Sat, Nov 24, 2001 im iftikhar anwar pakistan

im mcse
and like this feild very much
and finding job

Thu, Nov 1, 2001 mike canada

im from canada just about to finish high school and doing my mcse certification, net + and cisco(cicp). with all that in hand
what is the average i could make without a college degree

Thu, Oct 11, 2001 Giancarlo Italy

I agree with Leon from UK, I'm looking for numbers of Certified people world wide, even if I only would like to know how many MCPs, MCSEs, etc.. are in Italy.
Let us know please ....

Tue, Oct 9, 2001 Renato Brazil

I´m from Brazil, and here, certifications doesn´t have strong impact over salary. Only for MS solution providers (that need MCP´s)

Wed, Oct 3, 2001 Leon P UK

Does anyone really know exactly how many MCSE's and MCDBA's there are world wide ?

Wed, Oct 3, 2001 Jade Milwaukee

This is a very good site. It gives alot of information to most of your questions and has alot of charts and regional information to let you know what your certificate is really worth and how you can go about making more money.

Fri, Sep 7, 2001 Lars Sweden

I live in Sweden and even if I have MCSE on NT 4, I still havent got a job cause the bad jobsituation here. Every company wants experienced ppl and without experience the certificate is useless. I can just congratulate the ones who got over $ 40 :)

Tue, Aug 7, 2001 Andrew Craib London

Are there any international (UK) salary comparisons availale?

Thanks

Add Your Comment Now:

Your Name:(optional)
Your Email:(optional)
Your Location:(optional)
Comment:
Please type the letters/numbers you see above

Redmond Tech Watch

Sign up for our newsletter.

I agree to this site's Privacy Policy.