Never Again

I've Put My Finger on the Problem...

Powering down the wrong server makes for a sticky situation, plus another admin shares his virus horror story.

• By Chris Grant and Adrian Dickreiter
• 06/01/2006

There was nothing abnormal about doing this, except for one fateful day. This day someone was working on a server, went back to the rack, pressed the power button and immediately realized -- while his finger was still holding in the button -- that he was pressing the power button on the wrong server.

He intended to cycle a MetaFrame server with no users; instead, the Compaq Proliant 7000 file server that held user profiles and home directories for about 800 users was hanging in the balance by one person's finger. He also realized that as long as he was holding in the power button, the system wouldn't come crashing down.

The design of the power button saved him, in more ways than one. While standing there holding the button, he was calling through the data center to have someone come to his aid.

A fellow server sys admin heard his cries for help over the machine noise; they then started brainstorming to resolve this critical problem. They determined that they could install the Compaq utilities on the system and then disable the power button entirely. After this was done, he finally let go. The server -- and the 800 users -- continued to work and no one ever knew the difference. Fortunately (or maybe unfortunately), this story didn't make it to management until several years later.

After that, we began stressing the importance of a good label maker for the server team! -- C.G.

Education in Viruses
The Blaster and Welchia viruses hit our community college campus in August 2003. Naturally, about this time, campus registration was in full swing. Quite mysteriously, the network started to slow down. Network segments began to drop off altogether. Failures in a network as large as this were common, but the situation quickly became worse, and within hours the entire campus was crippled. Just before we completely lost Internet access we downloaded the latest Symantec virus definitions and ran them on a few workstations to see if, by chance, a virus had infected our network. We found minimal traces of the Blaster worm and a massive infection of Welchia.

Suddenly, dial-up and DSL connections on campus became priceless. We quickly got into the hands of every technical support staff member on campus printouts detailing the virus removal instructions, a procedure list and CDs including the latest removal tools, the Symantec virus definitions and the Microsoft patches necessary to prevent a re-infection. It took the support staff about three days to clean andpatch the majority of the workstations on campus.

All the while registration was next to impossible to carry out, and the functions of all faculty, staff and administration were hampered. Total cost to the college for technical support personnel was estimated to be several thousand dollars. This excluded the lost time for the entire workforce of the college, and lost tuition and fees from students who gave up trying to register for classes.

In the end, while no one enjoys performing updates on computers, it had to be done. The administration didn't immediately jump at purchasing a patch management solution, but fortunately, Microsoft's SUS server was a viable option. I no longer work at the college, but at my new job, it's been my responsibility to oversee patch management and Symantec updates, both of which are faithfully kept up-to-date. -- A.D.