In-Depth

What Were They Thinking?

An HP hardware troubleshooting guru shares some of the worst snafus he's experienced so you can learn from them.

When it comes to configuring systems and networks, there's always more than one way to skin a cat. In fact, IT managers face so many options that it's easy to get in trouble.

Because we usually learn more from what someone did wrong, rather than what they did right, I'll share a couple of the more memorable encounters I've had with customers in my years as an HP hardware troubleshooter.

What's in a Name?
Back in the mid-1990s, an international customer was having all sorts of network issues. After initial discussions failed to identify the obvious reasons for network resources intermittently dropping offline, I flew out to their office on the Gulf Coast for a closer look.

On arrival, I discovered that network resources local to the company's U.S. facilities were fairly stable, and the same was true for the European offices. But communication between the U.S. and European offices was spotty. The configuration included a mixture of Windows NT 3.51 and 4.0, with Windows-based clients using the Microsoft implementation of TCP/IP. The company clearly needed consistent name resolution between the United States and Europe to resolve the issue.

As I began my investigation, I discovered some legacy-name resolutions, like network-based and local LMHOSTS files, remained in place. It looked like the IT department hadn't succeeded in fully purging the network when they implemented WINS resolution for their enterprise. So I went to work cleansing the rest of the network-based LMHOSTS files. This helped, but only marginally. Communications between network resources on each continent remained intermittent.

It was time to dig deeper. I ruled out a physical disconnect by pinging remote servers by IP addresses. Even when U.S.-based clients couldn't access the servers by name, the servers remained accessible by IP address.

I discovered if I created an LMHOSTS file or used the IP address of the server along with the share, I could access network resources. That led me to investigate the WINS infrastructure. I soon found that the push-pull replication was fine stateside, but that the servers were replicating their information to a U.K. server I couldn't access. I also wanted to confirm their configuration and determine if their WINS hierarchy replicated to the same server.

The European configuration was similar to the one in the United States. I found that these servers, too, replicated to the same master WINS server in the U.K. IT office, with the same target IP address. But the staff didn't have access to the target server. In fact, they didn't even know it existed.

Finally, the U.K. IT team performed a network trace and located the mystery WINS server and its owner. It wasn't in the server room. Instead, the mystery box turned up in a random lab, independent of the IT organization. When the IT staff went to check on hardware, they found a couple printers, an old Compaq DeskPro 386/20 running Windows NT 3.51 server and a few other miscellaneous items.

The obvious question comes to mind: Why would an international company, with a global infrastructure, put the linchpin of its naming resolution on a desktop (and an old one at that!), rather than a server? The answer: By accident.

As it turns out, the server that should have been the WINS master had an IP address very similar to the DeskPro desktop. When two numbers in the address assignment were accidentally transposed, the error got copied by everyone configuring other WINS servers. The result: A snowball effect that directed traffic to an unassuming desktop.

The finding also explained the issue of sporadic name resolution, as a system in the lab would be rebooted at will. The "owner" of the DeskPro had no idea that his test system had unintentionally been utilized as the master WINS server for the entire organization.

I still wonder how something like that could happen.

Snafu

I was an IT manager with a major trucking firm, where we used SNA Server to let Microsoft clients talk to an AS/400 server. Every now and then somebody's connection would lock up and our procedure was to clear the hung connection so the user could establish a fresh one.

We started having a problem where all the AS/400 connections would fail simultaneously. It was an intermittent thing and we couldn't pin down the cause. Checking the server showed no connections and, more importantly, no error codes. A sweep for viruses turned up nothing. We were stumped!

After yet another failure, I asked the department administrative assistant, who in the past used to handle some IT functions, "Did you do anything with the SNA services?" And she said, "Sure. I cleared out the table."

There was that dawning, that moment, when I realized: Oh my god, she's been clearing it out every day. Whenever she got a call that a connection was hung, she thought her job was to just go into the table and clear everybody out. She had been doing this for weeks.

You should have seen my CIO. He just looked at me, shook his head, pointed at the door and said, "Get out."

Clap On! Clap Off!
Here's a classic from a peer of mine. He got a call from a customer, a small business with three servers, about a hard-to-pin-down power issue at their facility. It seemed one of the servers would intermittently lose power. The initial phone triage-such as plugging power cords into different outlets-didn't work, so an on-site visit was in order.

Suspecting a bad power supply, the consultant entered the server room. The door was wood, with a metal frame -- the kind that rattles when you close it with a little attitude. In the server room, my counterpart saw the server was plugged into a "Clapper" device, which in turn was plugged into the wall.

That's right, the very same Clapper that turns lights on and off when you clap your hands. As it turns out, when you closed the door with enough force, the door rattled and triggered the Clapper, which then cut off power to the server.

Be Nice
We've all heard the stories. There's the guy who kept floppy disks pinned to a filing cabinet with a magnet; or the woman who plugged a surge protector into itself and wondered why the system wouldn't power up. These are the tales that try the patience of any IT professional.

Still, whatever the situation, I encourage you to take the high ground. If you pull off a miracle recovery, be humble. If you encounter colossal acts of stupidity, be kind when explaining the problem. Think back to your first days with computers and you might understand why some people make mistakes that now make you cringe.

Take me, for instance -- on my first day in my first professional job, I sat at my desk for 10 minutes trying to figure out how to turn on my computer. There's hope for all of us.

About the Author

Tom Pruett has been with HP since 1997. During his career, he’s been an IT consultant (focused on Microsoft solutions), network architect and worked on a help desk. He currently resides in Huntsville, Ala., with his wife of 14 years and their four children.

comments powered by Disqus

Reader Comments:

Mon, Apr 21, 2008 Terry Austin

I always wondered what happened to Mr. Pruett. Now I know he's still putting out fires, and having a good laugh while he's at it.

Tom, you should tell the story about the nice lady who told you that you "didn't know Excel very well" that's a funny one!

Thu, Jul 20, 2006 Cliff Raleigh, NC

Thanks for the article, Tom. The next time I see someone at the copying machine making a copy of a CD to distribute their friends, I'll also remember those stellar bombshells' from the industry, such as the fan exhaust on the bottom of the laptop, the close-quarters printer with the single feed mechanism in the back, or my favorite; the PC standard that placed the USB ports within an inch from the bottom of the cabinet. Thanks for reminding me why they should put 1/2 inch pencils on 7- inch erasers!

PS...Is their a tool to remove a jammed CD or DVD from the drive, or are we still using a straightened paper clip?

Add Your Comment Now:

Your Name:(optional)
Your Email:(optional)
Your Location:(optional)
Comment:
Please type the letters/numbers you see above

Redmond Tech Watch

Sign up for our newsletter.

I agree to this site's Privacy Policy.