In-Depth
Fear and Loathing in Las Vegas
In a city of high stakes none were higher than during this tale of database corruption. But with tenacity and resourcefulness (and a little luck), this MCSE came out a big winner.
As I've spoken to other
engineers and DBAs, I'm sure I've said many a time that the keys to longevity
in the turbulent landscape that is Information Technology are tenacity and
resourcefulness. Of course, as I found out, the realities of IT can put
those virtues to the test.
Trudging
to an airport restroom to change out of gaping, split-seamed khakis that
had been ripped while climbing out of a luggage-laden airport taxi, I’m
sure I didn’t appear to be someone on his way to Las Vegas to compete
for a prestigious Industry Solution Award from Microsoft. I’m also sure
that those in the know would have had to stifle a snicker to find out
I had missed the flight by 20 minutes. I most likely appeared to have
about as much resourcefulness and tenacity as a Styrofoam cooler. But
after the sluggish start to this fateful journey, the silicon gods must
have been smiling on me, as I did manage to get myself and my traveling
companion to Las Vegas, albeit after 12 cumulative hours on stand-by as
penance for missing the flight. The companion, by the way, was the president
of the company.
Special Delivery
The server I had lovingly built up during the past several weeks was safely
nestled like a football under my left arm for most of the trip. Its immediate
purpose, before it would be converted to a VPN or fax server, was to show
off the company’s SQL Server-based application to a panel of judges and
several hundred Microsoft Healthcare Users Group (MSHUG) conference attendees.
My immediate purpose was to make sure it made the trip unscathed because
when it was finished “wow”-ing them in Vegas, it was to fly on to Texas,
where it would be used to demonstrate the same application to a large
group of executives in a large company who were interested in making
a large software purchase. Other than server bodyguard, my job
was a simple one: Point on the right icon at the right time and press
the page-down button when queued by “El Presidente.”
We arrived
at our booth early. Thankfully, all of the peripheral equipment we’d rented
was waiting for us. I immediately set up and confirmed that the server
had sustained the jostling and jouncing of being bumped across America.
All that remained was to fight back the nervous energy and jitters that
had been steadily growing as I saw just how enormous this show had turned
out to be.
Appropriately,
Las Vegas has every known cure for jitters. The next two days, as we waited
for our turn to present to the judges, were a blur of taxi rides, neon
lights and complimentary beverages. I also had the task of finding a special
slot machine to play my wife’s $20, which she had given me along with
a hopeful, “Good luck!”
Without a Hitch
Finally, our time came, and I dutifully displayed my rehearsed point-and-click
savvy to the eager, note-jotting judges. After the 15-minute presentation,
I was, simultaneously, overcome by relief and dry mouth. After a congratulatory
pat on the back from my boss, I headed for the nearest soda machine.
My mind was
reeling from the questions we’d been hit with by the judges. One question,
in particular, was, “Can your application be configured to launch other
applications?” Well, we had designed our menu structure to be able to
embed other applications, like Microsoft Excel, as links directly into
our database. But we’d only tried it a few times and had refrained from
doing a live demo of this ability. The judges seemed content with that.
OK, What Did You Do?
When I returned from my break, I saw my boss feverishly clicking on
myriad SQL errors popping up on the screen. He didn’t offer an explanation
or even acknowledge my presence. He just kept trying in vain to get back
into the program. I knew not to ask. Eventually
he relinquished the mouse with a statement just above the audible range:
“Can you
fix it?” That’s when instinct took over like an ER doctor pulling a trauma
patient back from the edge.
“OK. What
did you do?” I asked as I moved quickly into position.
“I tried
to embed a link to Excel in the database,” was his only reply.
I starting
digging and within three minutes had the ugly truth: The database was
corrupt. The 200MB database was corrupt. The 200MB database—with no local
disk dump and no connection to the office to get the backup—was corrupt.
I had to inform the next of kin, and he didn’t like it one bit. The large
demo in Texas was scheduled for the next afternoon. As we were consoling
one another, the announcement came over the loud speakers, “The show floor
will be closing in 15 minutes. Winners for the Industry Solution Awards
will be announced tomorrow morning. Thank you, everyone, for attending.”
Translation:
“Scram! And make sure you leave our equipment behind!”
“At least
we made it through the presentation,” I chided. The man who had signed
my paycheck for the past two years said, with a look, “Get to work. You
have 15 minutes.”
Is Anybody Home?
At this point, resourcefulness kicked in. Tenacity would show up much
later in the evening. I tried everything I could think of. My tier-1 tech
support background instructed me to bounce the server — no good. Database
Consistency Checks (DBCC) revealed errors of the irreparable variety.
I saw the floor workers stripping keyboards and monitors away from other
booths. Our time was up. I turned to my boss and said in a contrived,
yet somehow believable, tone, “Don’t worry, I’ll take care of it.”
It was about
5 p.m. in Las Vegas; therefore, it was 8 p.m. back in our Pensacola, Florida
office. Surely someone would still be there at 8 p.m. — that was the fortunate
side of working at a company of workaholics. Sure enough, a programmer
was still there. I hurriedly explained my dilemma. All I needed was for
her to take the backup copy of our database, compress it to an optimistically
small file and put it in an accessible location. She said she’d have it
posted in half an hour.
Intergalactic Inspiration
I had a relatively powerful machine at my disposal — but, without
a keyboard and mouse at the very least, it was useless; a modem would
be required as well. Armed with the company credit card and carte blanche
to get the job done, I set out to piece together the missing components.
On the way through the hotel, I ran into a few meandering Klingons. You
see, the Hilton, where we were staying, was also the home of “Star Trek:
The Experience.” I couldn’t avoid stopping in for a short visit at Quark’s
Bar — you know, just to say I had been there. From my vantage point in
the bar, I spotted an even more interesting site: a 24/7 copy shop right
across the way. I was instantly relieved to think I might not have to
carry a computer and, now, a brand-new monitor back home with me after
all. I raised a blue-frothy, cosmic toast to my fellow patrons and teleported
over to the copy shop.
The first
thing I noticed was the external 28.8Kbps modem attached to one of the
rental workstations. After much pleading, which soon to turned to groveling,
I convinced the patron using the system to hurry along. I had also, using
similar tactics, convinced the copy shop clerk to let me disassemble her
workstation and use the parts for the next three hours. (The gold corporate
credit card may have helped.)
I brought
my machine down and packed into a very small alcove where I had access
to everything I needed, including a phone line. I dialed into our RAS
server at work and easily found the file. It had compressed down to 15MB.
I started the download from the command prompt where I could use “xcopy”
with the /Z option. If I were to get cut off, which I assumed I would
at least once, I could re-establish the connection, start “xcopy” again
and pick up where it had been cut off. I asked the clerk to watch the
machine and I went off to play my wife’s $20.
Hitting it Big
I knew I’d be there for a while, so I took my time, trying to find
the perfect machine upon which to play the $20. Quarter by quarter, I
had played down to only three, so I figured I’d move to the dollar machines.
If I was going to lose, I might as well lose a little faster. I dropped
all $3 in, pulled the handle and prepared to walk away. To my utter astonishment,
I hit triple sevens. Twenty minutes and $275 later I was back at my computer
with only 300KB remaining on the download. When it was finished, I uncompressed
the file, restored the database, and was able to get back into our program.
Life was
good.
I didn’t
tell my boss until the next morning that I had everything back together.
We met at the awards ceremony, and I’m sure I looked quite haggard. When
it came time to present the award for our category, I felt the butterflies
returning. I knew that I wouldn’t be the one to physically accept the
award if we won, but I was nervous just the same. We did win, for the
first time after several years of coming very close, and the trip to Texas
went off without a hitch. But the best part of the adventure wasn’t winning
the award, or getting a large client, or even scoring big at the slots.
It was leaving Vegas with the realization that — in the face of seemingly
overwhelming adversity — resourcefulness and tenacity (and, perhaps a
little intergalactic help) can bring you through.
About the Author
Rodney Landrum is an MCSE working as a data analyst and systems engineer for a software development company in Pensacola, Florida. He has a new book from Apress entitled ProSQL Server Reporting Services.