Your Testing Environment
Developers have a challenging task when it comes to setting up software to match client requirements. Here's how to stay sane with the shifting landscape of beta and release software.
Independent software developers have a challenging task when it comes
to setting up software to match client requirements. In this column, I'll
give you some tips on how to stay sane while dealing with the shifting
landscape of beta and release software and how best to manage your own
mini-network for productive development and testing.
Here's the basic problem: If you're developing software for a wide customer
base (which might be for half a dozen consulting clients or a commercial
product that you sell to all comers), sooner or later you're going to
get a call—"It doesn't work!" At that point, you need to play detective.
In many cases, you'll end up determining that the software seems to have
been installed correctly, but that the user is running it on different
hardware or software than you use for development. Perhaps they're your
first user with Windows ME, or they've downloaded a new beta version of
Internet Explorer (which might well break your XML code, if you depend
on the MSXML parser), or they have four IDE hard drives in their machine,
or they're using the Hebrew version of Windows, or…the possibilities are
It's Called Hardware for Good Reason
Hardware problems can be the most intractable ones to deal with, simply
because of the difficulty of duplicating the hardware environment. True
story: I once worked on a subcontract for a very large software corporation
developing some software that was being tested internationally. The Japanese
subsidiary reported that the program mysteriously bombed on most of their
computers. After a lot of back-and-forth e-mail, we figured out that it
was just trying to open a temporary file when it locked up. After a lot
more e-mail, we finally managed to find a computer in our area that was
a duplicate of the ones they were using. That was when I learned that
NEC had shipped a bunch of computers where the hard drive was drive A.
And our code was trying to save a temporary file to the root of the C
drive! OK, boneheaded move (the correct answer, of course, is to ask Windows
where the temporary directory lives), but half a dozen developers couldn't
find the right answer without the hardware, no matter how long we stared
at the source code.
One brutal fact of this business is that (unless you're working for a
large corporation with a very large test lab) you will never be able to
buy hardware to duplicate all of the cockamamie configurations that your
customers are using. Despite this, your first line of defense in coping
with software testing should be to build up your own network of computers.
I recommend at least three test machines in addition to your development
machine, all networked together. That should give you enough resiliency
to deal with multiple problems, as well as provide backup for the inevitable
hardware problems. Fortunately, you don't need to spend a whole bundle
of money on this. First, even though you need the latest-and-greatest
whiz-bang hardware for your own desk, your customers are likely using
older computers that you can match without excessive spending. Second,
you don't want three identical brand-new computers. I recommend buying
three different used computers, just to triple your chance of finding
odd hardware interactions. If you poke around on eBay, you'll find Pentium-class
computers that are probably as good as those used by the majority of your
customers starting at a few hundred dollars. Throw in a keyboard-mouse-video
(KVM) switch, some cables, and a hub, and you can set up the nucleus of
your test lab for a thousand dollars or so.
I strongly suggest that you not do your testing on your main development
box. Why not? Because sooner or later you're going to have to install
some beta software or obscure driver on the test machine. The best way
to avoid having junk pile up to the point where you waste a couple of
days reinstalling Windows and all of your development applications is
to never install beta software on your development box. In fact, don't
install anything that's not necessary to your productivity on that box.
Keep a second computer by your desk if you can to hold betas and games
and the latest unstable peer-to-peer Internet application—and be
prepared to format that hard drive once every six months or so, when it
starts crashing on a regular basis.
Of course, if you're like me, you'll never throw a computer away, and
eventually end up with a dozen test boxes. This is a good thing, so long
as you can convince your significant other that you really need all those
Software Can Get Costly
After you've got hardware to work with, you need to start worrying about
software. Here's a short list of the questions that I usually ask to try
to track down pertinent software information:
- Did the application ever work? Did it just stop working? If so, what
was the most recent thing you installed?
- Which operating system are you using? Which service packs? Which hotfixes?
- What version of Internet Explorer is installed? Any other web browser?
- What version of Office is installed? Visual Basic? Any service packs?
Take my word for it: You do not want to buy every bit of software you'll
need piecemeal. Even apart from the nuisance value, you'll spend way too
much money, particularly if you need versions in other languages (and
you will need them if you sell any copies to other countries; it's not
unusual for a bug to only surface on a particular version, say French
Windows 98). I highly recommend that you sign up for a subscription to
the Microsoft Developer Network, if you're not already a member. Subscriptions
come in a variety of levels and prices, depending on which software you
need; you'll find the details at http://msdn.microsoft.com/subscriptions/prodinfo/levels.asp.
Personally, I've never regretted spending the money for MSDN Universal
By the way, it's not too tough to debug on French Windows (or German,
or even Chinese) even if you don't speak French, because all of the menu
items remain in the same order.
Testing Environment Shortcuts
But getting all the software you need is only half the battle. You can
waste a lot of time installing and reinstalling operating systems. Two
types of tools can help you minimize this waste of time: disk imaging
and virtual machines.
With disk imaging products, you can take a pristine copy of an operating
system and save it to a CD (or tape or a file). Later, when you need that
operating system again, it's much easier to restore the CD than it is
to go through the whole reinstall. If you go this route, after a while
you'll build up a set of CDs customized to your own hardware and the way
you like to have your Windows set up, and be able to quickly bring up
a test box with the desired configuration. Some products in this category
include Norton Ghost (http://www.symantec.com/sabu/ghost/ghost_personal/)
and PowerQuest Drive Image 5 (http://www.powerquest.com/driveimage/).
Useful though disk imaging is for building up a catalog of configurations,
it's not the best answer for managing multiple operating systems. The
idea of a virtual machine is to run one operating system inside of a window
on another operating system. For years people said that the Intel x86
architecture didn't support virtual machines well enough for this to be
a workable idea, and then along came VMware (http://www.vmware.com).
VMware itself installs as an application on your Windows or Linux PC.
Then you open the VMWare window, push the virtual power switch—and
a second PC boots up! Plug in a CD-ROM, and it will install a new operating
system inside the window. VMware offers many other features, including
solid networking from inside the virtual machine, save to disk, and an
"undo" facility that can remove changes made by errant software. I can't
recommend it highly enough; right now on one of my boxes I have a Windows
XP virtual machine and a FreeBSD virtual machine both running in windows
on top of a Windows 2000 Server installation. The only drawback is that
you need a lot of RAM for this to work well; I'd recommend at least 512MB
if you're going to make heavy use of this strategy for testing and debugging.
Finally, if you do end up spending time duplicating a customer's environment
to track down some obscure problem: take notes. You may want to look at
a commercial bug-tracking system; there are dozens to choose from. Or
you may want to simply keep a Word document or Access database of your
own to store this information. But whatever you do, don't depend on memory
to tell you what the fix was for running your software on Bulgarian Windows!
I hope you find some of that advice helpful—I know I wish someone
had told me about disk imaging long before I found out about it myself.
Did I miss a trick you find essential for development and debugging? Write
me at email@example.com
to let me know!