Foley on Microsoft

Windows Vista Testing: The School of Hard Knocks

Microsoft's made some improvements with its Vista beta testing program, but it could be doing more.

• By Mary Jo Foley
• 04/01/2006

First, for the record, let's give Microsoft kudos where they're due.

Microsoft started laying the groundwork for the Vista CTP program more than a year before delivering the first Vista beta drop in July 2005. Microsoft's Core Operating Systems Division (COSD), which focuses on improving the processes behind Windows development, has been toiling to make Windows more modular. As a result of their work, it's proven easier for Microsoft to add (and cut) features and do so in a more controlled way throughout the Vista beta process. And because of that architectural involvement, the Windows team launched a CTP program much like the ones the company's developer division has been pioneering for more than a year.

But it hasn't been all roses in Windows-testing land.

Microsoft's Connect beta servers were completely overwhelmed by testers attempting to download the February Vista test build. Several testers told us they simply gave up after several unsuccessful attempts to get the bits. Unless Microsoft puts more Connect servers in place or releases the anticipated April "Consumer" CTP simultaneously on Connect and on the Microsoft Developer Network (MSDN), the situation is likely to repeat itself this month.

In February, once testers did manage to get the code, they discovered a problem that testers of Microsoft's tools CTPs had come to know all too well: Different betas and CTPs don't mix well. Often times, even simultaneously released CTPs of WinFX and Vista don't mix.

The February Vista CTP was incompatible with Office 2007 Beta 1. But many testers didn't discover this until after they had already tried getting the two to interoperate. (Microsoft was slated to fix this problem in March with a technical refresh of the Office 12 Beta 1 code.)

It's not just logistics that are at issue. While the Microsoft developer division is very clear in distinguishing between CTPs and betas, the Windows team has muddied the waters, confusing its testers, partners and customers.

You want a CTP? Here's our latest, the Windows team offered. You want a full-fledged beta? No problem: This CTP is a beta, too. Huh?

No one should forget that Microsoft officials themselves coined the CTP term. When Microsoft launched the first CTP builds of Visual Studio more than a year ago, execs were precise in noting that CTPs were not betas, and thus, not of beta quality. CTPs were interim test builds, snapshots. As such, they shouldn't be tested or deployed the same way full-fledged betas are, they said.

Now, as Vista enters the final development stretch, Microsoft is redefining its terms all over again. The December Vista partner CTP, the February enterprise CTP and an April consumer CTP aren't mere CTPs: All of them also are Beta 2 builds, in Microsoft's new taxonomy.

To some, these are mere semantics. But I say, let's call a beta a beta, and a CTP a CTP. Keeping track of Microsoft's disappearing and reappearing features is tough enough without the definitions of commonly accepted terms changing, too.

What do you say? Is the Windows Vista CTP program worth the growing pains? Can Microsoft smooth the CTP process for Longhorn Server, Vista R2, Vienna and other Windows follow-ons? Or would you be content to go back to the plain-old beta process, minus CTP builds? Write me at [email protected] and let me know what you think.