Foley on Microsoft

Windows Vista Testing: The School of Hard Knocks

Microsoft's made some improvements with its Vista beta testing program, but it could be doing more.

Some time this month, if Microsoft sticks to schedule, the company will deliver its latest test build of Windows Vista to hundreds of thousands of testers. Seems like a perfect time to contemplate how well Microsoft's Vista beta/Community Technology Preview (CTP) program is going.

First, for the record, let's give Microsoft kudos where they're due.

Microsoft started laying the groundwork for the Vista CTP program more than a year before delivering the first Vista beta drop in July 2005. Microsoft's Core Operating Systems Division (COSD), which focuses on improving the processes behind Windows development, has been toiling to make Windows more modular. As a result of their work, it's proven easier for Microsoft to add (and cut) features and do so in a more controlled way throughout the Vista beta process. And because of that architectural involvement, the Windows team launched a CTP program much like the ones the company's developer division has been pioneering for more than a year.

But it hasn't been all roses in Windows-testing land.

Microsoft's Connect beta servers were completely overwhelmed by testers attempting to download the February Vista test build. Several testers told us they simply gave up after several unsuccessful attempts to get the bits. Unless Microsoft puts more Connect servers in place or releases the anticipated April "Consumer" CTP simultaneously on Connect and on the Microsoft Developer Network (MSDN), the situation is likely to repeat itself this month.

In February, once testers did manage to get the code, they discovered a problem that testers of Microsoft's tools CTPs had come to know all too well: Different betas and CTPs don't mix well. Often times, even simultaneously released CTPs of WinFX and Vista don't mix.

The February Vista CTP was incompatible with Office 2007 Beta 1. But many testers didn't discover this until after they had already tried getting the two to interoperate. (Microsoft was slated to fix this problem in March with a technical refresh of the Office 12 Beta 1 code.)

It's not just logistics that are at issue. While the Microsoft developer division is very clear in distinguishing between CTPs and betas, the Windows team has muddied the waters, confusing its testers, partners and customers.

You want a CTP? Here's our latest, the Windows team offered. You want a full-fledged beta? No problem: This CTP is a beta, too. Huh?

No one should forget that Microsoft officials themselves coined the CTP term. When Microsoft launched the first CTP builds of Visual Studio more than a year ago, execs were precise in noting that CTPs were not betas, and thus, not of beta quality. CTPs were interim test builds, snapshots. As such, they shouldn't be tested or deployed the same way full-fledged betas are, they said.

Now, as Vista enters the final development stretch, Microsoft is redefining its terms all over again. The December Vista partner CTP, the February enterprise CTP and an April consumer CTP aren't mere CTPs: All of them also are Beta 2 builds, in Microsoft's new taxonomy.

To some, these are mere semantics. But I say, let's call a beta a beta, and a CTP a CTP. Keeping track of Microsoft's disappearing and reappearing features is tough enough without the definitions of commonly accepted terms changing, too.

What do you say? Is the Windows Vista CTP program worth the growing pains? Can Microsoft smooth the CTP process for Longhorn Server, Vista R2, Vienna and other Windows follow-ons? Or would you be content to go back to the plain-old beta process, minus CTP builds? Write me at mjfoley@redmondmag.com and let me know what you think.

About the Author

Mary Jo Foley is editor of the ZDNet "All About Microsoft" blog and has been covering Microsoft for about two decades. She has a new book out, Microsoft 2.0 (John Wiley & Sons, May 2008), about what's next for Microsoft in the post-Gates era.

comments powered by Disqus

Reader Comments:

Tue, Apr 4, 2006 Matt Philmon Charlotte, NC

Microsoft has made a mess of the process, but I still value the CTP process and have found, as a developer, that it's been very helpful in preparing for Vista. They need to clean up the nomenclature and get the servers needed to support the process, but I'd say overall it's been very successful.

Tue, Apr 4, 2006 beBoy Toronto

I would agree the 4-4-06 comments "these are all incremental builds on the way to Beta2" "At no time has MS called any build Beta 2". Please confirm this at Vista beta "cafe newsgroup", it has been discussed before.

No I don't think the CTP sounds confusing, those big CTP releases shouldn't have the Beta tag as yet. To testers (including me), we know the quality for each CTP release that MS claims to have, and we treat those big ones (Dec CTP, Feb CTP) as milestones of CTP progress to Beta, so internally it should not be confusing.

To public (MSDN/Technet subscribers), I think the team has not stresses enough the importance and the role for each CTP, which in turns will lead to confusion of people thinking those CTPs are just another CTP quality build.

It also should not be confused that all milestones build are qualified as CTP builds, as the supposed-to-be Nov CTP was indeed a milestone, but too bad it was too buggy.

Tue, Apr 4, 2006 Anonymous Anonymous

At NO TIME has Microsoft called ANY build Beta 2, your information is incorrect. The build tag in the lower right of the screen says that, but these are all incremental builds on the way to Beta 2.

Mon, Apr 3, 2006 techSage Anonymous

Microsoft already has added servers to increase the capacity of Connect. Also, having beta tested MS software since Windows 95 and MSN v1 Betas, I think this CTP process is a huge improvement of the process and is better for everyone involved.

Add Your Comment Now:

Your Name:(optional)
Your Email:(optional)
Your Location:(optional)
Comment:
Please type the letters/numbers you see above

Redmond Tech Watch

Sign up for our newsletter.

I agree to this site's Privacy Policy.