Foley on Microsoft

Windows Vista Testing: The School of Hard Knocks

Microsoft's made some improvements with its Vista beta testing program, but it could be doing more.

Some time this month, if Microsoft sticks to schedule, the company will deliver its latest test build of Windows Vista to hundreds of thousands of testers. Seems like a perfect time to contemplate how well Microsoft's Vista beta/Community Technology Preview (CTP) program is going.

First, for the record, let's give Microsoft kudos where they're due.

Microsoft started laying the groundwork for the Vista CTP program more than a year before delivering the first Vista beta drop in July 2005. Microsoft's Core Operating Systems Division (COSD), which focuses on improving the processes behind Windows development, has been toiling to make Windows more modular. As a result of their work, it's proven easier for Microsoft to add (and cut) features and do so in a more controlled way throughout the Vista beta process. And because of that architectural involvement, the Windows team launched a CTP program much like the ones the company's developer division has been pioneering for more than a year.

But it hasn't been all roses in Windows-testing land.

Microsoft's Connect beta servers were completely overwhelmed by testers attempting to download the February Vista test build. Several testers told us they simply gave up after several unsuccessful attempts to get the bits. Unless Microsoft puts more Connect servers in place or releases the anticipated April "Consumer" CTP simultaneously on Connect and on the Microsoft Developer Network (MSDN), the situation is likely to repeat itself this month.

In February, once testers did manage to get the code, they discovered a problem that testers of Microsoft's tools CTPs had come to know all too well: Different betas and CTPs don't mix well. Often times, even simultaneously released CTPs of WinFX and Vista don't mix.

The February Vista CTP was incompatible with Office 2007 Beta 1. But many testers didn't discover this until after they had already tried getting the two to interoperate. (Microsoft was slated to fix this problem in March with a technical refresh of the Office 12 Beta 1 code.)

It's not just logistics that are at issue. While the Microsoft developer division is very clear in distinguishing between CTPs and betas, the Windows team has muddied the waters, confusing its testers, partners and customers.

You want a CTP? Here's our latest, the Windows team offered. You want a full-fledged beta? No problem: This CTP is a beta, too. Huh?

No one should forget that Microsoft officials themselves coined the CTP term. When Microsoft launched the first CTP builds of Visual Studio more than a year ago, execs were precise in noting that CTPs were not betas, and thus, not of beta quality. CTPs were interim test builds, snapshots. As such, they shouldn't be tested or deployed the same way full-fledged betas are, they said.

Now, as Vista enters the final development stretch, Microsoft is redefining its terms all over again. The December Vista partner CTP, the February enterprise CTP and an April consumer CTP aren't mere CTPs: All of them also are Beta 2 builds, in Microsoft's new taxonomy.

To some, these are mere semantics. But I say, let's call a beta a beta, and a CTP a CTP. Keeping track of Microsoft's disappearing and reappearing features is tough enough without the definitions of commonly accepted terms changing, too.

What do you say? Is the Windows Vista CTP program worth the growing pains? Can Microsoft smooth the CTP process for Longhorn Server, Vista R2, Vienna and other Windows follow-ons? Or would you be content to go back to the plain-old beta process, minus CTP builds? Write me at mjfoley@redmondmag.com and let me know what you think.

About the Author

Mary Jo Foley is editor of the ZDNet "All About Microsoft" blog and has been covering Microsoft for about two decades. She's the author of "Microsoft 2.0" (John Wiley & Sons, 2008), which examines what's next for Microsoft in the post-Gates era.

Featured

  • Microsoft Offers More Help on Windows Server 2008 Upgrades

    Microsoft this week published additional help resources for organizations stuck on Windows Server 2008, which fell out of support on Jan. 14.

  • Microsoft Ups Its Carbon Reduction Goals

    Microsoft on Thursday announced a corporatewide carbon reduction effort that aims to make the company "carbon negative" by 2030.

  • How To Dynamically Lock Down an Unattended Windows 10 PC

    One of the biggest security risks in any organization happens when a user walks away from their PC without logging out. Microsoft has the solution (and it's not a password-protected screensaver).

  • First Stable Chromium-Based Microsoft Edge Browser Released

    Microsoft on Wednesday announced the first release of its Chromium-based Microsoft Edge browser at the "stable" commercial-release stage.

comments powered by Disqus

Office 365 Watch

Sign up for our newsletter.

Terms and Privacy Policy consent

I agree to this site's Privacy Policy.