News

Microsoft Promises Greater Transparency on Windows 10 Update Quality Issues

• By Kurt Mackie
• 11/14/2018

Microsoft on Tuesday kicked off a discussion series in which the company is promising to be more transparent about Windows 10 servicing and associated quality issues.

The first communication of the series, authored via a Windows post by Michael Fortin, corporate vice president for Windows at Microsoft, comes in the context of the rerelease of Windows 10 version 1809. Microsoft had originally issued Windows 10 version 1809 on Oct. 2, but later suspended it due to data loss issues for some users.

Windows as a Service
A few years back, Microsoft had shifted to a "Windows as a Service" approach with Windows 10. Under this feature-update scheme, new OS features arrived biannually, in the spring and fall. It's a faster update pace than the old service-pack model Microsoft had with Windows 7, where new OS features arrived perhaps every two or three years. The Windows 10 Enterprise edition does afford a "long-term servicing channel" option that's similar to the old service-pack model, but Microsoft recommends its use only for things like medical devices. It generally discourages organizations from using the long-term servicing branch option.

Early on, at the start of the Windows as a Service process, Microsoft was successfully sued over its disruptive OS updates pushed down via Windows Update to machines. However, the company ultimately never budged from its faster release approach with Windows 10. Microsoft has typically argued that this faster update release pace is necessary, both for innovation purposes and for ensuring security.

Successful Approach
In his announcement, Fortin argued that given the complexity of having to support different Windows and applications versions, as well as multiple drivers and device types, Microsoft has been doing a good job with Windows 10 updates. He offered a few numbers to illustrate the burdens Microsoft faces in maintaining Windows 10 update quality:

• Over 700 million monthly active Windows 10 devices
• Over 35 million application titles with greater than 175 million application versions
• 16 million unique hardware/driver combinations

Next, Fortin offered the following chart, suggesting that the customer-reported incident rate associated with Windows 10 releases has been declining since the release of Windows 10 version 1507 more than three years ago:

Possibly, the number of Windows 10 version 1809 quality complaints by customers hit a high of about 500 devices out of a million devices, depending how this chart is interpreted.

In any case, Microsoft has been following a "data-driven listening approach" to track the quality of Windows 10 releases, Fortin argued. Microsoft's "customer service call and chat volumes" have been decreasing even as the number of Windows 10 users has been rising, he added.

Testing Shift
On the testing front, Fortin admitted that Microsoft's developers now do the "functional testing" of Windows 10, and Microsoft also now relies more on customer engagement and telemetry reporting to check Windows 10 quality issues:

We shifted the responsibility for base functional testing to our development teams in order to deliver higher quality code from the start. We also changed the focus of the teams that still report to me who are responsible for end-to-end validation, and added a fundamentally new capability to our approach to quality: the use of data and feedback to better understand and intensely focus on the experiences our customers were having with our products across the spectrum of real-world hardware and software combinations.

Microsoft's shift away from using dedicated quality assurance (QA) testers has been noted by others. Veteran Microsoft reporter Mary Jo Foley noted that Microsoft cut its internal Windows QA staff about four years ago, while a Bloomberg report described Microsoft's cuts to its QA subcontractors.

Chris Morrissey, a member of Microsoft's Windows servicing and delivery team, had noted back in May in a Microsoft Tech Community post that Microsoft's software team now builds automated tests based on artificial intelligence:

Our approach in Windows has evolved to a focus on quality upstream, so testing occurs earlier and more often by Software Engineers as part of Development. For us, end to end testing occurs through many mechanisms. As an example the teams that used to build automated tests now build responsive data-driven and AI-driven approaches to monitor both Windows Insider "flights" and feature updates. This gives us more realistic coverage of the issues customers see in the real world in a way that allows us to be fast and responsive. 

Fortin noted that Microsoft uses machine learning to prioritize which devices should get Windows 10 updates. "If we detect that your device might have an issue, we will not offer the update until that issue is resolved," he added. Still, he admitted that some users will experience problems under Microsoft's approach.

"Even a multi-element detection process will miss issues in an ecosystem as large, diverse and complex as Windows," Fortin stated. Microsoft is planning to increase its transparency with "clear and regular communications with our customers when there are issues," he added.

Quality Trends
Fortin's promise likely stems from the many quality issues that became apparent with the release of Windows 10 version 1809. He noted in his announcement that its delay last month was "the first time in Windows 10's 'Windows as a Service' history that we have taken such an action." Based on his overall comments, though, Microsoft isn't proposing to do anything more than increase its communications when things go awry.

However, many observers see problematic Windows 10 releases as more of a recurring pattern than the downward trend characterized by Fortin. For instance, such points have been made by Susan Bradley, a Microsoft Most Valuable Professional (MVP) and moderator of the Patchmangement.org list-serve discussion forum for IT pros.

Bradley found an entirely different view of Windows as a Service quality in her informal survey of IT pros. She wrote a letter to Microsoft officials begging them to slow down the Windows 10 releases and focus more on quality. At the time, she had referred to quality issues associated with Microsoft's July 10 Windows and .NET Framework releases, which had created problems for users of Microsoft's business software. Bradley also argued that the complexity of Windows versions was just not helping with the patch and update process.

For now, though, Microsoft appears to be staying the course, and doubling down on Windows as a Service.