In-Depth

Microsoft Outlines Its Tooling Future: Moves Away from Traditional Testing Methods with Windows 10

• By Kurt Mackie
• 06/21/2017

IT pros may be wrestling with how to test and maintain Window 10, given the operating system's faster release cycles, but to hear Microsoft tell it, testing is just passé.

Microsoft actually stopped testing its internal "quality" operating system updates after the release of Windows 8. As a result, the team at Microsoft has seen no issues with the line-of-business apps it tests, so it likely won't go back to testing those quality OS updates in the future.

At least, that's the somewhat radical viewpoint espoused during a Tuesday Web presentation by Robert Van Winkle, principal program manager, and Brent Barnett, program manager, on the Device Experience and Productivity Services team at Microsoft. They outlined a sort of easy-street role for IT, where end users do all of the application compatibility testing. Sometimes, end users will even do the Windows 10 update installs, too.

Microsoft used to do traditional Windows OS deployments, which took a year of planning. However, since Microsoft moved to performing in-place upgrades with Windows 10, there have been fewer internal help desk calls. Microsoft uses 25,000 employee volunteers to test Windows as a service with its early Windows 10 internal releases. They choose which applications to test, and social media is used to communicate instead of going to the help desk.

The June 20 talk, "Windows 10 Deployment: Tips and Tricks from Microsoft," is now available on demand with sign-up at this page.

Microsoft's Testing Approach
Microsoft has about 1600 applications for testing with Windows 10 releases each month. However, it only tests about 250 of the critical business applications prior to its OS releases because it doesn't think it's necessary to test them all. It expects end user testing at Microsoft to surface any problems with the other applications that might be used with its Windows 10 builds. Typically, Microsoft expects to catch problems with those other untested applications through social media communications by the Microsoft testers. The application testing is conducted via a virtual machine client farm, which provides about 300 virtual machines using 10 servers. The use of virtual machines helps with the OS refreshes.

Most of the applications that Microsoft tests with Windows 10 are browser-based apps, and the Microsoft Edge browser is the main browser used for testing. Universal Windows Applications represent just one percent of the apps tested. Microsoft has about 20,000 devices it uses for the testing.

Microsoft claims to be getting 100-percent pass rates in its critical business application test results with recent releases of Windows 10 (see chart). While there haven't been compatibility problems with the applications tested, Microsoft has seen issues with services, such as VPNs.

It's not clear whether Microsoft expects organizations to see those same results. The implication of the presentation seemed to be that if Microsoft is seeing those compatibility numbers, then organizations ought to be seeing them, too.

Microsoft uses Windows Upgrade Readiness, part of Microsoft Operations Management Suite, to check for compatibility and driver issues. It can be used to remove devices with known issues. Upgrade Readiness integrates with System Center Configuration Manager.

Other Microsoft IT Plans
Microsoft has other seemingly radical plans for its own IT department. For instance, it's working on various tooling shifts (see chart).

For example, there's the obvious shift from using Active Directory on premises to using the Azure Active Directory service. Microsoft is also planning to move from using System Center Configuration Manager (SCCM) to using Microsoft Intune for its device management. The only thing holding Microsoft back on the shift to Intune is security policy.

On the OS deployment side, Microsoft is moving from using custom IT-built images to using generic OEM-built images, which is conceived as a way of helping to reduce complexity. For managing quality OS updates, Microsoft is switching to Windows Update for Business, a Windows 10 update service, rather than using its other management tools like SCCM. Lastly, on the reporting side, Microsoft is using its Windows Analytics tool that's part of Microsoft Operations Management Suite, which is typically used for managing cloud services.

During the Q&A part of the presentation, the Microsoft officials denied that Windows Server Update Services (WSUS) was going to get replaced by Windows Update for Business. However, they did suggest that enterprises should evaluate the use of Windows Update for Business vs. WSUS.

The presenters also explained during the Q&A that Microsoft is currently working on building a migration path from SCCM to Microsoft Intune. The plan is to get the two management systems to coexist first so that SCCM can be removed later. In general, Microsoft sees Intune as the future for management, but SCCM will likely stick around.

"Internally, we're definitely trying to move away from SCCM for management and moving to Intune," Barnett said. "We definitely see that as being the future for managing systems. With that said, It's also important to recognize that there's a vast array of enterprise customers with different needs, and so there are places where SCCM's probably going to be predominant and make sense for years into the future."

The presenters noted one caveat when relying on Microsoft Intune and Azure AD. Organizations will lose Group Policy capabilities. If Group Policy is needed, then organizations should stick with using SCCM, they advised.

On the applications side, Microsoft is planning to transition to using the Windows Store for Business service, tied to Microsoft Intune, to manage app deployments. The presenters also noted that Microsoft Intune has a new feature called "Sidecar," which is part of the Windows 10 creators update and the latest version of Intune. The Sidecar feature can be used for handling complex MSIs. It lets IT pros create a script to install all of the items that should be installed on every device in the enterprise. Possibly, Sidecar is documented somewhere, but it's not listed in Microsoft's "What's new" documentation.

Time will tell if Microsoft's vision will prevail and filter down at the practical level within IT shops. For IT pros managing complex computing environments, Microsoft's twice-per-year major Windows 10 OS update releases likely will pose challenges. It may be especially problematic when "legacy" or custom-built applications need to be maintained, for which Microsoft admittedly won't be running any tests. The burden, as always, will fall on IT pros should things break.