In-Depth

Patching Made Simple

Patching your servers is an art that takes time to master. Here's a paint-by-numbers kit to help you get started.

• By Greg Shields
• 06/01/2005

As an IT admin, you can tell when that day is coming. Your palms get sweaty. Your heart beats faster. You start to shake and a feeling of dread comes over you. You consider the advantages of bungee-jumping without the bungee.

Yup, Patch Tuesday is coming again. Is it too late to call in sick?

Known as Black Tuesday in some circles, the second Tuesday of the month, when Microsoft releases its patches, has in a short time become your worst enemy.

But there's a way to fight back. You'll never completely vanquish that nightmare but you might battle it to a draw. To do that, you'll need to learn the basics of patch management and build on them.

Black Tuesday: Mark It in Pen
Let's first talk about schedules. Except in special cases, Microsoft releases patch announcements to the general public on the second Tuesday of every month. Some months there are no new patches, but most include at least a few and, sometimes, as many as 10 to 15. The patches traditionally hit e-mail inboxes between 4 p.m. and 5 p.m. PT.

The format of the announcement is the same every month, to make it easy to find the basic information. Each announcement includes the patch number—the MS05-015 nomenclature—and the bulletin number, usually a six-digit number. The first two digits of the patch number give you the year of its release, followed by a three-digit number showing its release order. The first patch of 2005 gets the designation MS05-001, the second MS05-002 and so on.

The bulletin number corresponds to the Microsoft Knowledge Base article associated with the problem discovered in the software. As you know, if you've ever been on Microsoft's support site, not all bulletins involve patches, but all patches involve bulletins. This number is useful as it will be your way to find the details when searching http://support.microsoft.com.

The e-mail notification includes the patch version number, the affected and non-affected software, a PGP signature you can use to verify the authenticity of the message, and the patch's "criticality" level (more on that in a second). A URL to detailed information about the patch is also included. In every case, you should use this link to determine if the patch affects your software and how.

Criticality and Supersedence
"Criticality" is a patch management buzzword for, "How fast do I really need to get this on my machines?" Microsoft defines four levels of criticality:

• Critical: A vulnerability whose exploitation could allow the propagation of an Internet worm without user action.
• Important: A vulnerability whose exploitation could result in compromise of the confidentiality, integrity or availability of user's data, or of the integrity or availability of processing resources.
• Moderate: Exploitability is mitigated to a significant degree by factors such as default configuration, auditing or difficulty of exploitation.
• Low: A vulnerability whose exploitation is extremely difficult, or whose impact is minimal.

Your speed of deployment is a decision made in concert with your business. If your business is particularly sensitive to downtime and has appropriate controls in place to mitigate the effects of a particular vulnerability—like firewall restrictions and system lockdowns, for example—you should adjust your deployment timing appropriately. Above all, before Black Tuesday arrives you should have a time frame for when all systems should be patch compliant.

New to WSUS since its RC1 release is the ability to change the default port that clients look to for updates. During the WSUS setup, you have the option of choosing the default IIS port for HTTP, port 80. Alternatively, you can choose to create a separate WSUS Web site on alternate port 8530. Once setup completes, you can go back into the IIS console and change this alternate port to any of your choosing. If you change the default port on the WSUS server, you'll need to change it for the clients as well. When setting up the clients for the alternate port, use http://WSUSServer:8530 in Group Policy. Why do this? Consider the example where a bad exploit has somehow gotten into your network. All your Web servers are down because this exploit is transmitted across the default IIS port, port 80. You're trying to get the patch out to fix the problem and bring your Web servers back online, but your WSUS infrastructure is also down because it uses the same well-known port as the exploit. If you'd changed your default port, you'd be happily patching systems using WSUS. If you didn't, you're running around to each machine with CDs in hand. What's even cooler, but out of scope of this article, is combining this alternate port with SSL. Using SSL within your WSUS infrastructure ensures that your clients never get viruses disguised as patches from a hacked machine masquerading as a WSUS server. If you combine that SSL safety net with a non-standard port, you have maximum assurance that updates are secure and deployable during times of crisis. Check out the Deploying Microsoft WSUS guide for details on how to set this up. — G.S.

Remember that Microsoft's criticality may not necessarily be your criticality. That being said, Microsoft releases Critical patches for a reason—they have a high likelihood of infection through a mechanism that doesn't need any action by a user.

The other major factor in determining what and when to patch is supersedence. Supersedence is the idea that some later-released patches fix problems already fixed by previously released patches. MS04-045, "Vulnerability in WINS Could Allow Remote Code Execution," for example, supersedes MS04-006, "Vulnerability in the Windows Internet Naming Service (WINS) Could Allow Code Execution," because it fixes the vulnerability identified in MS04-006 as well as additional problems found since the release of that patch. So if you haven't already installed MS04-006, you don't need to bother if you install MS04-045.

Supersedence information used to be difficult to locate without lots of research and cross-referencing between available patches. However, with the release of Windows Software Update Services (WSUS)—discussed shortly—supersedence information is now much easier to find.

Take a look at Figure 1. Notice in the patch detail information that the highlighted patch, Cumulative Security Update for Outlook Express 6 Service Pack 1, supersedes 330994 and is superseded by KB823353.

Figure 1. Windows Software Update Services (WSUS) helps out the harried admin by listing supersedence information (or, what previous patches the current patch covers). (Click image to view larger version.)

Cheaters Always Prosper
Even with all the info in the WSUS tool and on the Microsoft Web site, you're probably going to need to create your own little patch "Cheat Sheet." Keeping straight all those patch designators, descriptions, criticality and supersedence information is difficult when paging through the WSUS interface. Creating a one–page spreadsheet will save you hours of paging through WSUS or returning to Microsoft's Web site every time you need to check status.

Creating this cheat sheet sounds like a daunting task, but with 45 patches released in 2004 and 51 in 2003, you're going to have a difficult time keeping them straight. Some of the patches that go back to 2003 are still critical. Keep your patches straight by keeping one tab that shows details of every patch, and a second tab showing just the patches that make up your minimum baseline. It's also helpful to include a recommended course of action for every row in your Cheat Sheet: Patch Immediately, Defer, Delay until Next Service Pack and so on. An example Cheat Sheet is shown in Figure 2.

Figure 2. Keeping a spreadsheet list of patches released by Microsoft and your current server patch status is one trick to keeping your network secure. (Click image to view larger version.)

WSUS Setup Is a Snap
Now that we've gone over the basics and the process, let's talk about the tools. Microsoft designed WSUS with the small business in mind: It's free and has light hardware requirements. With no software requirements other than the free IIS, SQL Desktop Engine, BITS 2.0 and .NET v1.1 runtime components, it's easily integrated with already-existing intranet server hardware. If you manage a small infrastructure with a handful of servers and a slightly bigger handful of workstations, you're really going to enjoy this tool.

To set up WSUS, ensure IIS is installed and active on an internal server, then log on with Administrator privileges and run the setup program. Setup will ask you for a location to store software updates and prompt for installation of the SQL Desktop Engine, to which it defaults for its database. It will create an IIS Web site instance, which will become your Control Panel for configuring WSUS. Once the setup is complete, you'll do all configuration from http:///WSUSadmin.

Go Directly to the Source
During this setup process there's one nifty WSUS component you should be ready for. During setup only, you'll have the option to Select Update Source. Here you'll decide if your clients obtain their patches from your local WSUS server or from Microsoft's www.windowsupdate.com site. You will still approve and reject patches as you see fit, but the mechanism of distributing the patches will differ. This choice has implications for your environment:

• If your clients are desktops that primarily reside within the confines of your business LAN, you will probably want them to update from your WSUS server.
• If your clients are laptops that primarily roam outside your LAN, or if your clients are in networks with low bandwidth connections to the WSUS server but fast Internet connections, consider having them update from Microsoft.

Updating directly from Microsoft is a huge advantage to administrators in laptop-heavy or high-latency environments. You can retain the ability to approve only those patches you have tested, but your laptop users will update those patches straight from Microsoft via the Internet.

For this functionality to work, your laptops will need to log in to your LAN, either directly or though a VPN, to receive the update notifications from WSUS. Also, be aware that because Update Source settings are configured during setup, you'll need to create two separate WSUS servers if you want both deployment options.

On Target
When Microsoft released the previous version of WSUS, called Software Update Services (SUS), there was no mechanism to identify groups of machines to patch. Short of setting up tiered SUS servers with different approval schedules, patching was an all-or-nothing activity. Once a patch was approved, SUS clients would begin patching the server at their next detection cycle.

WSUS fixed this limitation by adding "target groups." These machine groups, shown in Figure 3, give you the freedom to patch systems on different schedules. As an example, you'll probably create a test group of machines. You can approve patches to this test group first, so you can pre-test newly released patches prior to general release. Similarly, you can delay patching patch-sensitive or downtime-sensitive environments until an appropriate outage window.

Figure 3. WSUS lets you select groups of computers to patch, rather than applying the patch to all computers at once. (Click image to view larger version.)

You can populate WSUS target groups via the client side, using Active Directory GPOs or, where AD isn't present, manual Registry modifications. Alternatively, you can populate groups from the server by manually dropping each machine into its appropriate group. If you use AD, using client-side targeting makes life easy by mapping your target groups to your OUs.

Only one method can be used at a time per WSUS server. Configure this from the WSUS administrative console by clicking Options | Computer Options. Here, you can choose to Use the Move computers task in Windows Server Update Services for server-side targeting or Use Group Policy or registry settings on client computers for client-side targeting.

When Time Is of the Essence At times you just don't have time to wait for the WSUS detection cycle to complete, because it can take up to 22 hours. Sometimes, when testing or under imminent attack, you may need to get a client into WSUS and patched right away. When this happens, get the client into the correct OU (if using client-side targeting) and run the command wuauclt.exe /detectnow. This command will force the client detection process to begin immediately. Shortly thereafter, you'll see the new client show up in the Computers list on the WSUS admin Web site. If a client is already in WSUS but you've changed its group membership, use the command wuauclt.exe /resetauthorization /detectnow. WSUS stores client-side targeting information in a cookie on the client itself. This command forces that cookie to expire and the client to re-start the detection process. — G.S.

A Side Order of Reboots
No guide to patching is complete without a discussion of reboots. Most patches require reboots to complete the installation process, and depending on the network, those reboots can be painful.

Like most other client configuration tasks, WSUS handles reboots through AD. Open your Group Policy Management Console and navigate to Computer Configuration | Administrative Templates | Windows Components | Windows Update. As of the WSUS RC1 release (the latest available version at the time of this writing), Microsoft included 10 different Group Policy settings. If you only see four settings to modify, you probably have an old version of the WUAU Administrative Template. You'll need to add the template to this GPO for the AD component of WSUS to be fully functional. The WUAU template, named wuau.adm, can be found in the C:\WINNT\INF of your WSUS server.

For reboots, two configurations are most important: Configure Automatic Updates Properties and No auto-restart for scheduled Automatic Updates installations.

The first of these identifies whether clients download and install the update or merely download the update and wait for the user. This is also where you set the schedule for the update. The second determines how your clients handle reboots. If this policy is set to Enabled, computers will not automatically restart when required to complete a patch installation.

If your clients remain logged in when they go home, and you've set this second policy to Enabled, your clients won't complete the patch installation. You will, however, probably retain your job, since your users won't lose their open files.

What all this means is you need to be aware of how your business handles "forced" reboots. Although it delays the patching process, it's probably a good idea to set the "No auto-restart" option to Enable until you've notified users via e-mail about the addition of "forced" reboots.

Move to the Sequel: Patching for Experts
With the right information and tools, the nightmare that is Black Tuesday may start to feel more like mild indigestion. You've gotta plan for it, analyze the patches you get and test everything before you send any out the door. But patching should be just part of the routine. Develop your skills, starting with the lessons here, and never again fear the dreaded Black Tuesday.