The Perils of Patching
The process of keeping systems updated shouldn't be treated as the mindless that we mistake it for; no strategy is an invitation to disaster.
- By Roberta Bragg
In recent columns I've been advocating automatic updating for home users and
small businesses. As expected, several of you relayed tales concerning automatic
patching and how it can cause problems. I won't belabor the point, but I will
repeat it one more time: Determine the patching strategy that makes sense for
your business. This strategy is based on your risk assessment. In a very small
business or home network, the risk of disaster from not applying patches overwhelms
the risk that a patch may break some functionality.
However, Kevin Perry, Director of Information Technology for Southwest Power
Pool (SPP), www.spp.org, reminds us that in
at least one industry, there's greater risk in a patch breaking functionality
than there is in a patch not getting deployed. SPP includes as members 14 investor-owned
utilities, six municipal systems, eight generation and transmission cooperatives,
three state authorities and one federal government agency, three independent
power producers, and 13 power marketers. SPP coordinates, promotes and communicates
information about reliability in all aspects of the energy its members serve
to its more than 4 million customers. It covers a geographic area of 400,000
square miles, with a population of more than 18 million people.
Mr. Perry has this to say about automatic updates:
"Roberta: While in many cases, and especially in the case of home users,
automatically updating the Windows operating system and applications is desirable,
we must be careful to not make it a mandatory standard across the board as Microsoft
is supposedly considering. Within the electricity sector, we have critical systems
that operate the grid and keep the lights on. Some of these systems are Windows
based, some are Unix based, and all need to be patched as security upgrades
are announced. And within the industry, we are generally very good about applying
the patches as quickly as we can, as well as mitigating risk by other means.
The issue is that we must first carefully test the installation of the patch
in an offline environment to ensure that it does not break the critical third-party
applications we are running.
"The last thing I want to read by candlelight is that the next massive
power outage occurred when patches were automatically installed to our critical
"I am not suggesting that automatic updates applied to a critical system
will result in the next great blackout. Within the electricity industry,
we have recently implemented a cyber security standard that requires utilities
operating control areas to test updates to their critical operations control
center systems in an isolated environment, i.e. an offline system. That does
not mean that the critical production system won't crash when the update is
applied, but it minimizes the risk. And that is one of the reasons our critical
systems are redundant as well. We have to be able to control when the update
is applied as part of our risk mitigation strategies."
This is an important statement for a couple of reasons. First, it's a good
reminder for all of us. Decide what your risk factors are and act accordingly.
For many of you, that means a structured patch implementation process that includes
Second, there are vulnerable computer systems in places you never imagined.
Control systems that used to run on proprietary processors and use proprietary
communication protocols now run on Windows, Unix and Ethernet. This means administrative
consoles, control systems and other components that are part of a utility or
manufacturing plant must be protected in much the same way we protect ordinary
networks; in some cases, more so. These control systems may be just as vulnerable
as ordinary networks to attacks by individuals and by worms and viruses.
No, I don't think that utility companies are ignoring common sense and hooking
their control systems directly up to the Internet so that operators can surf
the Web. But they might be connecting them to office networks that are connected
to the Internet. They might also be in danger if remote connections are provided
for monitoring and maintenance purposes. In fact, these computers and their
networks are subject to all the vulnerabilities that the ordinary network has.
Perry's e-mail is significant for a number of reasons. The director of IT for
an organization of power companies is taking issue with me for advocating automatic
updating, and has laid out the possible horrific consequences of a thoughtlessly
applied patch. And his organization has given approval to having his e-mail
quoted. This tells me that the risk, while present, isn't being ignored. It
tells me that work's being done to secure these systems. The people responsible
for security there are being proactive. They're doing risk evaluation, testing
patches and applying them.
Please don't take this to mean that every utility company and manufacturing
plant is doing all they can; I don't know that.
Here's what I do know, however: If you're part of an organization that uses
similar Windows-based control systems, and your organization hasn't been addressing
this issue, it's time to do so. Traditionally, process people and IT people
have their differences. I have some thoughts on how to help them work together
to secure their systems and would be happy to talk to you. If you're addressing
security issues having to do with control systems, I'd also be happy to hear
Roberta Bragg, MCSE: Security, CISSP, Security+, and Microsoft MVP is a Redmond contributing editor and the owner of Have Computer Will Travel Inc., an independent firm specializing in information security and operating systems. She's series editor for Osborne/McGraw-Hill's Hardening series, books that instruct you on how to secure your networks before you are hacked, and author of the first book in the series, Hardening Windows Systems.