A Windows XP Post-Mortem: Fallout, Lessons Learned and Next Steps

We know that tens of thousands of copies of Windows XP are still in use, and many will likely remain so. We can't just arbitrarily ditch critical services just because they happen to be on XP.  But we can sit for a moment and think about how we got here. How are we running "mission-critical" services on a 12-year-old operating system? Is there any reason to worry about it? How can we change our IT management practices so that we don't end up in this situation again? In this six-part article, we'll conduct an XP post-mortem, and see what lessons we should be taking away from the OS that won't let go.

Well, Here We Are
Microsoft has officially ended support for Windows XP, meaning there will be no new updates. No Security Essentials updates. No patches. No nothing.

Well, not "ended," really. Some entities -- notably, so far, the Dutch and U.K. governments -- are opting to pay Microsoft millions for "custom support," so that Microsoft will still produce patches and whatnot for them. That support is limited in what it will do, and is intended to give organizations more time to transition. You know, more than the half-decade-plus they've already had.

But for the rest of the world, XP is over. Although plenty of folks will still be using it. According to Netcraft , there are thousands of Web sites still running on "XP servers." Yeah, think about that -- XP servers. But it's OK -- XP isn't alone. The Australian Postal Corporation, for example, appears to be running their online postage sales site on a Windows NT 4.0 server. Yikes.

A lot of folks, myself included, aren't very patient with the amount of XP still running around the universe. In my personal blog, I posted a rant that rails against the managerial incompetency that resulted in governments paying millions of taxpayer dollars to support an obsolete operating system -- a situation I feel was totally avoidable. Don't worry; this blog won't be a rant. It'll be a much more professional, managerial look at how we got here, and what we could have done differently.

Most "we have to keep running XP" situations probably come not from people who were caught unprepared by the whole end-of-life situation (something Microsoft has repeatedly extended in past years), but rather from people who have systems who have to run XP. Embedded systems. Specialized software. That kind of thing. Understand that those situations don't make all of this acceptable, but we're stuck with what we're stuck with. The point of this series is to try and make the current situation a teachable moment, so that we can learn from what we've done, and to see if we can do better the next time around.

Let's be clear: we're not really trying to assign blame here. But running on obsolete OS for mission-critical applications is an incredibly poor practice, and we do need to acknowledge that. We also need to look back and figure out why things turned out like this, and try to learn something from it. There's a very real knee-jerk reaction to say, "well, it's the best we could do," but that isn't true. We can do better.

But first, we need to talk about the very deal dangers of running XP today, and consider some practical steps to make sure we're still safe.

The Very Real Business Dangers Ahead
The problem we now have to deal with is ensuring our mission-critical services running on XP stay running. We're in a bit of a situation that IT teams haven't been in before. In the past, hackers and other attackers were primarily coming after us for the fun of it, and for the street cred in saying they'd successfully done so. There was some risk in running an outdated operating system, but it was perhaps minimal. As the installed footprint for an OS dwindled, it became a less-attractive target to hackers because they couldn't make a big splash anymore.

Hacking today is a for-profit enterprise. Hackers have a purpose: they're trying to, in most cases, steal something that will earn them a profit. They'll install Bitcoin miners and steal processing power. They'll steal customer information and sell it. Imagine being able to steal information from a hospital's XP-based blood analysis computer, or being able to fiddle with the XP-base dispatching system used by a truck company. Those hackers could wreak havoc. But they could also make a lot of money.

Hackers today are a lot less likely to try and make a big splash. They're smarter than that, now -- they're not script kiddies. I wouldn't be surprised, for example, if someone's discovered some exploit in XP and has held on to it for years, waiting until Microsoft would no longer fix the problem. Now they can sail into tens of thousands of computers and start attacking essentially unimpeded.

That should terrify you.

If hackers are able to install some malware, it probably won't be very visible. It'll sit in the background doing its thing. If it lies low, and only attacks XP machines, there are good odds the antimalware vendors of the world won't notice it, meaning it'll be able to run for a good long time.

The lesson of Target is that it only takes one time to cost you billions. Even if you've never been hit before, when you're running an easy target like XP -- and doing so to run mission-critical services, not just someone's word processor -- you're an attractive target.

So what can you do to help protect yourself?

• Keep your antimalware software up-to-date. Most vendors are continuing to support XP for a while longer.
• Ensure XP is as cut off from outside communications -- and that includes your own LAN -- as possible. Restrict both incoming and outgoing communications using Windows Firewall or by using third-party local firewall software.
• Think defense in depth. Assume each defense you have will be broken, and have something behind it ready to pick up the slack.
• Store as much data as possible off of the XP machine. Make it as much of an appliance as possible. Make an image of it (ensuring there's no malware running at the time), and restore to that image regularly. That's a great way to help limit malware's reach.
• Upgrade Internet Explorer 6 to the latest version possible. IE 6 is a known cesspool of weaknesses and exploits.

Yes, XP is going to require a lot more work from you to keep it safe – just like owning a 12-year-old car requires you to spend more on maintenance. Don't fail to do that work. I realize that few organizations have security and risk mitigation as a real part of their culture, but it only takes once for an attacker to put you out of business.

So, with an idea of what dangers XP presents, and what you can do to try and mitigate those, let's start thinking about what we could do differently, in IT management, to not be in this situation again.

Next Time: Planning for Obsolescence
Windows XP was likely one of the most popular and best-selling business operating systems of all time. It's where Microsoft really took off, and tens of thousands of companies built solutions for XP. From specialized hardware to embedded systems, XP was everywhere.

When we started making all those XP acquisitions, we never really thought about the day when XP would go away. Security wasn't a big deal to most organizations, and up to that point we'd upgraded our computers every three years or so. Nobody at the time imagined a 12-year-old operating system. Heck, in 2002 when XP came out, most of us hadn't even had a local area network for 12 years, yet! So we made a lot of IT acquisitions -- software, hardware, and more -- without really considering that the OS they all relied on would eventually become obsolete.

But it did.

And now we should know that everything in IT will become obsolete. And we should plan for that. Every single IT acquisition you make -- and I'll discuss a couple of specific situations coming up -- should include the question, "what will we do when this is obsolete? What's the upgrade path?"

Vendors might not be ready to answer that question -- so we're all going to have to work together to do so. I can guarantee that if you buying or not buying a vendor's produts, someone will find an answer for you. And those answers don't actually have to be difficult or complex, as we'll explore. But the point is to ask the question.

And I'm not, as some folks might suggest, being a "single-issue voter," here. I'm not saying that having an upgrade path defined ahead of time is the only possible thing you should consider in an acquisition. But I'm betting it's not been on the list of discussion topics before, and I'm saying it must be something you discuss going forward.

So, aside from getting a vendor's solemn promise to upgrade their software to support all new versions of Windows going forward -- a promise you're unlikely to get, anyway -- what can you do?

Next Time: Specialized Software
Let's start with the question of specialized software. In fact, let's first distinguish "general-purpose software" like Microsoft Office, which we all know will have an upgrade path, from the more-expensive "specialized software" that you use to actually run your business.

Start by looking at vendors' track record for upgrades. Have they commonly offered upgrades to accommodate new versions of Windows? What did those upgrades look like? Were they painful, or relatively smooth? Does the application's basic architecture lend itself to upgrades? For example, does it keep data in an external store, making it less reliant on the local OS? Does it use a lot of version-specific tricks that might not carry forward into later versions?

Most importantly, what can the vendor do to ensure that you have options? Most governments and larger corporations require software vendors to put their source code in escrow. Escrow services keep a copy of the source code, and only release it under specific circumstances that are defined in the escrow agreement. Many customers use checksums of the source code to verify that the code they're running in production matches. With escrowed code, if a vendor goes out of business, you have the option of hiring someone else to maintain the application -- including modifying it to run on a newer OS, if needed. Your escrow agreement might allow you to access the source code if the original vendor no longer maintains that version of the software, so that you can do so yourself. It may sound weird, but escrowing like this has been a common practice for more than a decade -- and it's something any sized organization should do when it comes to specialized software.

If your software acquisition includes a maintenance agreement, then ensure it requires the vendor to support new versions of Windows within some reasonable window of time. After all, that's the whole point of maintenance.

The point is to work with vendors to ensure that your expensive new software purchase won't mandate that you be stuck on a specific version of Windows forever and ever. Give yourself options by planning ahead for obsolescence.

Next Time: Specialized and Embedded Devices
This is a tougher set of considerations. In some respects, specialized hardware solutions running an embedded OS are just appliances. Do you worry about the OS your toaster is running? Well... yes, you do, if it's storing important data or plays a critical role in your business.

Again, this is where a discussion with the vendor is important. How long will you depreciate this new acquisition? 10 years? Then get the vendor to commit to providing updates during that period -- and to escrow their source code in case they go out of business.

Further, understand exactly where the device's potential security concerns exist. Does it communicate with a network? Does it store customer data locally? If we do need to run its operating system for years after obsolescence, how would we lock it down to protect it so that we can continue using it? Knowing those answers up front lets you have a plan in place -- even if that plan includes just leaving the original OS intact for years. In other words, in some cases running an obsolete OS can be fine -- if you're prepared for it.

But be prepared. Today, there are thousands of point-of-sale systems in retail and restaurants that still run Windows XP operating systems, including XP Embedded variants. You know these things are connected to a lightly secured network in most cases (hello, Target), and yet they're dealing with important customer data every minute of the day. "Well, we're a tiny business -- nobody would attack us." Not true. An attacker can make hundreds of thousands of dollars by capturing a few dozen credit cards, and they're more likely to attack a small, poorly defended network to get those card numbers. And the costs to you, as a tiny business, will be ruinous.

On the other hand, you may have computers that aren't network-connected. They just sit in a corner making copies of keys or something. Those can likely be left alone. "Disconnected" is an incredibly safe state, requiring physical access to compromise.

So next time, make sure you know the plan for upgrading those specialized devices. If you're talking to a vendor who doesn't have a plan, consider talking to their competition instead. See if you can get everything your company needs and an upgrade plan -- even if that "plan" doesn't include actually upgrading the OS on the device.

• Ask vendors to participate in source-code escrow for all software that you buy. Dozens of firms offer this service (search for "source-code escrow" in your favorite search engine), and it provides you with a last-ditch way of dealing with software if the vendor can't, or won't.
• Look at vendors' track records for upgrades. Were they usually forthcoming with new versions, or do they tend to stick you with what you bought and leave you there?
• Before acquiring new solutions, understand how they interact with the outside world. How can you secure them should you need to run them past the point of obsolescence?
• Consider the lifespan of the hardware or software you're buying, and try to take steps to ensure its continued support through at least that lifespan. And then plan to replace it once that lifespan is up. A 10-year solution that you're running into its 14th year is probably not meeting your business needs any longer, anyway.
• Deciding that you'll run an old OS is fine; letting it happen to you is not.

A lot of us are going to be stuck running XP because of non-decisions made years ago -- in many cases, by different technology leaders. But we can learn from the past, and we can try to actively manage so that it doesn't happen again.

Posted by Don Jones on 04/17/2014 at 11:09 AM