Thin Clients, Fat Heads

This business owner’s thin-client Windows network was impregnable. Or so he thought, until he met Bob…

I get a call from one of my customers. “There’s something wrong with the network. Everything is really slow.” I tell him I’ll look into it right away. As I start to check it out, I get a call from another client.

“We’re really crawling here. Is there something wrong?” I tell her that we’re already checking it out, and we should have things back to normal soon.

Another client, then another, calls, each with the same complaint. One more comes in. This time my customer offers an explanation: “I think there’s a virus on the network!”

Virus? No Way!
A virus? There’s no way we could have a virus. Yes, this was all happening in the midst of a serious virus outbreak, and yes, many large corporations had already been infected, but not our systems. Let me explain.

My company, EasyOffice Network (EON), provides clients with comprehensive, thin-client–based, fully-managed IT solutions. We supply, build, configure, manage and retain ownership of all computer systems used by our clients. I’ve built most of these IT infrastructures from the ground up and have employed best practices in how we manage everything from desktop machines to file severs to routers to firewalls. I’ve been doing this for a long time and am proud of a long track record of keeping my clients’ systems in good working order.

We’re able to do this thanks to several key elements of our service model:

 The capabilities of Terminal Services in a Windows 2000/XP environment.

 The licensing agreement we have with Microsoft that allows us to rent the use of Microsoft software to our clients.

 The ability to remotely administer the computers and network systems in place at our client sites.

 The fact that our high-end data center systems are used to support several clients.

 The proprietary expertise we incorporate into our system configurations.

Because we use a server-based computing model with thin-client workstations, the technical requirements for one of our IT solutions are quite different than those for a traditional office network environment. To illustrate, let’s look at a typical EON solution.

Our customer, Little Guy Industries (LGI), has two locations: their main office in Los Angeles and a smaller branch office in San Jose. We put a file server, an application server, a SQL server and a domain controller in the Los Angeles office. In San Jose, we placed a domain controller (which also serves as a file server) and an application server. The application servers run Terminal Services. All servers are configured with real-time virus-checking software.

Thin Is In
For the clients, we use LGI’s existing Dell computers—a mixture of older Celeron- and Pentium III-based machines. We rebuild all the Dells with a standard Windows XP Professional configuration. These machines are used to do one thing, and one thing only: run the Remote Desktop Client to connect to an application server. They won’t be used for Internet browsing; they can’t even get out to the Internet! As such, we don’t install any kind of virus scanning software on clients—remember, all program activity takes place on the application server through a terminal services session; nothing actually takes place on the client machine.

The offices are each connected to the Internet via DSL, protected by industry-standard firewall appliances. Interoffice network communication is handled by a hardware-based VPN. Only the servers have access to the Internet, and then only through a firewall.

Now that you have an idea about the typical EON environment, let’s see what happens when trouble’s afoot. Suppose it’s a Tuesday morning, and word of a new virus threat makes the rounds. One of our standard procedures is to check the firewall traffic logs to see what kind of packets are hitting our networks from the outside, and sure enough, we find evidence of attempted attacks. We contact our upstream Internet provider (if they haven’t already contacted us first) and ask them to keep tabs on the traffic going through their networks.

We also check our server logs for evidence of virus activity. They show lots of attempts but no infections. Thanks to the faithful real-time virus scanning software installed on each server, no one suffers any ill effects from this attack.

We’ve survived one of the more destructive and widespread virus attacks that have come down the pike in a long time. Feeling ever so smug in our prowess over infectious bytes, we send an e-mail to all our clients detailing how severe this virus attack was to many Fortune 500 corporations—and emphasizing how not one of our clients suffered even as much as a sniffle.

Virus Outbreak No. 2
Several weeks later, another virus outbreak. We dutifully carry out our standard procedures, and again our servers are undaunted. We’re thinking about sending out another boastful e-mail—but wait! Our clients start reporting problems—slow performance, unable to connect, dropped terminal server sessions.

OK, so we hold off on sending the “We beat the virus” e-mail to our customers and start troubleshooting. Could these problems be related to the virus outbreak? No, we think, just an untimely coincidence. There’s got to be something we can find to explain these problems.

After observing heavier than normal network traffic, we find that there is, indeed, evidence of virus activity on our internal networks. So I ask my techs to pore through the firewall logs again. But this time, I tell them to look at activity from inside our NAT-compliant, firewall-protected, private networks. “But the only way into our private networks is through public Internet gateways, and we’ve already proven that nothing came through any of our firewalls,” my senior tech points out.

I repeat my instructions. He be-grudgingly complies with my request. Later that day I get an e-mail from my tech. He tells me that he was able to narrow down where all this virus mayhem began—at one of our client sites (let’s refer to this client as “Bob”), from inside the private network. As I’m contemplating how this could have happened—how a virus could just appear in a private network without any trace of it coming through the firewall—I get another e-mail. This one is from Bob.

He tells me that he thinks his laptop is infected with a virus and wants to know what to do about it. “What laptop?” I think to myself. We never sold or discussed a laptop computer with this client. I e-mail him back. I tell him how to discover the MAC address of his NIC and give this bit of information.

I pass this info on to my tech, and he confirms that Bob’s laptop is the source of all this extra network traffic. Further analysis reveals that none of our other internal subnets started having problems until after Bob’s laptop started doing its thing.

Secure Your Thin-Client Network
Securing a thin client network is similar to securing a traditional network. You want to protect network systems from outside attack, inside attack, and system failures. Here are three tips specific to thin client-based networks:

1 It’s all about the servers. Not only do you need to protect the servers from viruses, hardware failure and so on, you also need to protect them from the users themselves. If a user hoses his or her Office installation in a traditional network environment, you have one computer (and one user) down; in a thin-client environment, a hosed Office install puts everyone out of commission.

2 Tame Internet Explorer! Running IE through a remote desktop session can open up your application server to hundreds, if not thousands, of security issues. Spyware, toolbars, popups, and all those other Internet annoyances can cripple a terminal server. Either tightly lock down IE on the server, or let your users browse locally.

3 Group Policy is your friend. Proper use of group policies, including the “User Group Policy loopback processing mode” policy, found in the Computer Configuration | Administrative Templates | System | Group Policy container, will make the job of securing terminal servers a whole lot easier.

Virus? Way!
Sure enough, all our virus issues started with Bob’s laptop. He brought it into the office, plugged it into the network, and that was it. Our servers were undaunted. They were used to being buffeted with attacks. But our thin-client workstations? They were never designed to be in a hostile environment. Their only purpose in life is to connect with a Remote Desktop Connection to a Terminal Server, all inside a private network.

OK, so we needed to get these thin-client machines working properly again. And we needed to do this in a way that minimized downtime. Thankfully, this was pretty easy. Remember, all the applications used by our customers are run from terminal servers, and all their data files are stored on servers, as well. Because the thin clients didn’t have anything installed on them other than the operating system and remote desktop clients, all we had to do was re-image them, a 10-minute process using our disk imaging software.

But, even at only 10 minutes a pop, with dozens of machines to re-image in disparate locations, it still took the better part of a day to complete the task. No data was lost, and our clients suffered little actual downtime.

Lessons Learned
But Bob’s little laptop adventure did make us think about revamping our security procedures—and our contracts. From now on, we’ll be stricter in enforcing our policies and making sure our clients are held accountable for following them.

For starters, we reworked our standard contract to include specific language about our customers’ responsibilities regarding their own equipment. But anyone who’s in this line of work knows that a contract is only good after the fact. Because our goal is to have satisfied clients, we also revamped our thin-client image to include anti-virus software. Yes, it costs us more; but, hey, our customers are worth it.

We’re also examining more sophisticated technology solutions to address this area, specifically in the areas of DHCP and network switching. If we can prevent a non-EON client device from getting a valid IP address in the first place, we’ve pretty much nipped the problem in the bud. Again, this increases costs, and at some point we’ll have to pass them on to the client.

EasyOffice Network
A typical EasyOffice Network thin-client arrangement. (Click image to view larger version.)

Are YOU Ready for Bob?
“OK, Kevin,” you may ask, “nice story and all, but what does this have to do with me?” Well, if you’re an IT pro responsible for maintaining computer systems (or even if you’re not), realize that you can’t just rely on technology to keep things running smoothly. This latest wave of viruses may not have caused you or your users any trouble, but eventually you’re going to have deal with a Bob of your own. And all the anti-virus software in the world won’t protect you. Are you ready for Bob?

comments powered by Disqus

Reader Comments:

Sun, May 30, 2004 Pete Anonymous

Why not simply configure terminal services to require IPSEC (KB315055), and use a client side firewall to allow only the IPSEC required ports and put some ACLs on your routers. That would at least reduce your risk a bit.
Also, a full-blown Windows XP OS is not a thin client not matter how you tweak and strip it. There is NO security without host-based security. I could walk into one of your client work areas, covertly jack a small cheap off-the-shelf wireless device into their network, and remotely have a field day with all of their systems and your servers. If I use a custom-made wireless box, with wireless power-on and power-off, you would probably never even know, let alone figure it out.

Wed, May 26, 2004 IR8 Toronto, Canada

Reading and then re-reading this article I would agree that it was a very BIG assumption regarding one's infrastructure to leave anti-virus etc. off the clients. These are after all full blown OS's despite the fact only a very limited set of functions are being used. In that light I would not blame the client entirely even though the onus of blame lies with the client. You can chalk this up to experience but if you were smart and had the $$$ I would suggest having your network/procedures/etc. reviewed by outside security people. Especially those conversant with ISO17799 (after all you don't want to repeat a mistake by assuming ...). BTW as to keeping unauthorized machines off the network a MS Enterprise CA type of setup with IPSec policies should help somewhat. And if one was paranoid, I would also randomize the local admin passwords on the clients with cusmgr and poll those clients via script for rogue local admins (use the WINNT provider - it's the easiest).

Mon, May 17, 2004 Rik QLD

XP SP2 around the corner, built in S/W 'FW' (OK so maybe not a fully featured one). TCPIP stack with options to restrict trafficports. How about logically separating the various networks from each other too with different network IP ranges. Does Email fit in here anywhere? Definitely lock down the DHCP and use some secondary measures - MAC spoofing can be too easy. But I still agree with those that say thanks for speaking up.

Thu, May 6, 2004 Anonymous Anonymous

Good article, good lesson.

Wed, May 5, 2004 Randy Spokane, WA

Excellent article! I read the comments from others and I think alot of people missed the point. The whole reason the article exists is to point out that you need to consider all angles when deploying information systems, including more secure environments like terminal services. Of course we all know you should put virus protection on any machine that has an OS on it. Kevin points this mistake out. It's still a good lesson. One thing to mention is that any future equipment purchases in a terminal services environment, to consider stateless thin clients. These devices have nothing more than firmware to allow for them to connect to a terminal server. Although not 100 percent impervious, there are not many, if not lately, any virus that attack such devices and would help to strengthen security and reliability at the client side.

Tue, May 4, 2004 TechGuy Anonymous

These are not Thin Clients, these are XP machines running Remote Desktop. This means they are PC with an OS. An OS that is very virus prone. Are there any Windows Update Patches on these machines? You would one keep up with that too. No anti-virus on a PC that is one of the biggest no-nos. I can’t believe that someone with such supposed expertise would have done this. We run some PCs with only Remote Desktop but they all have virus scan and windows updates are pushed to them all the time. We have found it way more cost effective to buy real thin clients with a UNIX kernel and they only do one thing Remote Desktop.

Tue, May 4, 2004 Anonymous Anonymous

Good article

Tue, May 4, 2004 Forest Anonymous

Thanks for exposing your vulnerablilties for others to learn. Not having a virus scanning software on PC's, even thin clients is a bad idea. But security is a compromise based on how much money and technology you can afford... or not afford to do without.

Tue, May 4, 2004 Nathan Anonymous

How about using MAC address security on your switches so that only approved MAC addresses can use ports.

Tue, May 4, 2004 Anonymous Anonymous

I'm sorry but to presume no Anti-Virus is required on "thin" clients is to violate a cardinal rule of security (no it's not thin - it's XP running TS sessions). Your weakest link was your client PCs. To assume it's not vulnerable ... Finally why not use PKI to validate devices grabbing DHCP leases?

Tue, May 4, 2004 Anonymous Anonymous

If you find a solution Let me know;)

Tue, May 4, 2004 Anonymous Anonymous

Nothing new

Add Your Comment Now:

Your Name:(optional)
Your Email:(optional)
Your Location:(optional)
Please type the letters/numbers you see above

Redmond Tech Watch

Sign up for our newsletter.

I agree to this site's Privacy Policy.