Thin Clients, Fat Heads

I get a call from one of my customers. “There’s something wrong with
the network. Everything is really slow.” I tell him I’ll look into it
right away. As I start to check it out, I get a call from another client.

“We’re really crawling here. Is there something wrong?” I tell her that we’re already checking it out, and we should have things back to normal soon.

Another client, then another, calls, each with the same complaint. One
more comes in. This time my customer offers an explanation: “I think there’s
a virus on the network!”

Virus? No Way!
A virus? There’s no way we could have a virus. Yes, this was all happening
in the midst of a serious virus outbreak, and yes, many large corporations
had already been infected, but not our systems. Let me explain.

My company, EasyOffice Network (EON), provides clients with comprehensive, thin-client–based, fully-managed IT solutions. We supply, build, configure, manage and retain ownership of all computer systems used by our clients. I’ve built most of these IT infrastructures from the ground up and have employed best practices in how we manage everything from desktop machines to file severs to routers to firewalls. I’ve been doing this for a long time and am proud of a long track record of keeping my clients’ systems in good working order.

We’re able to do this thanks to several key elements of our service model:

The capabilities of Terminal Services in a Windows 2000/XP environment.

The licensing agreement we have with Microsoft that allows us to rent
the use of Microsoft software to our clients.

The ability to remotely administer the computers and network systems in
place at our client sites.

The fact that our high-end data center systems are used to support several
clients.

The proprietary expertise we incorporate into our system configurations.

Because we use a server-based computing model with thin-client workstations, the technical requirements for one of our IT solutions are quite different than those for a traditional office network environment. To illustrate, let’s look at a typical EON solution.

Our customer, Little Guy Industries (LGI), has two locations: their main
office in Los Angeles and a smaller branch office in San Jose. We put
a file server, an application server, a SQL server and a domain controller
in the Los Angeles office. In San Jose, we placed a domain controller
(which also serves as a file server) and an application server. The application
servers run Terminal Services. All servers are configured with real-time
virus-checking software.

Thin Is In
For the clients, we use LGI’s existing Dell computers—a mixture of older
Celeron- and Pentium III-based machines. We rebuild all the Dells with
a standard Windows XP Professional configuration. These machines are used
to do one thing, and one thing only: run the Remote Desktop Client to
connect to an application server. They won’t be used for Internet browsing;
they can’t even get out to the Internet! As such, we don’t install any
kind of virus scanning software on clients—remember, all program activity
takes place on the application server through a terminal services session;
nothing actually takes place on the client machine.

The offices are each connected to the Internet via DSL, protected by industry-standard firewall appliances. Interoffice network communication is handled by a hardware-based VPN. Only the servers have access to the Internet, and then only through a firewall.

Now that you have an idea about the typical EON environment, let’s see what happens when trouble’s afoot. Suppose it’s a Tuesday morning, and word of a new virus threat makes the rounds. One of our standard procedures is to check the firewall traffic logs to see what kind of packets are hitting our networks from the outside, and sure enough, we find evidence of attempted attacks. We contact our upstream Internet provider (if they haven’t already contacted us first) and ask them to keep tabs on the traffic going through their networks.

We also check our server logs for evidence of virus activity. They show lots of attempts but no infections. Thanks to the faithful real-time virus scanning software installed on each server, no one suffers any ill effects from this attack.

We’ve survived one of the more destructive and widespread virus attacks
that have come down the pike in a long time. Feeling ever so smug in our
prowess over infectious bytes, we send an e-mail to all our clients detailing
how severe this virus attack was to many Fortune 500 corporations—and
emphasizing how not one of our clients suffered even as much as a sniffle.

OK, so we hold off on sending the “We beat the virus” e-mail to our customers and start troubleshooting. Could these problems be related to the virus outbreak? No, we think, just an untimely coincidence. There’s got to be something we can find to explain these problems.

After observing heavier than normal network traffic, we find that there is, indeed, evidence of virus activity on our internal networks. So I ask my techs to pore through the firewall logs again. But this time, I tell them to look at activity from inside our NAT-compliant, firewall-protected, private networks. “But the only way into our private networks is through public Internet gateways, and we’ve already proven that nothing came through any of our firewalls,” my senior tech points out.

I repeat my instructions. He be-grudgingly complies with my request. Later that day I get an e-mail from my tech. He tells me that he was able to narrow down where all this virus mayhem began—at one of our client sites (let’s refer to this client as “Bob”), from inside the private network. As I’m contemplating how this could have happened—how a virus could just appear in a private network without any trace of it coming through the firewall—I get another e-mail. This one is from Bob.

He tells me that he thinks his laptop is infected with a virus and wants to know what to do about it. “What laptop?” I think to myself. We never sold or discussed a laptop computer with this client. I e-mail him back. I tell him how to discover the MAC address of his NIC and give this bit of information.

I pass this info on to my tech, and he confirms that Bob’s laptop is
the source of all this extra network traffic. Further analysis reveals
that none of our other internal subnets started having problems until
after Bob’s laptop started doing its thing.

Secure
Your Thin-Client Network

Securing a thin client network is similar to securing a traditional network. You want to protect network systems from outside attack, inside attack, and system failures. Here are three tips specific to thin client-based networks:

1
It’s all about the servers. Not only do you need to
protect the servers from viruses, hardware failure and
so on, you also need to protect them from the users
themselves. If a user hoses his or her Office installation
in a traditional network environment, you have one computer
(and one user) down; in a thin-client environment, a
hosed Office install puts everyone out of commission.

2
Tame Internet Explorer! Running IE through a remote
desktop session can open up your application server
to hundreds, if not thousands, of security issues. Spyware,
toolbars, popups, and all those other Internet annoyances
can cripple a terminal server. Either tightly lock down
IE on the server, or let your users browse locally.

3
Group Policy is your friend. Proper use of group policies,
including the “User Group Policy loopback processing
mode” policy, found in the Computer Configuration |
Administrative Templates | System | Group Policy container,
will make the job of securing terminal servers a whole
lot easier.

Virus? Way!
Sure enough, all our virus issues started with Bob’s laptop. He brought
it into the office, plugged it into the network, and that was it. Our
servers were undaunted. They were used to being buffeted with attacks.
But our thin-client workstations? They were never designed to be in a
hostile environment. Their only purpose in life is to connect with a Remote
Desktop Connection to a Terminal Server, all inside a private network.

OK, so we needed to get these thin-client machines working properly again. And we needed to do this in a way that minimized downtime. Thankfully, this was pretty easy. Remember, all the applications used by our customers are run from terminal servers, and all their data files are stored on servers, as well. Because the thin clients didn’t have anything installed on them other than the operating system and remote desktop clients, all we had to do was re-image them, a 10-minute process using our disk imaging software.

But, even at only 10 minutes a pop, with dozens of machines to re-image
in disparate locations, it still took the better part of a day to complete
the task. No data was lost, and our clients suffered little actual downtime.

Lessons Learned
But Bob’s little laptop adventure did make us think about revamping our
security procedures—and our contracts. From now on, we’ll be stricter
in enforcing our policies and making sure our clients are held accountable
for following them.

For starters, we reworked our standard contract to include specific language about our customers’ responsibilities regarding their own equipment. But anyone who’s in this line of work knows that a contract is only good after the fact. Because our goal is to have satisfied clients, we also revamped our thin-client image to include anti-virus software. Yes, it costs us more; but, hey, our customers are worth it.

We’re also examining more sophisticated technology solutions to address
this area, specifically in the areas of DHCP and network switching. If
we can prevent a non-EON client device from getting a valid IP address
in the first place, we’ve pretty much nipped the problem in the bud. Again,
this increases costs, and at some point we’ll have to pass them on to
the client.

Are YOU Ready for Bob?
“OK, Kevin,” you may ask, “nice story and all, but what does this have
to do with me?” Well, if you’re an IT pro responsible for maintaining
computer systems (or even if you’re not), realize that you can’t just
rely on technology to keep things running smoothly. This latest wave of
viruses may not have caused you or your users any trouble, but eventually
you’re going to have deal with a Bob of your own. And all the anti-virus
software in the world won’t protect you. Are you ready for Bob?