You Got Hacked! Now What?
Hacks are a fact in a connected world. After discovering and expelling the intruders, you have to clean up their messes.
Sunday, 7:30 p.m.
You get a call from the help desk saying that
no one can log on to the domain. You connect to the VPN and give it a
look. For some reason, Active Directory is refusing requests. You start
investigating. It turns out the server is out of disk space. How can this
be? After all, you’ve just installed a larger hard drive.
You frantically start freeing disk space on the box by combing through
Windows Explorer, hoping to find the files using all the space. You move
the page file to another drive. You delete old service pack backups and
all temp files.
Then you stumble across a folder using 35 GB of space. Looking inside,
you find thousands of MP3s and hundreds of games. Checking the files’
properties to find the owner reveals that administrators appear to own
the data. How is this possible?
Find Out What Really Happened
It turns out that someone’s been using your server as a storage
bin for his or her music and games. The question: Is this a hack? Or is
there another explanation?
That’s what you have to find out first. Remember: Not every situation
that looks like a hack actually is. For instance, if you find a copy of
pcAnywhere on your server and disconnect it from the network because you
think it was hacked, you may do more harm than good. Maybe it’s there
because your Web developer added it to do a mission-critical upgrade from
home over the VPN. Note that I’m not recommending putting remote control
software on your Web servers; I’m just saying to get all the facts before
you take action and be sure to communicate with everyone involved.
Here are some of the warning flags that you may have been hacked:
- Logs showing repeated failed login, FTP and telnet attempts.
- Finding software you didn’t place on the server like Symantec’s pcAnywhere,
or the open source Rconsole or AT&T’s Virtual Network Computing (VNC).
- Running out of disk space on servers that shouldn’t be full.
- Constant virus outbreaks despite having anti-virus software on all
of your machines.
- Periods of high network usage at odd times.
Get a Mop and a Broom
If you discover that your network’s been compromised, take immediate
corrective action. That means first locking out the attacker. The easiest
way to do that is by disconnecting your computer from the network. Then
comes the process of finding out where the hacker’s been and cleaning
up the messes he or she has made. Follow the footprints, which may have
been left in the some of these places:
- New user accounts.
- Additions to group membership.
- Changed user rights.
- Programs configured to start automatically.
- Altered, added or deleted file, share and registry permissions.
- Altered, added or deleted Web permissions.
- Added files and features.
We’ll go through these areas one by one. As you progress, remember to
make copies of all logs and to document everything that you find. In addition
to electronic notes, I recommend keeping a physical notebook for all your
servers; this way your notes can’t be easily erased.
Before doing anything else, change the passwords for all user accounts
on the machines in question, starting with the administrator account.
If a hacker has gained administrative access to your system, assume he
or she has compromised all the system’s passwords. Tons of easily accessible
tools exploit local and domain databases once you have administrative
Look for New User Accounts and Additions to Group
Most companies change their passwords every 30 to 90 days. This
makes a stolen password good for a limited time only, so the first thing
hackers will do is create user accounts. These accounts must be deleted.
You should have a copy of your user list in your security notebook. (If
not, make one.) Print out a new list and compare the two.
Verify all new accounts, especially accounts that appear to be service
accounts. Hackers create user accounts that appear as service accounts
(such as iusrcomputername instead ofiusr_computername or svcveritas),
hoping administrators will overlook them.
The first groups to check are the built-in groups like Administrators
and Server Operators. If an attacker created user accounts, chances are
good he or she gave one of them elevated privileges. Remember to document
everything as you go.
Check for Programs Configured to Start Automatically
Always look for programs that are set to start automatically. These
are dangerous, as you could be running a program and not even know it.
Always check the following places:
- AT Scheduler and Task Scheduler
- Startup Folders
Verify that all scheduled jobs should be there and are configured correctly.
Anytime you schedule a job, note the details, including date and business
reason for implementing the job, in your server notebook. What appears
as a normal scheduled task could be a password-gathering tool scheduled
to run every night at midnight under the system’s credentials.
Always check your services. Check the account they’re using and what
executable they’re starting. Manually start the service to verify that
it works correctly. Hackers will replace the executable used by the service
with a new executable. At first glance, everything appears to be normal.
However, when the service starts, the damage begins.
Programs can also be configured to start when a user logs in or opens
a certain program. Be sure to check the startup folders for all of the
profiles on your system. Also check out the following keys in the registry
to look for run entries:
Check File, Share, and Registry Permissions
Always assume a hacker has given himself or herself permissions
to everything. Check your local file system. Look for new shares and check
permissions on existing shares. If a hacker has rights to a user’s directory,
he or she could use it to upload harmful scripts or programs. Thus, when
a user clicks on that new file in the directory, the hacker’s tool is
launched. Also check registry permissions, especially the keys listed
above. If the hacker sets these keys to Everyone Full Control, he or she
can easily configure them to run programs later.
Check for Files and Features Added by the Hacker
Look for any files that shouldn’t be on your computer. It’s not
a bad idea to print your PC or server’s directory and store it in your
notebook. This way you can easily print out another directory (dir > lpt1)
and compare the differences. Even better is to use a third-party program
such as Tripwire (www.tripwire.com) to check data integrity. Tripwire
creates a snapshot of your system and stores it in a database. Then, later,
you can run another snapshot and compare the changes.
Also check for normal features of Windows that may have been installed
by the attacker such as IIS, RAS, FTP and Telnet Server. If these features
aren’t needed, they shouldn’t have been installed; uninstall them. Hackers
always try to leave themselves several back doors into the system. What
easier way to do that than with the built-in features provided by Microsoft?
Should I Rebuild?
From a security standpoint, it’s always better to rebuild your
system and restore your data from clean backups. This way you know your
server’s clean. I recommend using a third-party imaging product like Norton’s
Ghost or PowerQuest DriveImage to create an image of all of your servers
when they’re built. Then you can rebuild a server in less than 10 minutes.
Minimizing Successful Attacks
There’s no way to guarantee you’ll never be hacked again. There
are only best practices you can follow to limit the likelihood of being
hacked. Apply all service packs and security updates; most compromised
systems aren’t fully patched. Install virus protection. Lock down your
computers so that only the services needed are available. You may want
to install a third-party firewall product on your server to help with
this. Thoroughly locking down a server can be a time-consuming process
but probably less so than cleaning up after an attack.