Scanning for Rootkits

In an earlier article, Understanding Rootkits, we talked about how to guard against system intrusions and rootkits. Although it is best to have a server that nobody can break into, in real life every server connected to a network is a potential target.

If a server is compromised, it is equally important to realize this before it's too late and take measures to survive the attack.

Is My Server Compromised?

Usually, the first sign that a server might be compromised is simple anomalies in the behavior of the server. One of the more common anomalies one might notice is a change in how one or more of the core system utilities behave. For instance, a command-line switch to 'netstat' or 'ps', which you used to use without a problem everyday, might start returning an error message. The reason for this is that intruders replace these utilities with versions designed to hide their malicious activities. The utility they replace your original one with might be a different version, or it could have been compiled with different options, and as a result, it does not have the same options you are used to.

Another anomaly that should raise a big red flag is a change in your bandwidth-usage patterns. If you or your hosting company routinely monitors your bandwidth usage, you might notice an increase in the amount of traffic your server is pushing compared with your normal traffic patterns. This is usually caused by intruders using your server to distribute copyrighted software, commonly known as 'warez'. Remember that you might be the target of a legal action as a result of such activities on your servers.

Ideally, a server administrator should not wait until all the alarms go off before a server is checked for signs of compromise, because the less time an intruder has the opportunity to spend on a server, the less damage he or she will be able to inflict. (Although a very malicious intruder can potentially wipe out the whole system seconds after gaining root-level access.) For this reason, it is important to conduct server-security audits periodically, and to know as soon as possible when a server is compromised.

Port Scanning

An intruder will almost always try to install a backdoor on a server in order to have easy access to it at a later point. Because the local system utilities will probably be infected, they can't be trusted to detect the intruder. However, a port scan initiated from an external machine might reveal open ports that should normally be closed. The de facto standard port scanner in the Unix world is nmap. A command line as well as a GUI version is available. Without going into the details of the tool itself, here's how to scan all ports on a server for listening services:

This particular variation of the command scans all available ports (1 to 65535 inclusive) on 192.168.0.1. While the open ports above can be trojaned versions of ftp, ssh, or http, a real trouble sign is when you have ports with numbers higher than a few thousand where very few legitimate services normally listen to incoming connections. It is possible, and advisable, to periodically scan your whole network and identify any differences there might be between the outputs for different time periods.

RPM as a Security Tool

RPM (Red Hat Package Manager) is a popular packaging system used by some of the major Linux distributors including Red Hat, Mandrake, and SuSE. Every time an RPM package is installed, the package manager puts information about the package and the individual files the package contains into a central RPM database. Although not foolproof, this information can be used for performing quick and easy system audits. You should not rely on the RPM method when it claims the system is clean, but there's little need to doubt when it reveals the system binaries have been modified.

RPM records and tests the checksums of all files in a package, including files that are expected to change over time such as various configuration files and other files such as passwd, shadow, and group files. This makes it hard to conduct a full check and make sense out of the results. You can make this task easier by running the test on a handful of important core packages. These packages include net-tools, fileutils, util-linux, procps, psmisc, and findutils, among others.

Here's how to check the signatures on an RPM package:

rpm -V procps

If there are no changes to the package files, you will be returned to the command line. If there are changes you will get an output like this:

S.5....T c /root/.bash_profile

The first column shows what kind of a change has been made to the particular file listed on the last column since the package was originally installed.

There's also an easy way to test all It can be used to detect
changes in files not part of the base packages listed above. You can use rpm -Va > filename.txt to redirect the whole report to a file and try to identify any important changes using your own judgment.

You can also use the RPM utility to see if a particular file belongs to a package. This comes in handy when you find well-known binaries in unusual places.

rpm -qf /path/to/file will either tell you the file is not part of any package, or give you which package it is a part of. You can then run rpm -V on that particular package to detect changes.

Please keep in mind that you can only trust RPM when you get negative results. It is possible to modify the RPM binary, the operating system, or the RPM database so that the changes will not be detected. For this reason, it is a good idea to use the RPM method along with "chkrootkit".