Open source software security

Investigating Rogue Ports

Often times a system administrator will find open ports on a machine and wonder what they are for. It is easy enough to check to see the default use of a well known port (for instance, SMTP uses the default port 25), but this is no guarantee that the actual port observed is using the default service. It is very easy to utilize non standard ports to run services. SSH uses the default port 22, but it is a simple matter of changing the configuration file to allow SSH to listen on an alternative port. Given this situation, how can a sysadmin track down the use of a port?

Several methods exist for discovering the use of an open port. One of the best tools in a sysadmin's arsenal is NMAP port scanner. NMAP usually requires elevated privileges to run, but can be installed so that regular system users can utilize it. A Windows port exists as well and even includes a graphical front end. I find the command line version easiest to use on both Windows and Linux/Unix hosts. To start with we'll explore our target (in this case our local machine) to find what ports are open. Note that when NMAP is run on the local host it operates behind any firewall filters, so it may show ports that are in fact blocked from external access. To scan the local host use:

You can use the netstat command in a similar way. Using the '-l' to show listening services, '-p' to show the process identification number (PID), and '-n' flag to suppress lookups. The initial output will display relevant information:

You can omit the '-l' flag to inspect active connections. This output can be trusted more than the original NMAP scan because here we're looking at the information the machine is reporting from within. NMAP can be used to much more intelligently scan a host, however. Using the '-sV' flag NMAP will carry on a dialogue with the open ports and attempt to identify what service is actually running on the port (rather than simply reporting the default services that are usually running on the port). Remember that the scan above reported port 24 as follows:

24/tcp open priv-mail

NMAP shows that port 24 is open and that that port is usually used for priv-mail service. Lets try the more interactive scanning and see if we can determine the service that is actually running. Using the '-sV' flag NMAP will engage in a full protocol communication, examining sequences, not just banners, to attempt to determine the service:

You'll notice that port 24 has now been identified as running an SSH session. Our next step in determining the actual process that might be using the port is the lsof tool. Lsof stands for 'list open files'. We're going to use the '-i' flag, which lists files opened by a specific internet address. In this case we're going to use an abbreviation of the full form utilized for the address, which is 'protocol@[hostname|hostaddress]:[service|port]'. Using the following command:

# lsof -i :22

We can list the files utilized by the current process running on the local port 22. This gives results similar to:

You'll notice we can find the executable being used, the PID and even the user account that is running the process. The PID is especially useful because we can actually drop into the PID directory in /proc and examine what resources the process is using:

As you can see it is easy to spot the executable being run as well as several supporting components of the SSH process. We can also look at environmental variables set up by the process using the following:

This shows the executable and all the supporting libraries that it is using. These could be important if you want to verify the validity of the process (for instance if you're hunting for a trojan). Of course, if you suspect a trojan is on the machine you may also likely suspect a rootkit that could be subverting the operations of lsof and other programs on the machine. In the case of a legitimate service though you should have most of what you need. For one further step you can use RPM to figure out what package the executable belongs to. Using the query:

# rpm -qf /usr/sbin/sshd
openssh-server-4.6p1-1mdv2007.1

You can find out what package the SSH daemon belongs to, in this case openssh-server-4.6p1.

Conclusions

Tracking down a the actual usage of a port on a machine is a little more complex than simply consulting the /etc/services file. Although you can certainly check on ports using:

This only gives you the name of the commonly used protocol on the specific port. To be sure of what processes are using an open port you need to check a little more. Using tools like NMAP you can carry out this discovery remotely. If you have local shell access using lsof and the PID will give you more concrete information about the processes using the open port. Be careful though, as malicious rootkits could be subverting your investigative programs. If you're wary of hostile activity on your machine it is much safer and wiser to boot from trusted media to investigate a filesystem. Using a live CD or bootable linux distribution is often a good option in these circumstances.