12.4 Managing Services Securely

Once you have deployed a Unix server on a network, it is important
that you manage it securely. You
should periodically monitor your server and the network for potential
problems or abuse. Most network topologies provide three locations
for monitoring:

You can monitor on the hosts themselves, either by monitoring the
packets as they enter or leave the system, or by monitoring servers
that are running on the system.

You can monitor the local area network. If you have an Ethernet hub
on your network, you can monitor by attaching another computer to the
hub that is running a network monitor or packet sniffer. If you have
a switched network, you will need to create a mirror
port or monitor point.[40]

[40] Typically, mirror or monitor ports can be created on managed
switches, but cannot be created on low-cost switches.

You can monitor information entering or leaving your network at the
point where your network connects to other networks.

Most monitoring involves one or at most two of these systems; the
most secure networks combine all three.

You may also wish to employ other methods, such as network scanning,
to detect vulnerabilities before attackers do so.

12.4.1 Monitoring Your Host with netstat

You can use the netstat command to list all of the
active and pending TCP/IP connections between your machine and every
other machine on the Internet. This command is very important if you
suspect that somebody is breaking into your computer or using your
computer to break into another one. netstat lets
you see which machines your machine is talking to over the network.
The command's output includes the host and port
number of each end of the connection, as well as the number of bytes
in the receive and transmit queues. If a port has a name assigned in
the /etc/services file,
netstat will print it instead of the port
number.

Normally, the netstat command displays Unix
domain sockets in addition to IP sockets. You can restrict the
display to IP sockets only by using the -f inet
(or -A inet in some versions) option.

The netstat program displays only abridged
hostnames, but you can use the -n flag to
display the IP address of the foreign machine, and then look up the
IP address using another tool such as host. This
is probably a good idea anyway, as IP addresses are harder to hide
and the hostname lookup itself may alert an attacker that you are
monitoring them.

The first line indicates an IMAP connection with the computer called
KITCHEN. This connection originated on port 1922
of the remote machine. The second line indicates an in-process HTTP
connection with the host z06.nvidia.com. The
third line is an HTTP download that has entered the TIME_WAIT
state.[41] The
subsequent lines are various TCP connections to and from other
machines. These lines all begin with the letters
"tcp4", indicating that they are
TCP servers running on top of IPv4.

[41] RFC 793 defines the TCP states, in case
you're interested. TIME_WAIT means that the local
process is "waiting for enough time to pass to be
sure the remote TCP received the acknowledgment of its connection
termination request."

With the -a option, netstat
will also print a list of all of the TCP and UDP sockets to which
programs are listening. Using the -a option will
provide you with a list of all the ports that programs and users
outside your computer can use to enter the system via the network.

The lines in the middle of the listing that end with
"LISTEN" indicate TCP servers that
are running and listening for incoming communications. Clearly, this
computer is running a lot of services. Some of the ports on which
this computer is listening have no matching name in the
/etc/services file, and are therefore listed
only by their numbers (i.e., 1022, 1023). This should be a cause for
further investigation. Unfortunately, netstat
will not give you the name of the program that is listening on the
socket.

To determine which process is listening on port 1022 and port 1023,
we can use the lsof[42]command:

[42]lsof stands for "list open
files"?on Unix systems, network connections,
like nearly everything else, can be treated as a kind of file.
It's available from ftp://vic.cc.purdue.edu/pub/tools/unix/lsof/.
If it's distributed with your operating system, make
sure that nonprivileged users can't use it to see
files and devices owned by other users; if they can, rebuild it from
the source code.

The rpc.statd program has a history of security
vulnerabilities (See http://www.cert.org/advisories/CA-2000-17.html).
The mountd program is part of the NFS system; it
also has a history of security vulnerabilities (see http://www.cert.org/advisories/CA-1998-12.html.
Fortunately, all known vulnerabilities in both of these programs have
been fixed.[43]
Perhaps even more importantly, the r2 computer
is protected with a host-based firewall that blocks all connections
to port 1022 and 1023 that originate on the Internet.

[43] Of course, we don't know
about the unknown problems!

We can verify that r2's
firewall is in proper operation
by probing the system with the nmap[44] command from an outside host:

[44]nmap is a powerful port scanner that is
distributed with many Unix operating systems, or available from
http://www.insecure.org/nmap.

So it seems that the firewall is properly working. This is always a
good thing to verify.

12.4.1.1 Limitation of netstat and lsof

There are many
ways for a program to be listening for commands over the Internet
without having the socket on which it is listening appear in the
output of the netstat or
lsof commands. Attackers have used all of these
techniques to create programs that wait for an external trigger and,
upon receipt of the trigger, carry out some predetermined action.
These programs are typically called zombies.[45] They can be
used for many nefarious purposes, such as carrying out remote
denial-of-service attacks, erasing the files on the computer on which
they are running, or even carrying out physical attacks (through the
use of control equipment that may be connected to the computer).

[45] Not to be confused with
defunct
processes, which are listed by the ps command as
type "Z" (zombie). Defunct
processes are usually the result of sloppy programming or unusual
system events, but don't usually represent a
security concern in and of themselves.

Here are a few of the ways that a zombie might be triggered:

The zombie might join an Internet Relay Chat room. If a special
password is typed in the room, that might be a signal to the zombie
to attack a remote machine.

The zombie might periodically probe a web page or make a specific DNS
request, and carry out an attack depending upon the response that it
receives.

The zombie might listen to the raw Ethernet device in promiscuous
mode and initiate an attack when a command is sent to another
computer that is on the same local area network. To further confuse
matters, the zombie might initiate its attack using packets that have
forged source addresses so that they appear to come from the other
computer, when in fact they do not.

The zombie might run inside the system's kernel and
be activated if an otherwise legitimate IP packet is received that
has a particular IP or TCP option set. The trigger might be a
sequence of values in the time-to-live field.

Instead of listening to the network, the zombie might instead probe
the computer's log files. The zombie might be
activated if a particular URL is fetched from the web server, or if a
particular firewall rule is exercised.

Any of the above triggers might be set up so that they work only if
they are exercised at a particular time.

To complicate matters even further, your system's
kernel or utilities might be patched so that the process or TCP
sockets associated with the zombie do not appear in program
listings.

12.4.2 Monitoring Your Network with tcpdump

You can use the
tcpdump command to watch packets as they move
over your network. This command is included as a standard part of
most Unix systems. The tcpdump command attaches
to an interface and reads all the packets, optionally displaying them
in your terminal window or writing them into a file. You can also
specify a filter?either simple or complex?using the
tcpdump filter language.
( Solaris systems come with a program
called snoop that performs much the same
function; you can also download a copy of
tcpdump to run with Solaris systems.)

Specifies the number of packets that should be recorded. By default,
tcpdump runs until you type Ctrl-C.

-F file

Specifies a file that contains the filter to use

-s snaplen

Specifies the number of bytes of each packet that
tcpdump should record. Normally,
tcpdump records only the first 68 bytes of each
packet. This is useful for traffic analysis, but is not useful if you
are interested in viewing the contents of the packets. Specify
-s 4096 to be sure that you are getting complete
packets.

-w file

Specifies that tcpdump should write its packets
to a file, rather than displaying them on the terminal.

-r file

Specifies that tcpdump should read packets from
a file, rather than from the interface. This can be useful when
analyzing dump files that have been previously created.

-n

Specifies that tcpdump should not convert IP
addresses to hostnames. Normally, tcpdump
performs a reverse name lookup on each IP address that it sees.
Unfortunately, this can slow your system dramatically. It can also
tell an attacker that you are monitoring the network.[46]

[46] An attacker can send packets to a nonexistent host with a
source IP address that is not used for any other purpose. If a
reverse DNS query is performed for that IP address, the attacker
knows that somebody is sniffing the traffic on the network.

-v, -vv, -vvv, -vvvv

Prints increasingly verbose amounts of information.

The tcpdump command is a great tool for learning
what is going over your network. It can also be a great tool for
violating people's privacy because people often send
information over a network that is confidential, but without using
encryption to ensure that it stays confidential. Remember that
tcpdump captures packets moving over the
network, whether or not they are originating or destined for your
host. (Also remember that others on the network may also have access
to tcpdump!)

Be aware that some local, state, and national laws may apply to the
monitoring of network traffic. Wiretapping, email privacy laws, and
even espionage statutes might come into play. Thus, if you intend to
monitor network traffic, you should obtain the advice of competent
legal counsel before doing so. Also note that your users may view
excessive monitoring or storage of logs to be a serious invasion of
privacy. Monitoring should be kept to a minimum, and you should
ensure that your users understand that you are doing it?finding
out by accident will do little to build trust and cooperation with
your user population.

For example, to display the next 16 packets moving over a network,
you might use this command:

Significantly more sophisticated analysis can be performed using
snort, an open source intrusion detection
system, or using a commercial network forensics analysis tool, which
will monitor all of the traffic that passes over the network,
reassemble the TCP/IP streams, and then analyze the data. Systems
like these not only monitor packets, but also analyze them and
compare them to rules designed to detect intrusion attempts. For
example, here's some of the output of
snort:

This log shows several intrusion attempts on the host 10.13.6.226,
including a scan for an open SOCKS proxy (which can be abused by
spammers), an attempt to overflow the FTP server's
buffer with a long USER command, an attempt to access the
formmail CGI script (which has been abused by
spammers), an attempt to overflow the SMTP server's
buffer with a long RCPT TO command, and an attempt to access one of
the private directories used by web sites managed with Microsoft
FrontPage.

12.4.3 Network Scanning

In
recent years, a growing number of programs have been distributed that
you can use to scan your network for known problems. Unfortunately,
attackers can also use these tools to scan your network for
vulnerabilities. Thus, you would be wise to get one or more of these
tools and try them yourself, before your opponents do.

There are several kinds of network scanners available today:

Commercial network scanners

These programs, typified by the Internet Security
Scanner (ISS), systematically probe all of the ports of every host on
your network. When a TCP connection is accepted, the scanner examines
the resulting banner and looks in its database to see if the banner
corresponds to a version with known security vulnerabilities. The
advantage of these systems is their large database of
vulnerabilities. The disadvantage of these systems is that they are
frequently licensed to scan a limited set of IP addresses, and those
licenses can be quite expensive.

Freeware security scanners

These programs are similar to commercial scanners, except that they
are usually open source tools that are freely available. An early
example of such a program was SATAN; a popular and
up-to-date one is Nessus. The advantage of these systems
is their cost. The disadvantage is that their databases are can be
limited when compared to the commercial offerings, though Nessus in
particular has a very extensive database.

Security Scanners

Before
1990, people didn't like to talk about security
vulnerabilities. It was difficult to find published papers describing
vulnerabilities: how they came to be, how they were found, and how to
fix them. To find out about problems one had to get a
"friend of a friend" introduction
to one of the closed mailing lists, or else get a copy of a closely
guarded technical report detailing problems. The result was that most
system administrators didn't know how to secure
their systems, and researchers couldn't perform
organized studies of vulnerabilities.

That changed when Dan Farmer wrote the COPS
scanner under the direction of Gene Spafford at Purdue, then released
it on the Usenet. Suddenly, there was a widely available system that
scanned for vulnerabilities that were known to several experts and
also gave recommendations on how to patch them. (It was an explicit
design principle of the project to provide information about how to
fix the problems rather than exploit them.)

COPS has been downloaded, modified, used, extended, emulated, and
studied by hundreds of thousands of people in the intervening years.
It succeeded in many ways beyond the authors'
expectations. However, there are at least two sets of thoughts that
we pose from this experience that are interesting to contemplate:

As recently as early 2002, Spafford is still getting requests for
help with configuring COPS and interpreting the output! The software
is 12 years old. What does it say about sysadmins that they still
expect this software to work? What does it say about the coding and
design used that parts of COPS still do work?
What does it say about coding practices that many versions of Unix
and Linux are still shipped with vulnerabilities that COPS
identifies?

Had Farmer and Spafford taken out software patents on the scanner and
some of its subsystems, the patents would still be in force and the
authors might be very wealthy. However, would the overall state of
security on the Internet be better today as a result? Would there be
fewer security tools developed and in use? Maybe people would be less
tolerant of flaws in deployed systems if scanning tools were less
available? (Contrast this with what patents on public-key
cryptography did to security.) Does this imply anything about
software patents?

As with most other aspects of Internet technology, it is amazing to
see how much has changed in only 10 or 15 years. And to think what
the next decade will bring.

Freeware port scanners

These programs
typically scan for open ports and services, but they do not include a
database of known vulnerabilities. One example of these programs is
nmap, the network
mapper, which can perform a large number of different kinds of scans
and is also remarkably good at identifying the operating system of a
remote machine based on the way it responds to scanning. A typical
output from nmap was shown earlier, in Section 12.4.1.