Introduction

In April, a CERT advisory
announced the discovery of two separate
buffer-overflow vulnerabilities in Snort,
a popular security-monitoring
tool used for detecting suspicious network activities.
This development was disturbing and ironic:
system administrators install and run
programs like Snort to improve security,
and don't often consider
the possibility that the tools themselves might be attacked
and exploited to create entirely new security holes.
It's therefore important to understand precisely what happened here,
especially since the same mechanisms used against Snort
could threaten other security tools.

In this article,
I will review the attacks that have been launched against Snort in the past,
as well as the recent (and more serious) buffer overflows.
In each case, I'll discuss the ways Snort developers
have responded to the attacks, and the
strategies system administrators can take
to minimize the risks.
Furthermore, I'll show that Snort's vulnerabilities extend
to other security-monitoring tools,
implying that we need to be careful when we use them, as well.
Finally, I'll summarize techniques to do just that: secure monitoring.

Evasion and Stealth Techniques

Snort is a network intrusion detection system, or NIDS.
It sniffs packets from the network, and analyzes them
using a database of pattern-matching rules.
These rules identify a wide range of suspicious or hostile activities,
such as misuse of network protocols, attempted buffer overflows,
denial-of-service attacks, port scans, and so on.
When such activities are detected,
Snort can be configured to log the network trace data
for more detailed examination,
and to send alerts so that system administrators will notice.

Since the main purpose of Snort is detection,
a primary goal of attackers is evasion.
If attacks can be structured so that they are overlooked by Snort,
then system administrators will be left with a false sense of security -- arguably a worse situation than if Snort had not been used at all.

The evolution of Snort has been driven largely
by the endless arms race between the attackers and the defenders.
The open source distribution of Snort
allows the bad guys to study the detection mechanisms and
devise ways to bypass them.
The Snort developers,
aided by reports from the user community and
examination of often widely available attack tools,
then respond by enhancing the algorithms, and
the cycle continues.
The history of innovation on both sides has been fascinating
and instructive.

When new malicious network activities are discovered,
pattern-matching rules are added to Snort's database.
These rules are stateless,
meaning that only one packet is examined at a time.
Snort also supports a variety of preprocessors
that track multiple packets,
and hence are stateful.
The modular design of Snort allows preprocessors
to be easily added or modified, and
more complicated evasion attempts
have been countered by improvements in the preprocessors.
The increasing focus on preprocessors for Snort
is reminiscent of the expansion of Linux firewall technology
from stateless ipchains
to stateful iptables.

Attackers frequently exploit IP fragmentation for evasion.
For example, the nmap
port-scanning tool provides an option
to use tiny, fragmented packets
that break headers for higher-level protocols
into pieces during transmission.
Snort's frag2 preprocessor reassembles the packets,
so the headers (and the payload) can be scanned all at once.
Similarly, many aspects of TCP can be abused by attackers, for example, by retransmitting packets with slight variations,
in the hope that they will be incorrectly handled by Snort.
The stream4 preprocessor catches these protocol anomalies
and ensures that Snort will interpret the packets
as a destination host would.
In addition,
stream4 reorders packets and reassembles entire TCP sessions,
which allows other preprocessors to easily access data as it
would have been reliably delivered by TCP.
These operations make stream4
one of the most important preprocessors
in Snort's arsenal.

A favorite technique for attackers
to defeat simple pattern-matching rules
is restructuring network data in many different,
but equivalent forms.
A variety of Snort preprocessors respond
by converting specific protocols
to standardized, canonical formats
for more reliable matching
by other rules and preprocessors.
The http_decode preprocessor
rewrites the URLs used by HTTP.
Similarly, the rpc_decode preprocessor
reassembles RPC data
that has been split into record fragments in various ways.
Attackers have also developed tools like
ADMutate
to generate polymorphic shell code for use in buffer overflows,
and the experimental fnord preprocessor
attempts to detect these;
this is an area of active research.

Some stealth strategies are based on timing.
For example, nmap can do very slow port scans,
in the hope that probes separated by long intervals
will not be noticed.
The portscan preprocessor detects these
by performing a statistical analysis of the traffic, and
the newer, experimental
portscan2 and conversation preprocessors
offer potentially improved detection.

There is really only one good defense
against these evasion techniques
for system administrators:
keep your Snort installation up to date.
Install new rules frequently: fortunately, this is easy,
since the rules database can be downloaded separately
from the Snort web site.
Upgrade Snort frequently too,
in order to obtain the latest preprocessors,
and be sure that the desired preprocessors
are enabled in the snort.conf configuration file.

Denial-of-Service Attacks

Denial-of-service attacks can launched directly against Snort.
These attacks are designed to disrupt the NIDS operation,
and are in some sense complementary to evasion tactics,
since they are certainly easy to detect.
A barrage of traffic will at least contribute to the load
on whatever machine is running Snort
and will consume system resources,
which makes this form of attack potentially more serious.
Snort is lightweight and efficient, and can therefore
tolerate high traffic rates,
but other NIDS programs can be overwhelmed,
and some have reportedly even crashed
when subjected to these assaults.

Attack tools like
stick and
snot
have been developed to construct random noise packets based on Snort rules.
These packets are designed
to match the patterns in the rules,
and hence produce a flood of bogus alerts.
The goal is to conceal some other genuine attack
within an avalanche of false alarms.
A cunning plan.

Snort retaliates by using the stream4 preprocessor:
the random noise packets can be readily identified (and ignored)
because they do not belong to a valid TCP session.
This is a consequence of the stateless nature of the Snort rules
used by the noise generators.

System administrators can best defend
against denial-of-service attacks
by running Snort on a separate, dedicated monitoring machine.
Any extra system resources that are consumed by Snort
during a denial-of-service attack
will therefore not affect network services
running on other machines.

If you are being bothered by stick-style attacks,
use Snort's -z est option
to limit alerts to only known, established TCP connections.
The stream4 preprocessor
must be enabled for this mode to work.

The Latest Threats: Buffer Overflows

The worst kind of attack to fear
is one that allows evildoers
to hijack the operation of Snort itself.
Unfortunately, two specific buffer-overflow vulnerabilities
have recently been discovered that show this is possible.

The security holes occurred in separate preprocessors:

rpc_decode:
An attacker could exploit a flaw
in Snort's computation of the size of packets,
and consequently overflow a buffer,
causing stack corruption.
This problem was found in March 2003,
and fixed by Snort 1.9.1.

stream4:
By using carefully crafted TCP sequence numbers,
an attacker could cause the overflow of an integer
used to check for buffer overruns,
leading to heap corruption.
This was exposed in April 2003,
and repaired in the latest Snort 2.0 release.

Both of these preprocessors are enabled by default,
and as we have seen,
stream4 is critical
for the stateful operation of other preprocessors,
as well as for protection against denial-of-service attacks.
You really wouldn't want to run Snort
with stream4 disabled.

The impact of these security holes is severe:
either can be used to cause Snort to crash, or
(even worse) to insert malicious code
where it will be executed by Snort.

The good news is that our knowledge of these vulnerabilities
and subsequent fixes are the result of security audits, and
Snort is an even more valuable tool if it is ironclad.
The modular design of preprocessors
encourages incorporation of new code,
which is an important advantage for Snort, but
the latest flaws demonstrate the need
for ongoing security audits,
especially for any preprocessors
that are enabled by default for production use.

The most obvious defensive strategy for system administrators
is to keep your Snort installation up to date, and
to monitor security advisories so you can apply patches
for any newly discovered holes.
Most system administrators are already doing this
for external network services -- the key here is recognize
that tools like Snort are also vulnerable,
and hence need attention too.

... but there is no reason to accept this risk.
Although Snort (like all network-sniffing programs) requires
superuser access to put interfaces into promiscuous mode (so they will act as wiretaps), Snort can and should run as an alternate user (and group, if appropriate)
after initialization is complete.
Use Snort's -u and -g options
to specify an unprivileged user and group,
chosen to minimize risk by ensuring
that only Snort's own log files are writable.

An even better strategy is to run Snort in a chroot jail.
Create a directory with all of the files that Snort needs,
remembering shared libraries and so on.
(Some experimentation will probably be necessary,
but it is worth the effort.)
Then specify the directory with Snort's -t option.
A chroot jail works best
when Snort runs as an unprivileged user -- this isn't an excuse to continue running Snort as root!

Finally, run Snort on a dedicated, untrusted machine.
This will help to contain the damage if Snort is ever compromised,
especially if the machine contains little or no state,
and can be easily rebuilt.

Vulnerability of Security Monitoring Tools

Buffer-overflow vulnerabilities are, unfortunately, still common,
as shown by even a casual examination of recent
CERT advisories,
and the techniques used against Snort
aren't especially original or innovative.
We often speak loosely of network sniffing programs (like Snort)
as "listening" to traffic, so it's easy to conclude that Snort's
recently discovered security holes are just another case of an
attack on a network service.
We've seen it all before, right?

Wrong.
In contrast to network servers like sendmail, BIND, and so forth,
Snort does not accept connections or receive datagrams
from client programs.
Instead, Snort passively
observes and analyzes network traffic,
and from a security standpoint
it is actually similar to other monitoring tools like
logwatch
or web log analyzers.
The network traffic plays the same role as a log file.

The easiest way to see this is to separate
the packet-capture and traffic-analysis functions
that are built into Snort.
Although Snort is often run in a configuration that does both,
this is not the only way to use the program,
nor is it necessarily the most secure mode of operation.

Packet capture can be performed
by a variety of network-sniffing programs other than Snort,
including
tcpdump
and
Ethereal.
This is relatively safe,
as long as the network data is not really analyzed,
but is simply dumped in "raw" form to trace files,
for example, using the command:

tcpdump -w tracefile ...

These tools all use the libpcap library
(provided with tcpdump),
and hence the trace files share a common format,
which is very convenient.

Traffic analysis can be done later,
possibly using a different program,
or on a different machine.
For example,
the tracefile saved previously
could be analyzed by Snort
using the command:

snort -r tracefile ...

Traffic analysis is considerably more dangerous than simple packet capture,
because (as demonstrated by the Snort buffer-overflow vulnerabilities),
malformed network data can cause program operations to go awry.

Network-sniffing programs like Snort
are often run on machines
with unconfigured interfaces, and
some system administrators even use
modified network cables
with severed transmit lines
to ensure that only passive observation can occur.
While these stealth techniques
undoubtedly offer some degree of protection
to the monitoring system as a whole,
the CERT advisory for Snort warns that:

"... it is not necessary for the attacker[s] to know the IP
address of the Snort device they wish to attack; merely sending
malicious traffic where it can be observed by an affected
Snort sensor is sufficient to exploit these vulnerabilities."

This is somewhat misleading, however -- the danger arises not from the observation
but rather from the subsequent analysis of the traffic,
and that could happen elsewhere.

The portability of network trace files amplifies the risks.
Traffic analysis is often performed on trusted machines,
deep within the perimeter protected by external firewalls.
Trace files can be sent by email, and
public collections are available on web sites
such as the Honeynet Project,
which offers periodic challenges
for study of attacks observed in the wild,
or the
Internet Traffic Archive,
which is more generic,
and unfortunately, hasn't been updated recently.

Although Snort performs more complex traffic analysis
than most other network-monitoring tools,
and hence offers more opportunities for attack,
it's important to recognize that
the other tools are also vulnerable.
Ethereal, for example, can perform TCP stream reassembly,
and even tcpdump's simpler packet printing code
is aware of many protocols.
Are buffer overflow bugs lurking in this code?

Furthermore,
other programs that analyze derived network data indirectly
could also be vulnerable.
The earlier analogy between network traffic and log files
offers a clue about how this could happen.
Could an attacker induce a network server
like sendmail or BIND
to emit a syslog message
that would be crafted
to compromise logwatch
when it later analyzed the system log?
Similarly,
could a web server be tricked
into recording an entry in its log file
that would subsequently confuse a web-log analyzer?
These kinds of attacks would be difficult
because they involve multiple interacting programs,
but they are certainly not impossible,
and they are conceptually similar
to the recently discovered Snort security holes.

Defensive Strategies for Secure Monitoring

When we recognize the threat of attacks on network monitoring tools,
and indeed any other tool that analyzes network data (even indirectly),
it is clear that we need to pay more attention
to writing and using these tools securely,
as we do for network servers.
Snort's buffer overflow security holes
were discovered during security audits -- similar audits for other network-sniffing programs
like tcpdump and Ethereal would be welcome news,
even (or especially)
if they lead to the discovery of other vulnerabilities.

Data from untrusted sources (that is, the network)
must be viewed with suspicion by security tools,
and subjected to sanity checks before use.
Programs need to verify the assumptions
(known as invariants)
that are critical for correct operation,
that is, packet headers don't lie about sizes,
integers won't overflow,
URLs aren't malformed,
printf-style format strings
contain no stray "%" characters, etc.
These invariants are different for each program,
so derived data that is considered sanitized by one program
must still be checked by other programs to be safe.
Perl provides a wonderful facility for tracking untrusted data:
taint mode.
Scripts like logwatch,
which is written almost entirely in Perl,
would benefit from use of Perl's -T option
to track tainted data read from log files.
The individual logwatch filter scripts
are analogous to Snort's preprocessor modules,
and they deserve a similar defensive posture.

Users of security tools and system administrators can protect themselves
first by recognizing vulnerable applications.
As we have seen, the most important principle is:
follow the data.
If it can be traced back to an untrusted source,
take steps to mitigate the risk.

The best way to minimize the potential impact of decisions
based on maliciously crafted data
is to analyze the data in a restricted environment.
This can be a dedicated, untrusted machine,
or a chroot jail.
At the very least, avoid running security tools as root --
choose an unprivileged user (and group, if appropriate)
that is only allowed minimal access to the data being analyzed,
and nothing else. The data often contains sensitive information
and requires protection from the general population anyway.
Separating the collection of data
from subsequent analysis steps
can make it easier to set up these safety nets.

Finally, be wary of automated processes
that are triggered by decisions
based on untrusted network data,
especially if the actions require privileged access.
For example, Snort provides experimental "flexible response" features
(fortunately disabled by default) for killing connections, and
a Perl script for dynamically modifying firewall rules,
both based on traffic analysis.
Imagine the havoc that could ensue
if an attacker managed to subvert or gain control
of these actions!

Conclusion

Monitoring can be done securely,
if proper attention is devoted
to the safe use of our available security tools.
By recognizing the vulnerabilities
and taking steps to run applications in a restricted environment,
we can derive the maximum benefit and enhance security,
without worrying (too much)
that our tools will be hijacked and used against us.

Robert G. Byrnes
is a software engineer at Curl Corporation, and has worked in the fields of networking, telecommunications, distributed computing, financial technology, and condensed matter physics.