Search

swatch: Automated Log Monitoring for the Vigilant but Lazy

Previously the Paranoid Penguin has
pondered a plethora of powerful programs pursuant to protecting
people's PCs from pernicious punks. [The right to excessive
alliteration revocable at any time—Ed.] One important feature
these tools share is logging; just as important as keeping system
crackers out is knowing when they've tried to get in. But who's got
the time or attention span to sift through scads of mostly
innocuous log files on every system they administer, every single
day?

swatch (the “Simple
WATCHer”) does. swatch, written
100% in Perl, monitors logs as they're being written to and takes
action when it finds something you've told it to look for. This
simple, flexible and useful tool is a must-have for any healthily
fearful system administrator.

Installing swatch

There are two ways to install swatch. First, of course, is
via whatever binary package of swatch, if any, your Linux
distribution of choice provides. The current version of Mandrake
has an RPM package of swatch, but none of the other more popular
distributions (i.e., Red Hat, SuSE, Slackware or Debian) appear to
include it.

This is just as well, though, because the second way to
install swatch is quite interesting.
swatch's source distribution,
available from
www.stanford.edu/~atkins/swatch,
includes a sophisticated script called Makefile.PL. The script
automatically checks for all necessary Perl modules and uses Perl
5's CPAN functionality to download and install any needed modules;
it then generates a Makefile that can be used to build
swatch.

After you've installed the required modules, either
automatically from swatch's Makefile.PL script or manually (and
then running perl Makefile.PL), Makefile.PL should return the
following:

Once Makefile.PL has successfully created a Makefile for
swatch, you can execute the following commands to build and install
it:

make
make test
make install
make realclean

The make test command is optional
but useful; it ensures that swatch can properly use the Perl
modules we took the trouble to install.

swatch Configuration in Brief

Since the whole point of swatch is to simplify our lives,
configuring swatch itself is, well, simple.
swatch is controlled by a single
file, default $HOME/.swatchrc. This file contains text patterns in
the form of regular expressions you wish swatch to watch for. Each
regular expression is followed by the action(s) you wish swatch to
take whenever it encounters that text.

For example, suppose you've got a web server, and you want to
be alerted any time someone attempts a buffer-overflow attack by
requesting an extremely long filename. By trying this yourself
against the web server while tailing its /var/apache/error.log, you
know that Apache will log an entry that includes the string “File
name too long”. Suppose further that you want to be e-mailed every
time this happens. Here's what you'd need to have in your .swatchrc
file:

As you can see, the entry begins with a “watchfor”
statement, followed by a regular expression. If you aren't
proficient in the use of regular expressions yet (you are
planning to learn regular expressions, aren't
you?), don't worry: this can be as simple as a snippet of the text
you want swatch to look for, spelled out verbatim between two
slashes.

swatch will perform your
choice of a number of actions when it matches your regular
expression. In this example we've told swatch to send e-mail to
mick\@visi.com, with a subject of BufferOverflow_attempt. Note the
backslash before the @ sign; without it, Perl will interpret the @
sign as a special character. Note also that if you want spaces in
your subject line, each space also needs to be escaped with a
backslash, e.g., subject=Buffer\ Overflow\
attempt. Actions besides sending e-mail include those
seen in Table 1.

For more details on configuring these and the other actions
swatch supports, see the swatch(1) man page.

Let's take our example a step further. Suppose, in addition
to being e-mailed about buffer-overflow attempts, you want to know
whenever someone hits a certain web page, but only if you're logged
on to a console at the time. In the same .swatchrc file, you'd add
something like this:

watchfor /wuzza.html/
echo=red bell 2

The event will then cause a beep and print to the console.

It's important to note you will only see these messages and
hear these beeps if you are logged on the console in the same shell
session from which you launched swatch. If you log out to go get a
sandwich, when you return and log back in, you will no longer see
messages generated by the swatch processes launched in your old
session, even though those processes will still be running.

When in doubt add either a “mail” action or some other
non-console-specific action, e.g., an “exec” action that triggers
a script that pages you. Unless, that is, the pattern in question
isn't critical.

Alert readers have no doubt noticed that the scenario in the
previous example will work only for Apache installations in which
both errors and access messages are logged to the same file. We
haven't associated different expressions with different watched
files, nor can we do so. But what if you want swatch to watch more
than one log file?

No problem. While each .swatchrc file may describe only one
watched file, there's nothing to stop you from running multiple
instances of swatch, each with its own .swatchrc file. In other
words, .swatchrc is the default but not the required name for
swatch configurations.

To split the two examples into two files, therefore, you'd
put the lines in the previous simple .swatchrc entry into a file
called, say, .swatchrc.hterror, and the lines in the previous
watchfor entry into a file called .swatchrc.htaccess.

Advanced swatch Configuration

So far we've only considered actions we want to be triggered
every time a given pattern is matched. There are several ways we
can control swatch's behavior with greater granularity,
however.

The first and most obvious way is to take advantage of the
fact that search patterns take the form of regular expressions.
Regular expressions, which really constitute a text-formatting
language of their own, are incredibly powerful and responsible for
a good deal of the magic that Perl,
sed,
vi and many other UNIX utilities
can do.

It behooves you to know at least a couple of “regex”
tricks, which I'll describe here. Trick number one is called
alternation, and it adds a “logical or” to your regular
expression in the form of a “|” sign. Consider this regular
expression:

/reject|failed/

This expression will match any line containing either the
word “reject” or the word “failed”. Use alternation when you
want swatch to take the same action for more than one pattern.

Trick number two is the Perl-specific regular expression
modifier “case-insensitive”, also known as “slash-i” since it
always follows a regular expression's trailing slash. The regular
expression /reject/i matches any line containing
the word “reject”, whether it's spelled “Reject”, “REJECT”,
“rEjEcT”, etc. Granted, this isn't nearly as useful as
alternation, and in the interest of full disclosure, I'm compelled
to mention that slash-i is one of the more CPU-intensive Perl
modifiers. However, if despite your best efforts at log-tailing,
self-attacking, etc., you aren't 100% sure how a worrisome attack
might look in a log file, slash-i helps you make a reasonable
guess.

If you wish to become a regular expression archimage, I
recommend the book Mastering Regular
Expressions by Jeffrey E. F. Friedl. See Resources for
details.

Another way to control swatch to a greater degree is to
specify what time of day a given action may be performed. You can
do this by sticking a “when=” option after any action. For
example, below I've got a .swatchrc entry for a medium-importance
event I want to know about via console messages during weekdays,
but I'll need e-mail messages to know about it during the weekend.
To do this I set the when option:

The syntax of the when= option is
when=range_of_days:range_of_hours.
Thus, we see that any time the message “file system full” is
logged, swatch will echo the log entry to the console in red ink.
It will also send e-mail, but only if it's Saturday (“7”) or
Sunday (“1”).

Running swatch

swatch expects .swatchrc to
live in the home directory of the user who invokes swatch.
swatch also keeps its temporary
files there by default (each time it's invoked it creates and runs
a script called a “watcher process”, whose name ends with a dot
followed by the PID of the swatch process that created it).

The -c path_to_configfile and
--script-dir=path flags let you specify
alternate locations for swatch's configuration and script files,
respectively. Never keep either in a world-writable directory,
however. In fact, only these files' owners should even be able to
read them.

For example, to invoke swatch so it reads my custom
configuration file in /var/log and also uses that directory for its
watcher process script, I'd use this command:

swatch -c /var/log/.swatchrc.access --script-dir=/var/log &

I also need to tell swatch which file to tail, and for that I
need the -t filename flag. If I wanted to use
the above command to have swatch monitor
/var/log/apache/access_log, it would look like this:

swatch generally doesn't clean up
after itself very well; it tends to leave watcher-process scripts
behind. Keep an eye out for and periodically delete these in your
home directory or in the script directories you tend to specify
with --script-dir.

Again, if you want swatch to monitor multiple files, you'll
need to run swatch multiple times, with at least a different
tailing-target (-t value) specified each time and probably a
different configuration file for each as well.

Fine-Tuning swatch (in Both Directions!)

Once swatch is configured and running, we must turn our
attention to the Goldilocks Goal: we want swatch to be running
neither too hot (alerting us about routine or trivial events) nor
too cold (never alerting us about anything). But what constitutes
just right? There are as many different answers to this question as
there are uses for UNIX.

Anyhow, you don't need me to tell you what constitutes
nuisance-level reporting: if it happens you'll know it. You may
even experience a scare or two in responding to events that set off
alarms initially but turn out to be harmless nonetheless. Read the
manual, tweak .swatch.rc and stay the course.

The other scenario, in which too little is watched for, is
much harder to address, especially for the beginning system
administrator. By definition, anomalous events don't happen too
frequently, so how do you anticipate how they'll manifest
themselves in the logs? My first bit of advice is to get in the
habit of browsing your system logs often enough to get a feel for
what the routine operation of your systems looks like.

Better still, tail the logs
in real time. If you enter the command

tail -f /var/log/messages

the last 50 lines of the system log will be printed, plus all
subsequent lines, as they're generated, until you kill tail with a
Ctrl-C. This works for any file, even a log file that changes
rapidly.

Another good thing you can do is to “beat up on” your
system in one virtual console or xterm while tailing various log
files in another. The tools we explored last month and the month
before, Nessus and
nmap, respectively, are perfect
for this.

By now you may be thinking, “Hey, I thought the whole reason
I installed swatch was so I wouldn't have to watch log files
manually!” Nope. swatch
minimizes, but does not eliminate, the need for us to parse log
files.

Were you able to quit using your arithmetic skills after you
got your first pocket calculator? No. For that matter, can you use
a calculator in the first place unless you already know how to add,
multiply, etc.? Definitely not. Same goes for log file parsing: you
can't tell swatch to look for things you can't identify yourself,
no more than you can ask for directions to a town whose name you've
forgotten.

Why You Shouldn't Configure swatch Once and
Forget about It

In the same vein, I urge you to not be complacent about
swatch silence. If swatch's actions don't fire very often, it could
be that your system isn't getting probed or misused often, but it's
at least as likely that swatch isn't casting its net widely enough.
Continue to scan through your logs manually from time to time to
see if you're missing anything, and continue to tweak
.swatchrc.

And don't forget to reconsider periodically the
auditing/logging configurations of the dæmons that generate
log messages in the first place.
swatch won't catch events that
aren't logged at all. Refer to the syslogd(8) man page for general
instructions on managing your syslog dæmon and the man pages
of the various things that log to syslog for specific instructions
on changing the way they log events.

Mick Bauer
(mick@visi.com) is a network security consultant in the Twin Cities
area. He's been a Linux devotee since 1995 and an OpenBSD zealot
since 1997, taking particular pleasure in getting these
cutting-edge operating systems to run on obsolete junk. Mick
welcomes questions, comments and greetings.