Once you put the bastion host into production, your job has only just
begun. You'll need to keep a close watch on the operations of the
bastion host. Chapter 12 provides more information
on how to do this; this section discusses specific concerns for
bastion hosts.

If you're going to monitor the bastion host, looking for abnormalities
that might indicate break-ins or other types of system compromise, you
will need to first develop an understanding of what the
"normal" usage profile of the bastion host is. Ask these
questions, and others like them:

How many jobs tend to be running at any one time?

How much CPU time do these jobs consume relative to each other?

What is the typical load at different times throughout the day?

Your goal is to develop an almost intuitive grasp of what your system
normally runs like, so you'll be able to recognize - and
investigate - anomalous activity very quickly.

Doing a thorough job of system monitoring is tough. Although the logs
produced by your system provide lots of useful information, it's easy
to get overwhelmed by the sheer volume of logging data. The important
information may often be buried. Too often, the logs end up being used
only after a break-in, when, in fact, they could
be used to detect - and thus perhaps stop - a break-in while it is
occurring.

Because each operating system and site is different, each bastion host
is configured differently, and each site has different ideas about
what the response of a monitoring system should be. For example, some
want email; some want the output fed to an existing
SNMP-based management system, some want the systems
to trip the pagers of the system administrators, and so on. Monitoring
tends to be very site- and host-specific in the details. However,
there are some useful tools out there that you should be able to
configure and adapt for your own use. The SWATCH
(Simple WATCHer) package is a good example.

SWATCH, developed by Stephen E. Hansen and E. Todd
Atkins, automates the monitoring of UNIX
systems. SWATCH enhances the standard
syslog facility in various ways. (See the
discussion of syslog in "Setting Up System
Logs" earlier in this chapter.) It sifts through the logs as
they're created by syslog, and takes certain
actions when certain types of log messages are found, e.g., sounding
an alert when repeated unsuccessful login attempts are made to the
same account, or a "file system full" message is encountered.
SWATCH also includes modifications for a number of
daemons to make their logging more useful; these include
fingerd, ftpd,
ruserok, rshd, and
login. For example, login
has been modified so that it allows only three login atempts; it
reports to syslog on any "Incomplete Login Attempt", "Repeated Login
Attempt", and "Root Login Refused" events; and it includes the account
names attempted and the originating host. SWATCH
can also watch files other than ones generated bysyslog. Appendix B gives you information
on where to get SWATCH.[4]

[4] The 1993 and 1994 USENIX/SAGE LISA conferences (see Appendix A for
information about USENIX, SAGE,
and the LISA conferences) have produced a number of
papers on other automated monitoring tools that were originally
intended for system administration use, but that might be adapted to
use in monitoring system security.

SWATCH is written in Perl, which is an
unfortunately powerful tool to have sitting on a bastion host; it
provides almost everything an intruder could get through having a
compiler except the ability to build new kernels. You will probably
want to run SWATCH on the machine that the bastion
host is logging to, rather than on the bastion host itself.