grok - an expert system for real time log analysis

Posted Mon, 06 Jul 2009

Table of Contents

Grok is simple software that allows you to easily parse logs and other
files. You teach grok how to parse data through the config file. Grok is
a system for reacting to events - those events being log entries.

My servers get hit with brute-force ssh login attempts almost every day.
They're also hit with several exploit attempts on varying protocols
daily. Chances are all Internet-facing systems will see this on a
regular basis. I've never had a successful break in, but why give script
kiddies a chance?

My logs end up riddled with line after line of failed logins, etc. So I
needed a system that could watch the logs

Failed login attempts are logged in /var/log/auth.log on
FreeBSD, so that's where I'll be looking for patterns to match. An
example of a failed login attempt is this:

This is an illegal login attempt for a username 'test' - this means there
is no user 'test' on that server. I have made the assumption that no one
will try to login to this machine with an invalid username more than once
or twice in a short while, so that will be my criteria for blocking these
script-kiddie login attempts.

Enter grok. This is a short perl program I wrote that allows you to
configure reactions to certain patterns. The configuration file format
is pretty straight forward, in my opinion.

The match will be triggered and a count will be increased. If the
threshold ever hits 4 in 300 seconds, the "reaction" will be executed. It
will run pfctl -t whores -T add 193.195.96.6, which will add
that IP to the 'whores' table in pf. Here is the pertinent section of my
pf.conf:

table <whores> persist file "/etc/pf.whores"
# Block whores.
block in log quick on $ext_if from <whores> to any
block out log quick on $ext_if from any to <whores>

I find this to be pretty useful in helping to keep my machine free of
brute-force attempts.

This, immediately, may not seem useful. However, you'll note that grok is
able to parse tcpdump output with very little effort on your part. Here's
another example, where we look for port scans using the new 'key'
feature:

grok is not a substitute for IDS software, nor is it IDS software, but
the point of the above demonstrations are to show the usefulness of both
the better capture naming and key generation. However, you may find it
useful to use grok for IDS-type activities.