Fun with Splunk: SSHD

Thought I’d share a bit on the tip of the iceberg, on what can be done with Splunk. Linux command line tools are still much needed for raw log analysis (since we can’t have the luxury of having a Splunk installation around and ready whenever we need it), but if setup and running properly, Splunk can be pretty helpful (and not to mention faster) for some things.

(This post is pretty unpolished, partly because I can’t be bothered to fiddle around with fitting the search strings into the width of the post, etc. Nonetheless, comments/discussions are always welcome heh)

One of my favourite tasks with log analysis is to get information on those people/bots which are brute forcing SSHD, so let’s start with SSH attacks as an example.

Prerequisites

Before we start off, we’ll need Splunk setup to be monitoring the appropriate logfiles. I configured and run the OSSEC and Linux apps for Splunk, so that the data inputs are taken care of for me. If you don’t want to run these apps, just make sure you index the /var/log and OSSEC alert logs locations. If you want to do the geolocation stuff the the MaxMind app for Splunk would be needed too.

List of SSH attacks

Let’s start off with a simple query to see the list of previous SSH attacks:

source=*auth* sshd invalid user from

Using this search string with the needed time range set shows a pretty graph of how many attacks we’ve got over time, along with the list of log entries for the attack.

Click to enlarge

Seems that the attacks everyday are few, probably due to OSSEC’s active responses. A quick search would confirm that OSSEC is blocking the offending hosts.

sourcetype="ossec_alerts"

action="SSHD brute force trying to get access to the system."

Click to enlarge

Drilling Down

Now we know that the attacks were especially active on the 22nd Feb, and OSSEC was responding correctly by blocking them off. Why the large numbers then? Was it because the attacks were from different IP addresses, or that that IP address was particularly persistent that day? We could find out by getting more information on the src_ips for the time range in question. First we click on the bar for the 22nd Feb, then the src_ip field in the sidebar.

Click to enlarge

With the time range fixed onto what we’re interested in looking at, and the src_ip field showing the unique source IPs that were blocked, the results show that it was most likely a persistent attack by these two IPs. A quick check with the auth logs tell the same story:

Click to enlarge

GeoIP Lookups

Now that we know which two IPs were actively poking around, let’s map them to a location. The MaxMind app for Splunk helps nicely for this task.

source=*auth* sshd invalid user from |

lookup geoip clientip as src_ip

Click to enlarge

The app and local geoip database does the lookups for us nicely, mapping to geolocation information like country, city, latitude, longtidue and region. The country information is available for most/all at least, the rest would be put in if available it seems.

List/Count of attacked userids for SSH

The strings for searching for this depends on your SSHD config, but for me searching for the invalid users is enough.

source=*auth* sshd invalid user from |

rex field=_raw "Invalid user (?<atk_user_id>S+) from "

Searching/sorting by the atk_user_id field would show us the attacked userids. Click on the “Events Table” button to show the table of results with only the fields that you’ve selected.

Click to enlarge

If we want a sorted list of the top attacked userids, pipe the search string to a top command.

source=*auth* sshd invalid user from | rex field=_raw

"Invalid user (?<atk_user_id>S+) from "

| top atk_user_id limit=1000

The Results Table should show automatically for this search.

Click to enlarge

Maybe we’d like an alphabetical list instead, so we just pipe the search to a sort command: