rsyslog – That's Geekyhttp://www.thatsgeeky.com
Engage your cerebral cortexSun, 21 Aug 2016 23:20:14 +0000en-UShourly1https://wordpress.org/?v=4.5.13Escalating Consequences with IPTableshttp://www.thatsgeeky.com/2011/02/escalating-consequences-with-iptables/
http://www.thatsgeeky.com/2011/02/escalating-consequences-with-iptables/#commentsSun, 13 Feb 2011 07:03:28 +0000http://www.thatsgeeky.com/?p=478Continue reading →]]>I have previously written a bit about using IPTables to limit brute-force attacks. For the past month, that system has been working quite well. The typical attack pattern resembled that in [graph 1, graph2]. A few days ago, however, an attack was implemented which ‘fell under the radar’, so to speak – instead of being a short-lived, high volume (60/min for 5 min) attack, this one was a slow and prolonged attack (1/2 min for 11 hrs) [graph 3, graph 4].

Improvements

Due to this, I have decided to augment my IPTables ruleset somewhat. There are a couple of points I found lacking in the previous revision. Firstly, repeat offenders did not have any extra consequences – whether you attacked for the first time or the tenth time, you were treated equally. Secondly, a slow attack was not effectively dealt with. Thirdly, the nature of the attack (quick vs slow) was not considered in the consequence. Finally, I wasn’t that pleased with the logging implementation – the log file was not exclusive, and no log rotation was setup. All of the above are addressed in this revision.

I decided on the following setting of bans:

5 connects/min

10 connects /10 min

30 connects/hr

Offence #1

30 min

2 hrs

1 day

Offence #2

2 hrs

1 day

1 wk

Offence #3

1 day

1 wk

1 mo

Offence #4

1 wk

1 mo

1 mo

Offence #5+

1 mo

1 mo

1 mo

Note: there is a big difference between 10 connects/10 minutes and 1 connect/min – if I was to open an SSH client and FTP client in succession, I would have 2 connections in under a minute and would trigger 1 connect/min – however, it would not trigger 10 connects/10 minutes (which reasonably, should not be occurring from a single user for these specific services).

Prerequisite

One further point, before getting to the rules – the recent module, by default, will only log 20 connections. If you attempt to add a rule with a hitcount exceeding 20 to iptables, an error will be thrown. The error (below) can be viewed by running dmesg:

xt_recent: hitcount (31) is larger than packets to be remembered (20)

Since I wish to use 31, it is necessary to modify the options for the module. To do so, modify (or create) /etc/modprobe.d/options, adding (or modifying):

options xt_recent ip_pkt_list_tot=35

(Note: xt_recent is the name of the module – on some systems, the module is named ipt_recent.)

The Rules

The following are the rules I have implemented. If you are adventurous, you can integrate these directly into /etc/sysconfig/iptables. Otherwise, it is probably better to add them individually to iptables (for the most part, just add iptables before each line – except the new chains, which need iptables -N added before them, and when done, view with iptables -S and save with iptables-save > /etc/sysconfig/iptables (backup first!)). Ensure you have a contingency in place if you get locked out of your system (either a cron task flushing iptables (iptables -F) or a ‘backdoor’ of some sort.

Explanations

The majority of the rules build logically of the previous set of rules. For explanations of the basic rule set, see the previous article. For a visual representation, I threw together a flowchart in an attempt to depict the rule set [Image 5]. The modifications are explained below:

-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

This was neglected in the original draft of the previous article – it helps to save a bit of processing power (automatically accepting previously established connections instead of further processing them), and helps to not disconnect a session that was valid, just because another session exceeded the limit (helpful if you are testing).

The difference here is that attackers are now added to one of 5 lists (BANNED1 – BANNED5), new packets are checked against each list – with a differing time. For example, if an entry is in BANNED3, no connections are permitted (on the specified ports), for 1 day (86400 s).

The ATTK_CHECK chain, now jumps to one of 3 ATTACKED chains depending on which rate triggers the jump.

-A ATTACKED# -m recent --rcheck --name BANNED# --rsource -j BAN#

If we make it to an attack chain, the user is going to be banned and the packed dropped Each of the rules in the ATTACKED# chains checks the banned lists, and if an entry is found jumps to the next ban list (escalating consequences for repeat offences).

Note that I wanted different minimums for different attack rates, which is why there are now three ATTACKED# chains – the shorter ban lengths are not used for the slower attack types. If the attacker is already in BANNED5 (the maximum ban length), we simply go to that the BAN5 chain (unlike the other levels, where we go to the next level).

The BAN# chain is where the actual action takes place. Firstly we log, and include a prefix – I have also included the ban time. Since I am now using rsyslog instead of the default syslog, the log-level is much less important than the log-prefix.

-A BAN# -m recent --remove --name BANNED# --rsource

This is mostly a cleanup step – trying to keep each attacker listed in only one ban list.

-A BAN# -m recent --set --name BANNED# --rsource -j DROP

Finally, we add or update the entry in the appropriate ban list, and drop the packet.

Logging Bans with rsyslog

As mentioned a few times above, I have switched from using the default syslog to rsyslog. Many distributions ship this as the default, and others have made it available in their repositories – it is a drop-in replacement for syslog but with quite a few useful additions. The following is for rsyslog (see ‘Upgrading syslog to rsyslog‘ for information on changing syslog).

Edit /etc/rsyslog.conf and add the following at the top of the listed rules:

:msg, contains, "IPTABLES" /var/log/iptables.log
& ~

(Note: while ‘startswith’ should work (instead of ‘contains’), I found that it didn’t, presumably an extra character is included at the start of the message).

Restart rsyslog:

service rsyslog restart

A new file should have been created at /var/log/iptables.log

Setting up Logrotate

Since, there is a good chance that the iptables log will grow considerably, it is probably advisable to setup logrotate to compress and rotate the logs periodically. I have set it up in the following way:

Notifications with Logwatch

Logwatch already has the capability to parse logged iptables messages, however, it is not probably pointed at the correct file. On Amazon’s Linux, the relevant file is located at /usr/share/logwatch/default.conf/logfiles/iptables. Edit this file, setting the values as below:

LogFile = iptables.log
Archive = iptables.log-*

You can see what the output of logwatch will be by running:

logwatch --service iptables --range yesterday --detail med --print

Given the added complexity of this new rule-set, I was pleased to note that CPU usage remained nominal. However, I do believe, that for any additional complexity, an implementation such as pam_shield (which does have rules that can be customized) would be preferable. For my setup, however, SSH is not authenticated through PAM, and I do not feel like changing the setup to use PAM, so I will be sticking with IPTables.

]]>http://www.thatsgeeky.com/2011/02/escalating-consequences-with-iptables/feed/20Upgrading sysklog to rsysloghttp://www.thatsgeeky.com/2011/02/upgrading-sysklog-to-rsyslog/
http://www.thatsgeeky.com/2011/02/upgrading-sysklog-to-rsyslog/#commentsSat, 12 Feb 2011 21:48:21 +0000http://www.thatsgeeky.com/?p=474Continue reading →]]>As a server setup gets more complex, it is possible that the default syslog (sysklogd) lacks the necessary finely grained filtering that may be desired. Common solutions to this are using syslog-ng or rsyslog. I have opted to go with the latter and found it to, so far, be an excellent replacement.

Installing rsyslog

Update: The latest version of Amazon’s Linux AMI uses rsyslog by default.
On Amazon’s Linux AMI, rsyslog (v4.6.2-1.6) is available in the amzn repository and can be installed via yum. At first glance, however, rsyslog appears to conflict with sysklog (specifically, the file /etc/logrotate.d/syslog). Attempting to uninstall sysklog is not an option, as it appears that pretty much everything depends on it). Renaming the conflicing file, of course, does not resolve the dependency.

The solution is to use yum shell to install rsyslog while simultaneously removing sysklog – the dependencies will be resolved without issue.

At the yum shell prompt, enter (wait for each command to execute before continuing):

You should note that both the install and remove were processed together, and the old syslog.conf was backed up. (Additionally, the conflicting file /etc/logrotate.d/syslog is also backed up and a new (slightly different) one created).

Next steps

Rsyslogd is a drop in replacement for syslogd – it starts with the same default set of rules (any custom selectors you added to syslog.conf are not copied over). It should work without any setup on a system which doesn’t have custom syslog rules setup. For systems with custom rules, you will have to manually copy the rules to the new conf file (most should not need modification). By default, rsyslogd is not started, although it is setup to start at boot. To start rsyslogd, run:

service rsyslog start

You can view startup settings for rsyslog by running:

chkconfig --list rsyslog

Which should return something similar to the following (default):

rsyslog 0:off 1:off 2:on 3:on 4:on 5:on 6:off

rsyslogd does not use the same configuration file as sysklogd, although the format shares many similarities. The file used by rsyslogd is /etc/rsyslog.conf (while the one used by sysklogd was /etc/syslog.conf)

Suhosin Example

As a quick example of one of the available features of rsyslogd that is not found in sysklogd, consider the following scenario:

I have suhosin installed on my server, and some scripts trigger it – however, it sends messages at a debug level, and so do other scripts. This results in suhosin notifications being logged to /var/log/messages as well as a custom log file I had previously setup.

With rsyslog, it is possible to look for a particular part of the message and filter using that, as well as to prevent subsequent selectors from being triggered.

While the message may change depending on the event, the program name (suhosin) remains constant and can be used to filter the message.

To setup such a filter, add the following to /etc/rsyslog.conf, at the top of the rules section (before any other rules).

:programname, contains, "suhosin" /var/log/suhosin.log
& ~

This will check the ‘programname’ to see if it ‘contains’ the text ‘suhosin’, and if it does, will redirect the message to the specified log file. The next line, discards the message, so that no further rules are triggered

Oddity with SpamAssassin

After installing rsyslog, I noticed that SpamAssassin would throw the following warning when started:

As such, sa-update, which restarts the service, would generate the same error (and cron would mail it out). To resolve this, modify /etc/sysconfig/spamassassin (Amazon’s Linux, RHEL/CentOS derived) appending --syslog-socket=native to SPAMDOPTIONS. The final result should resemble the following:

/etc/sysconfig/spamassassin:

SPAMDOPTIONS="-d -c -m5 -H --syslog-socket=native"

Process Dies

One of the problems I have found with this setup is that the process appears to die periodically as per the status below:

#service rsyslog status
rsyslogd dead but pid file exists

Further investigation into the problem reveals that when rsyslog receives the HUP signal, it will attempt a complete restart (however, it appears to not quite succeed in this case). The source of the HUP signal appears to be logrotate.

This result can be verified by running:

#kill -HUP `cat /var/run/syslogd.pid`
#service rsyslog status

The status message is the same as the one above.

From this state, rsyslog can be directly started (without even deleting the PID file) via:

#service rsyslog start

While an upgrade of rsyslog may resolve this issue (I haven’t tried), a simpler solution appears to be a minor modification to the configuration:

Edit rsyslog.conf, add the following at the end of the Global Directives section (just before the Rules section):

$HUPisRestart off

Sending the HUP signal no longer results in the termination of the process making rsyslog considerably more usable.