Now, this "Question" is meant to be a Community Wiki for collecting together various bits-n-pieces of iptables wisdom. Nothing is too common or too obscure. Post anything you know that would help others make the most of iptables.

Block ICMP (aka ping)

Consider adding a comment to the "spoofed" comments, so that less experienced users know why the source addresses are considered spoofed (...when arriving on a wan interface).
–
3moloMar 10 '11 at 15:07

Re the whitelist script: Theres iptables on line 8, and then $IPTABLES later on. Is it enough just to use iptables everywhere? Otherwise I suppose you'd need to assign something like IPTABLES=/sbin/iptables right?
–
UpTheCreekApr 25 '13 at 9:34

This means that a packet with the source address of 192.168.251.177 must first traverse hundreds of rules before it can get its verdict of ACCEPT.

Of course, experienced sysadmins will split the rules by subnet. But that still means hundreds of rules.

ipset to the rescue!

First, define an IP Set of ipmap type:

ipset -N Allowed_Hosts ipmap --network 192.168.0.0/16

Then, populate it with the addresses:

for ip in $LIST_OF_ALLOWED_IP; do ipset -A Allowed_Hosts $ip; done

Finally, replace the hundreds of iptables rules above with one rule:

iptables -m set --match-set Allowed_Hosts src -j ACCEPT

When a packet arrives, netfilter will perform a very quick bitmap search for the packet's source (src) IP against the Allowed_Hosts IP Set. All packets coming from 192.168.0.0/16 will experience one rule. And do believe me that searching a bitmap is at least two order of magnitudes faster than performing hundreds of iptables rule-checking.

ipset is not limited to IP addresses. It can also match based on ports, IP-port tuple, network/subnet addresses, IP-MAC tuple, and so on and so forth. And it can match those criteria as source or destination or a mix of both (in the case of tuples).

And finally, with ipset you can automatically put IP addresses in blacklists/whitelists. These blacklists/whitelists can also 'age', thus automatically deleting the IP address after a configurable amount of time has passed.

VERY IMPORTANT NOTE:

Some Linux distros do not have 'out-of-the-box' support for ipset. Two distros I know that do not support ipset out-of-the-box are Ubuntu and Debian. Running aptitude on them tantalizingly shows that an ipset package is available, but I strongly recommend you to not install that package for two reasons: (1) It's an old version of ipset, and (2) It won't work anyway.

Edit: The above is true for Ubuntu 10.04, the latest LTS at the time of writing. Ubuntu 12.04 has built-in support for ipset and/or xtables-addons, in the sense that installing the relevant package properly enables the features. I don't know the situation on Debian, though, having not used it for quite some time...

According to security.stackexchange.com/questions/4603/…. "There is no need to drop invalid or malformed packets, all of these attacks are a decade old. The Linux kernel devs are much more up to date than you concerning which kind of packets are valid and which not. "What about future flaws", some might argue. Well, how do you know the future flaw will be in the TCP handler and not in the iptables TCP parser?"
–
MattApr 23 at 20:30

I found it much more intuitive than direct iptables commands. Especially for people with past experience with other firewalls:

FireHOL is an iptables firewall
generator producing stateful iptables
packet filtering firewalls, on Linux
hosts and routers with any number of
network interfaces, any number of
routes, any number of services served,
any number of complexity between
variations of the services (including
positive and negative expressions).

Another GUI that can be used to configure iptables is Firewall Builder. It lets users create rule elements as objects in a database and then drag-and-drop these objects into a rules editor to create the desired firewall policy. The application then generates a script file with all the iptables commands required to implement the rules.

Unlike some other iptables GUI solutions where you can only manage one iptables configuration at a time, with Firewall Builder you can manage a large number of iptables configurations all from a single application. Firewall Builder runs on Linux, Windows and Mac OS X, has been around for over 10 years and has thousands of active users around the world.

Full Disclosure - I'm the co-founder of NetCitadel which is the company that develops Firewall Builder.

IP sets revisited

There is already an answer mentioning IP sets. However, it's rather one-dimensional in that it focuses on the performance gains over classic rules and the fact that IP sets mitigate the problem one has with lots of individual IP address that cannot easily be expressed as a subnet in CIDR notation.

Notation used below

For ipset I will use the notation read by ipset restore and written by ipset save.

Correspondingly for iptables (and ip6tables) rules I will use the notation as read by iptables-restore and written by iptables-save. This makes for a shorter notation and it allows me to highlight potential IPv4-only (prefixed -4) or IPv6-only (prefixed -6) rules.

In some examples we'll divert the packet flow into another chain. The chain is assumed to exist at that point, so the lines to create the chains are not produced (nor is the table name mentioned or the commands COMMIT-ted at the end).

For example I'll mainly focus on three set types: hash:ip, hash:net and list:set, but there are more than those and they all have valid use cases.

You can for example also match port numbers, not just IP addresses.

Saving and restoring IP sets as with iptables-save and iptables-restore

You can create IP set declarations in bulk and import them by piping them into ipset restore. If you want to make your command more resilient against already existing entries, use ipset -exist restore.

If your rules are in a file called default.set you'd use:

ipset -exist restore < default.set

A file like that can contain entries to create sets and to add entries into them. But generally most of the commands from the command line seem to have a corresponding version in the files. Example (creating a set of DNS servers):

Timeouts on IP sets

Timeouts in IP sets can be set as a default per set and also per entry. This is very useful for scenarios where you want to block someone temporarily (e.g. for port-scanning or attempting to brute-force your SSH server).

The way this works is as follows (default during creation of IP sets):

Timeout caveats

timeouts are a feature on any given set. If the set was not created with timeout support you'll receive an error (e.g. Kernel error received: Unknown error -1).

timeouts are given in seconds. Use Bash arithmetic expressions to get from minutes to seconds, for example. E.g.: sudo ipset add ssh_dynblock4 1.2.3.4 timeout $((120*60))

Checking whether an entry exists in a given IP set

Inside of your scripts it can be useful to see whether an entry already exists. This can be achieved with ipset test which returns zero if the entry exists and non-zero otherwise. So the usual checks can be applied in a script:

if ipset test dns4 8.8.8.8; then
echo "Google DNS is in the set"
fi

However, in many cases you'll rather want to use the -exist switch to ipset in order to direct it not to complain about existing entries.

Populating IP sets from iptables rules

This, in my opinion, is one of the killer features of IP sets. Not only can you match against the entries of an IP set, you can also add new entries to an existing IP set.

... with the intention to rate-limit connection attempts to SSH (TCP port 22). The used module recent keeps track of recent connection attempts. Instead of the state module, I prefer the conntrack module, however.

If you wanted to match the target/destination address instead, you'd use dst instead of src. Consult the manual for more options.

Sets of sets

IP sets can contain other sets. Now if you followed the article up to here you'll have wondered whether it's possible to combine sets. And of course it is. For the IP sets from above we can create two joint sets ssh_dynblock and ssh_loggedon respectively to contain the IPv4-only and IPv6-only sets:

which is a lot more concise. And yes, this is tried and tested and works like a charm.

Putting it all together: SSH brute-force defense

On my servers I have a script run as a cron job which takes a bunch of host names and resolves those to IP addresses, then feeding it into the IP set for "trusted hosts". The idea is that trusted hosts get more attempts to log into the server and aren't necessarily blocked out for as long as anybody else.

Conversely I have whole countries blocked out from connecting to my SSH server, with the (potential) exception of trusted hosts (i.e. order of rules matters).

However, that is left as an exercise for the reader. Here I'd like to add a neat solution that will use the sets contained in the ssh_loggedon set to allow subsequent connection attempts to be handed through and not be subject to the same scrutiny as the other packets.

It is important to remember the default timeouts of 90 minutes for ssh_loggedon and 30 minutes for ssh_dynblock when looking at the following iptables rules:

By now you should ask yourself how the connecting IP address ends up in the ssh_loggedon sub-sets. So read on ...

Bonus: adding the IP you log in with during SSH logon

If you have experimented with sshrc and friends, you'll have learned of its shortcomings. But PAM comes to the rescue. A module named pam_exec.so allows us to invoke a script during SSH logon at a point where we know that the user is admitted in.

In /etc/pam.d/sshd below the pam_env and pam_selinux entries add the following line:

session optional pam_exec.so stdout /path/to/your/script

and make sure that your version of the script (/path/to/your/script above) exists and is executable.

PAM uses environment variables to communicate what's going on, so you can use a simple script like this one:

Unfortunately the ipset utility doesn't seem to have the built-in smarts of netfilter. So we need to distinguish between IPv4 and IPv6 IP set when adding our entry. Otherwise ipset will assume we want to add another set to the set of sets, instead of the IP. And of course it's unlikely that there would be a set named after an IP :)

So we check for : in the IP address and append 6 to the set name in such case and 4 otherwise.

Something I do, mainly because of my ignorance of a more elegant solution, is to manually check my Nginx logs every 4 hours and the mail server logs every 2 minutes for excessive access by individual IP's. I run a few scripts together that:

Check the access.log and list off the top 10 IP's organized by how many hits they have to the server

Dump the results into a log file

Have another script look at that log file and ban any IP that has hit the server more than X amount of times over the past X hours

One thing that is VERY important to note here is that you NEED to setup a whitelist or you are going to start blocking a lot of authentic IP's from servers that you just receive a lot of email from or in the case of other logs, IP's that just hit your server a lot for legitimate reasons. My whitelist is just built into this script by adding grep pipes right after | grep ']' | that look something like this "grep -v 127.0 |".
You need to take the time to teach your server which high traffic IP's are legit and which aren't. For me this meant that I had to spend the first week or so checking my logs manually every couple of hours, looking up high traffic ip's on iplocation.net and then adding the legit ones like amazon, box.com or even my home/office IP ranges to this whitelist. If you don't you will likely be blocked from your own server or you are going to start blocking legit mail/web servers and cause interruptions in email or traffic.

Again I know this is crude as hell and there is probably a nice clean efficient protocol that does all of this but I didn't know about it and this thing has been going for a year or two now and keeping the bad guys at bay. The one thing I would very SERIOUSLY recommend is that you have a proxy or another server in the wings that you can use to access your main server.. The reason being is that if you are doing web development one day out of the blue and you ping your self 2000 times in 5 hours for some testing you could get blocked with no way back in except for a proxy.

You can see that in checkBadIPs.sh I've put grep -v 127.0 and in my actual files I have a ton of ignore rules for my own IP's and other trusted IP ranges but sometimes your IP changes, you forget to update and then you're locked out of your own server.

Anyways, hope that helps.

UPDATE

I have changed things a little bit so that now instead of checking every couple hours I have some logs checked every 2 minutes, mainly my ssh auth log and the mail log as they were getting pounded :(.

I setup specific scripts for each log file although it would be easy enough from the manual script I use myself when wanting to inspect logs. Looks like this:

Nice! I'd use ipset rather than elongating the iptables chain, but the idea is great and I think I'll apply them to my production servers. Thanks for sharing!
–
pepoluanAug 23 '14 at 14:48

2

I just read about ipset for the first time the other day and was gleefully delighted to learn about what it does.. I'm a little scared to implement it just because I'll probably muck it up in the beginning and shut the server down but it's on my list of things to learn here. That being said my iptable chain is probably around 30-40 items and only gets 1 new one every day or two so it's not at a point that I'm overly worried.
–
unc0nnectedAug 24 '14 at 19:05