Working with traps

Working with traps

For this feature set you need the latest version of firehol, currently in github.
You can get it with this:

# git clone https://github.com/ktsaou/firehol.git firehol.git

Once you do this, you will have firehol in firehol.git/sbin/firehol.in. You can run it from there.

There is a feature of recent iptables and kernels that allows us to build some sort of an IDS (intrusion detection system).

It works like this:

Linux kernel supports ipsets. Please check Working With IPSETs if you don't know what ipsets are.

iptables supports the SET action. This action allows adding and removing IPs to ipsets, dynamically, from within the firewall itself. To see its options check: iptables -j SET --help

We could therefore use these features in several ways:

Detect network-wide port scans
There are hundreds of people scanning the internet for unupdated and unsecured applications. These people usually focus on one port on all possible IPs.
What we want, is to find them and block them from our network for some time.
To do it, we will create an empty ipset named trap and when we detect that someone is trying to access forbidden services (i.e. we don't provide a public mysql server, but we see someone trying to connect to a mysql on our host or network), we could add his IP on the trap ipset for a day, or a week (this timeout is configurable per IP and is handled by the kernel, which will automatically remove the IP from the ipset after the timeout).
So, after a few days we will have a large blacklist of all the attackers found.
Another way of achieving the same (even a little better) is if you have a few spare public IPs. If you have unused public IPs, then you could trap all those trying to connect anywhere on these IPs. This is better because you will be able update your blacklist with all those scanning any port.

Detect single host port scans
The second way of using it is for detecting port scans on one host.
To do it, we will change the policy of the internet interface.
Normally you have either REJECT or DROP on the internet side. We will replace this with a custom action that when it detects that someone is trying to reach something that is not allowed, will add his IP on a new ipset named policytrap. This ipset does not block him. It just counts how many times he has tried to connect to something we didn't allow.
Once we have the policytrap ipset that counts the illegal accesses, we will add a rule to the firewall so that all IPs that have a count of more than 50, in less than an hour, will be automatically added to the trap ipset, that blocks them.

Knock-knock
Another way of using ipsets and traps, is to provide sequences of ports that once "knocked" in a short time and in proper sequence by an IP, this IP will be allowed to access a service, or just be removed from the blacklist.
This will work like this: You attempt to access port A and the firewall adds you to an ipset called step1 for 30 seconds. Now there is another rule that says if anyone on step1 accesses port B, then his IP is added on the ipset step2. We can continue adding steps as needed. These ipsets (step1, step2, etc) can be used by services to provide different levels of access (for example, the intranet web server requires 3 steps, but the ssh server requires 6).

The examples bellow refer to firehol.conf.

Decide which is your public interface

It is very important to apply all these rules on your internet interface. If, even by error, you apply it on your VPN or intranet link you may end up locked out.

world="dsl0"

We will use the ${world} variable from now on.

Keep in mind that if you have multiple internet interfaces, you can just add them to the same variable. Just separate them with spaces, but keep then inside the quotes (like this: world="dsl0 ppp12 eth4").

Whitelist

Before we proceed you should create a whitelist. The whitelist should include all IPs you know that should never be blacklisted for any reason.

Traps

ipset is the name of the ipset to use. If the ipset does not exist, the iptrap helper will create it.

type is either src or dst and defines which IP of the matching packets will be used.

seconds is the duration in seconds each IP address will remain in the ipset.

timeout and counters are mutualy exclusive. They control how IPs that are already in the ipset will be handled. timeout is the default and means that for every packet matching the IP address already in the ipset, its seconds will be reset and restart counting down, while counters means that for every packet matching the IP address already in the ipset, its packets and bytes counters will be incremented. Unfortunately, the kernel cannot do both.

The iptrap helper will compare packets against its first rule-params. If they match, and except rules have been given it will also check that the packets do NOT match against the excepted rule-params and then, it will add the IP of the packet (src or dst depending on type) to the ipset given, for timeout seconds.

Thats it. It does nothing more. The packet will continue to the next rule. It will not be dropped, rejected or accepted by the iptrap helper.

Now lets create our first iptrap. Remember this will not block anything. It will just add a few IPs to an, otherwise, unused ipset.

If you whois a few of the IPs you will get the idea: China (most of them), Russia, Colombia, a few Universities and virtual machine hosting companies. All these scanned my udp/5060 (SIP), tcp/23 (TELNET), tcp/3306 (mysql) and the other trapped ports.

Personally, I also have a few spare IPs. Public internet IPs that I never used, that do not have a dns name and are not routed to any real machine. I have setup on them a trap (on the router) for all possible ports. That ipset is 3 times longer!

Before blocking IPs we should prepare a way of unblocking us, so that in case we get locked out, we will have a way to get back in:

knock-knock

Using a very similar procedure we can create a method for removing ourselves from the trap ipset.

You get the idea. We just copy IPs from ipset to ipset. Each knock needs a different dport and a different ipset. The timeout we give is very small. 30 seconds to get from knock to knock.

Of course we can add as many steps as we like.

Without adding more rules, we could say somewhere in the firewall:

...
server ssh accept src ipset:knock3
...

Using just this, our ssh server will allow connections only from the IPs in knock3. And to get in knock3 to have to be in knock2... and in knock1... and move from knock1 to knock3 in 60 seconds (30 seconds per step).

So, we can have different servers requiring different levels of protection like this:

Apply it and try it.
Attempt to get into the trap ipset (you will not be locked out - we have not added the blacklist yet) and knock the ports in sequence to get out. Verify it works by running ipset list on your console.

Once you are confident that it works, lets drop some packets...

Blacklisting

Just one command will blacklist all the IPs in the trap ipset.

Before adding this command, please make sure that you have tried to enter and leave the trap ipset successfully.
The place you will add this command in firehol.conf is important. You have to put it below the untrap command. If you put it above, you will not be able to be untrapped.

It is easy to understand what this does. except src ipset:whitelist is redundant, I know. The ipset trap cannot have any of the IPs of the ipset whitelist since all the commands that add IPs to trap have already done this check. I add it because it is better to be safe than sorry.

The first argument to blacklist is full. full means you will not have any communication with the blacklisted hosts. If you replace full with input, then you will be able to connect to them, but they will not be able to connect to you.

Remember, add the blacklist command below ipuntrap.

This is it. Network scanning blacklisting and knock functionality.

Lets now move to trapping single host port scans.

Keep in mind this is the only blacklist command in this document. Only this command drops packets. All the other commands are just adding or removing IPs from ipsets. Without the blacklist command active, you can test everything without any risk for getting locked out of your systems.

Single host port scans

In order to detect single host port scans we need to examine all the packets not accepted by the firewall. What happens to these packets is controlled by the policy command. You can add one such command for each interface and router you have in your firehol.conf.

The default for interfaces is policy DROP. The default for routers is policy RETURN (routers do not drop or reject traffic by default - traffic in routers traverses all the routers you have and is dropped at the end of the firewall).

What can we use for a policy that will allow us to examine all packets rejected/dropped and possibly add them to an ipset?

Well, another new feature of FireHOL, is an extended action helper. The action helper allows us to define new actions like this:

table is optional, it defines a list of netfilter tables to add the action to. The default is filter which is ok for our needs.

type is the magic. It can be chain (or action which is an alias for chain), rule or iptrap. Each of these has its own type_params. Check the manual page in github repo (doc/firehol/firehol-action.5.md).

When packets are given to this action, they are first examined by an iptrap that tries to find if they are coming from dsl0, have a state NEW (they are not part of existing connections) and their source IP is not in the ipset whitelist. If all these are true, then it adds the source IP of the packet to a new ipset called policytrap for an hour (3600 seconds).

The counters keyword is very important. It makes iptrap increase the packets and bytes counters of the IPs that are already in the ipset. Without it, the default is to reset the timeout of the IPs already in the ipset.

Since ipset does not take any filtering action on the traffic, the next action that will take the packets is DROP.

So, what we have done here is that we created an action, that drops traffic, but before doing this, conditionally it adds the IPs we are interested in the ipset policytrap.

Try it. It does not blacklist anything yet. Change the policy of your internet interface to TRAP_AND_DROP.

(If you now use REJECT for your internet interface policy, just change the last DROP to REJECT and rename it to TRAP_AND_REJECT).

When you do this, you will be able to see what gets into the policytrap ipset and decide what to do next. This is mine:

I have quite a few IPs in it. Check the details. Each IP has a timeout, a packets counter and total bytes.

The packet counter is what we are interested in. It actually says how many times this IP has reached the policy of the interface.

Before doing anything more, use an online service and do a port scan on your IP. Once the port scan completes, check the packet counter of the IP that scanned you, on this ipset. It should be very high.

I suggest to temporarily comment out the blacklist line we added above. If the port scanner hits a port we have trapped, it will enter the trap ipset and be blocked before completing the test. So, for the test, just comment out the blacklist helper we added in the previous section.

Once you have tested it and feel comfortable with it, it is time to blacklist all the IPs in this set we consider a threat.

Personally, I consider a threat all IPs that have more than 50 packets dropped in an hour. Our blacklist will not wait for an hour. We will keep the IP in the policytrap ipset for an hour, but the blacklist will block the IP the moment it receives the 51st packet from it.

Another tricky command. This one is an iptrap like all the ones we have seen so far. It will add an IP to the trap ipset for a day (86400 seconds), much like all the previous iptraps did.

The trick here is the ipset match. The line ipset policytrap src packets-above 50 says to match the packet's source IP with all the IPs in the policytrap ipset, only if the packets counter for this IP in the ipset is above 50.

So, this iptrap command copies once more an IP from one ipset to another, but it is smart enough to do it only when the packets counter on the source ipset is above 50.

Now, with the blacklist command disabled, do the port scan again. The IP of the port scanner should now be in trap ipset, which means that the blacklist helper, once activated, will block it.

Important

In such a huge network such as the Internet, it is probable that you will encounter all kinds of problems with network traffic. Trapping and blacklisting IPs on the policy of an interface, means that even legitimate clients that face some network related issue may be trapped.

We should make the most to avoid such scenarios. Hopefully, FireHOL provides a few such ways:

Add protection strong or if your don't want this (strong applies a rate-limit on the connections), at least add protection new-tcp-w/o-syn to your internet facing interface and router. This will drop TCP packets that lost their way to you, buffered at some router and came reordered and too late. They are already useless, since the connection tracker has already decided they are not part of an existing connection.

The connection tracker sometimes closes sockets too early and when the other party sends TCP ACK+FIN packets, they get dropped. Make sure you have FIREHOL_DROP_ORPHAN_TCP_ACK_FIN=1 (this is already the default) so that TCP ACK+FIN packets do not reach our TRAP_AND_DROP action.

You can monitor which IPs get trapped due to the TRAP_AND_DROP action, by checking your iptables logs for lines with the text: "MOVED FROM POLICYTRAP TO TRAP".