Those who don't understand UNIX are condemned to reinvent it, poorly.

Main menu

Post navigation

Can you find the DDOS?

This kept me a little busy on Friday night, a long running DDOS hammering at my server, specifically at the VPS subnet, not caring if the IPs were even allocated.

I Reported it to my ISP quite immediately, but didn’t get an answer so far.

At some point I figured this (I guess some few hundred kpps) was just beyond what I could fix on my own, and that this, after all, had not been my weekend plan.

I throttled all traffic to somewhere around 2KB/s and went off to buy Batman Arkham City instead.

This is a weekly RRD that averages the numbers down, but makes better for a comparism. The small spikes are daily backups, a few GB give or take. On the long green one you’ll see how traffic went down after throttling, and you can see it took a full day till the attack finally wore out.

When I looked there was about 5MB/s of incoming SYN with all kinds of funny options, and around 5MB/s of useless ICMP replies from my box. Gotta love comparing this to FreeBSD boxes which simply auto-throttle such an attack right…

Lessons learned:

Syncookies are not optional, you WANT them enabled.

Your kernel will reply to anything it feels reponsible for, thats why I had to concern with the many-MB’s of ICMP replies for the unallocated IP under attack.

Nullrouting unused IPs was the most helpful thing I did.

Throttling was the second most helpful, just next time it needs to be a lot more specific.

IPTables & tc syntax is a complete nightmare when compared to any router OS. I wonder what they took before designing their options. Every single thing it can do is twisted until it’s definitely non-straightforward.

Methodically working on shapers and drop rules was the wrong thing to do! Either have them prepared and ready to enable, or skip it and look at more powerful means right away. If someone is throwing nukes at you, then don’t spend the last minute setting up your air defences. 🙂

enabling the kernel ARP filter might be the right thing to suppress unwanted.responses – or it might break VM networking.

The check_mk/multsite idea of running quite a few distributed monitoring systems is great. Even if I lost livestatus connectivity to the system it still DID do the monitoring, so once I had reasonable bandwidth again all the recorded data was there to look at.

IMO this is much more cruicial with IDS logs. It’s very rare, but there are cases where a big nasty DDOS is just used to hide the real attack.

It feels a smart move to plan for real routers on the network. Of course, that has certain disadvantages on the “OPEX” side of things. I got the routers, but rack units are not free

If you see a sudden traffic spike and spend hours trying to find a software bug or a hacked system, you might be looking at a DDOS probe. Look at this, recorded roughly two weeks earlier:

I noticed this because I had quite well-tuned traffic monitoring already, using the ISP’s standard tools. Even then my guts had been telling me this was someone probing the the targets performance etc. prior to a real attack.

And, finally: I guess I now lost more sleep to playing Batman, even forgot I wanted to go to a party on Saturday. Those damn sidequests 🙂