So while we were visiting the really nice Uluwatu temple with two friends/ex-colleagues, I learned, thanks to Uptime Robot and some Slack messages from my colleagues that our website had started to be sluggish/buggy/inaccessible.

After a lazy-shameless sudo service php5-fpm restart and seeing with htop that our EC2 m4.xlarge instance was still 100% on all 4 cores, it started to really look like an attack. Thanks to Fail2Ban, iptables and our Symfony2 refreshed code, I was feeling quite secured against SSH and SQL injections. But downtime is nether good for business, it was time to find my Mac, as Serverauditor iOS app is quite light to really investigate (but could have been enough as [spoiler] our quick-fix were quite easy to install).

Let’s investigate

Make the Apache2 logs readable behind an AWS ELB

We first discovered (with @remiii who was visiting me) that my Apache2 logs where always giving us the same IP address : the ones of the AWS ELB (Elastic Load Balancer). To fix this, I had to update my vHost configuration to change the IP to the one sent by the ELB in the X-Forwarded-For header :

Scale of the attack, or is-it a DDoS?

If the attack is not distributed, a single rule on iptables could shut down the attack in 2 sec…

We were now able to look at the logs and check the distribution of the IPs. I took the last 10000 lines of the access log, put them in a Google Sheets, kept only the IPs, used the UNIQUE() fonction to get distincts ones, and then a COUNTIF() function to get hits/IP.

It seemed to me easier than some grep pipe sed in command line #noob

What are the attacked pages?

By looking at Apache2 logs, we easily found that all requests were GET on the homepage.

Let’s fix this

iptables , catch ‘em all!

When I’m adding iptables rule

Based on our Google Sheets of cleaned IP, we first thought the attack might not be distributed enough and we could just block the most harmful IPs. We added some iptables -A INPUT -s xx.xx.xx.xx -j DROP on what seemed to be the most harmful IPs, but it did not lower the attack. And it also broke the website as some of these IPs where AWS internal ones

First conclusion: it’s a distributed attack, we cannot just block IPs.

Look how big is my instance

We first tried to switch from my EC2 m4.xlarge to a m4.10xlarge. As our websites were already down, it was quite easy to shutdown the instance, switch to the largest available, and start again. Sadly, it was 100% useless 😀

40 cores, but still unable to process 10 requests/sec

Smarter (also, working) solution : caching

After this first failure, we spent a few minutes in the villa pool, and thought that as the attack was only on the homepage, we could just serve a cached version of this one page.

In my HomeController.php , I just had to add this annotation before my action :

Bingo! Huge drop in CPU usage, but huge increase in bandwidth: we went from 30Mb/sec to nearly 130Mb/sec, as the instance was now able to serve the homepage to everyone. And we can now see that the attack ratio is quite huge. As the ~10 req/sec were spotted at 1.6Mb/sec incoming, we were sending 80 time what we were receiving. The bandwidth bill might be expensive this month.

Before the first orange bar, it’s the taxi back to reach my laptop.

Future solution

Hopefully, our opponent did not really tried for long to hack us, so the cache solution was enough. We could have played hide and seek on all the website pages : him starting to attack, me adding some cache annotation.

A more viable solution will be to progressively add HTTP cache on all our pages, and also to add CloudFlare so that requests are filtered by them before hitting our servers.

After words

Our attacker ransom message was stuck in our inaccessible CRM, so we only found it after we “fixed” his attack We got its IP, but it’s a Tor node.

Sending your ransom message in the system you’re currently hacking is not logic.

The day after, Sunday, our friend retried, but this time I had my laptop with me. CPU peak was the time for me to re-add the caching lines (that I had removed because of weird side effects)