The Porcupine Attack: investigating millions of junk requests

We extensively monitor our network and use multiple systems that give us visibility including external monitoring and internal alerts when things go wrong. One of the most useful systems is Grafana that allows us to quickly create arbitrary dashboards. And a heavy user of Grafana we are: at last count we had 645 different Grafana dashboards configured in our system!

grafana=> select count(1) from dashboard;
count
-------
645
(1 row)

This post is not about our Grafana systems though. It’s about something we noticed a few days ago, while looking at one of those dashboards. We noticed this:

This chart shows the number of HTTP requests per second handled by our systems globally. You can clearly see multiple spikes, and this chart most definitely should not look like a porcupine! The spikes were large in scale – 500k to 1M HTTP requests per second. Something very strange was going on.

Which is generated by starting at ef and adding the following repeating sequence.

4a 49 49 4a 49 49 49

The ‘random’ binary junk is actually generated by some simple code.

The length distribution of the requests is also interesting. Here’s the histogram showing the popularity of particular lengths of payloads.

About 80% of the junk requests we received had length of up to 511 bytes, uniformly distributed.

The remaining 20% had length uniformly distributed between 512 and 2047 bytes, with a few interesting spikes. For some reason lengths of 979, 1383 and 1428 bytes stand out. The rest of the distribution looks uniform.

The scale

The spikes were large. It takes a lot of firepower to generate a spike in our global HTTP statistics! On the first day the spikes reached about 600k junk requests per second. On second day the score went up to 1M rps. In total we recorded 37 spikes.

Geography

Unlike L3 attacks, L7 attacks require TCP/IP connections to be fully established. That means the source IP addresses are not spoofed and can be used to investigate the geographic distribution of attacking hosts.

The spikes were generated by IP addresses from all around the world. We recorded IP numbers from 4,912 distinct Autonomous Systems. Here are top ASN numbers by number of unique attacking IP addresses:

The traffic was truly global and launched with IPs from 121 countries. This kind of globally distributed attack is where Cloudflare’s Anycast network shines. During these spikes the load was nicely distributed across dozens of datacenters. Our datacenter in São Paulo absorbed the most traffic, roughly 4 times more traffic than the second in line – Paris. This chart shows how the traffic was distributed across many datacenters:

Unique IPs

During each of the spikes our systems recorded 200k unique source IP addresses sending us junk requests.

Normally we would conclude that whoever generated the attack controlled roughly 200k bots, and that’s it. But these spikes were different. It seems the bots rotated IPs aggressively. Here is an example: during these 16 spikes we recorded a total count of a whopping 1.2M unique IP addresses attacking us.

This can be explained by bots churning through IP addresses. We believe that out of the estimated 200k bots, between 50k and 100k bots changed their IP addresses during the 80 minutes between attacks. This resulted in 1.2M unique IP addresses during the 16 spikes happening over 24 hours.

A botnet?

These spikes were unusual for a number of reasons.

They were generated by a large number of IP addresses. We estimate 200k concurrent bots.

The bots were rotating IP addresses aggressively.

The bots were from around the world with an emphasis on South America and North Africa.

The spikes happened exactly every 80 minutes and lasted for 10 minutes.

The payload of the traffic was junk, not a usual HTTP request attack.

The payload had uniformly distributed payload sizes.

It’s hard to draw conclusions, but we can imagine two possible scenarios. It is possible these spikes were an attack intended to break our HTTP servers.

A second possibility is that these spikes were legitimate connection attempts by some weird, obfuscated protocol. For some reason the clients were connecting to port 80/TCP and retried precisely every 80 minutes.

We are continuing our investigation. In the meantime we are looking for clues. Please do let us know if you have encountered this kind of TCP/IP payload. We are puzzled by these large spikes.

If you’d like to work on this type of problem we’re hiring in London, San Francisco, Austin, Champaign and Singapore.

Donate

Like most websites GIXtools uses cookies. In order to deliver a personalised, responsive service and to improve the site, we remember and store information about how you use it. This is done using simple text files called cookies which sit on your computer. These cookies are completely safe and secure and will never contain any sensitive information. Accept