The general idea behind all amplification attacks is the same. An IP-spoofing capable attacker sends forged requests to a vulnerable UDP server. The UDP server, not knowing the request is forged, politely prepares the response. The problem happens when thousands of responses are delivered to an unsuspecting target host, overwhelming its resources - most typically the network itself.

Amplification attacks are effective, because often the response packets are much larger than the request packets. A carefully prepared technique allows an attacker with limited IP spoofing capacity (such as 1Gbps) to launch very large attacks (reaching 100s Gbps) "amplifying" the attacker's bandwidth.

Memcrashed

Obscure amplification attacks happen all the time. We often see "chargen" or "call of duty" packets hitting our servers.

A discovery of a new amplification vector though, allowing very great amplification, happens rarely. This new memcached UDP DDoS is definitely in this category.

The number of memcached attacks was relatively flat, until it started spiking just a couple days ago. Our charts also confirm this, here are attacks in packets per second over the last four days:

While the packets per second count is not that impressive, the bandwidth generated is:

At peak we've seen 260Gbps of inbound UDP memcached traffic. This is massive for a new amplification vector. But the numbers don't lie. It's possible because all the reflected packets are very large. This is how it looks in tcpdump:

The majority of packets are 1400 bytes in size. Doing the math 23Mpps x 1400 bytes gives 257Gbps of bandwidth, exactly what the chart shows.

Memcached does UDP?

I was surprised to learn that memcached does UDP, but there you go! The protocol specification shows that it's one of the best protocols to use for amplification ever! There are absolutely zero checks, and the data WILL be delivered to the client, with blazing speed! Furthermore, the request can be tiny and the response huge (up to 1MB).

Launching such an attack is easy. First the attacker implants a large payload on an exposed memcached server. Then, the attacker spoofs the "get" request message with target Source IP.

15 bytes of request triggered 134KB of response. This is amplification factor of 10,000x! In practice we've seen a 15 byte request result in a 750kB response (that's a 51,200x amplification).

Source IPs

The vulnerable memcached servers are all around the globe, with higher concentration in North America and Europe. Here is a map of the source IPs we've seen in each of our 120+ points of presence:

Interestingly our datacenters in EWR, HAM and HKG see disproportionally large numbers of attacking IPs. This is because most of the vulnerable servers are located in major hosting providers. The AS numbers of the IPs that we've seen:

Most of the memcached servers we've seen were coming from AS16276 - OVH, AS14061 - Digital Ocean and AS7684 - Sakura.

In total we've seen only 5,729 unique source IPs of memcached servers. We're expecting to see much larger attacks in future, as Shodan reports 88,000 open memcached servers:

Let's fix it up

It's necessary to fix this and prevent further attacks. Here is a list of things that should be done.

Memcached Users

If you are using memcached, please disable UDP support if you are not using it. On memcached startup you can specify --listen 127.0.0.1 to listen only to localhost and -U 0 to disable UDP completely. By default memcached listens on INADDR_ANY and runs with UDP support ENABLED. Documentation:

Internet Service Providers

In order to defeat such attacks in future, we need to fix vulnerable protocols and also IP spoofing. As long as IP spoofing is permissible on the internet, we'll be in trouble.

Help us out by tracking who is behind these attacks. We must know not who has problematic memcached servers, but who sent them queries in the first place. We can't do this without your help!

Developers

Please please please: Stop using UDP. If you must, please don't enable it by default. If you do not know what an amplification attack is I hereby forbid you from ever typing SOCK_DGRAM into your editor.

We've been down this road so many times. DNS, NTP, Chargen, SSDP and now memcached. If you use UDP, you must always respond with strictly a smaller packet size then the request. Otherwise your protocol will be abused. Also remember that people do forget to set up a firewall. Be a nice citizen. Don't invent a UDP-based protocol that lacks authentication of any kind.

That's all

It's anyone's guess how large the memcached attacks will become before we clean the vulnerable servers up. There were already rumors of 0.5Tbps amplifications in the last few days, and this is just a start.

Finally, you are OK if you are a Cloudflare customer. Cloudflare's Anycast architecture works well to distribute the load in case of large amplification attacks, and unless your origin IP is exposed, you are safe behind Cloudflare.

Prologue

A comment (below) points out that the possibility of using memcached for DDoS was discussed in a 2017 presentation.

Update
We received a word from Digital Ocean, OVH, Linode and Amazon that they tackled the memcached problem, their networks should not be a vector in future attacks. Hurray!

Dealing with DDoS attacks sound interesting? Join our world famous team in London, Austin, San Francisco and our elite office in Warsaw, Poland.

Over the past 6 months, volumetric DDoS attacks decreased by 82%. However volumetric attacks are not gone yet, still peaking as high as 418 Gbps. In this post, we cover the changing attack landscape and how Cloudflare adapts and evolves to keep the internet safe....

Back in March 2019, we released Firewall Analytics which provides insights into HTTP security events across all of Cloudflare's protection suite; Firewall rule matches, HTTP DDoS Attacks, Site Security Level which harnesses Cloudflare's threat intelligence, and more....