It's clobberin' time —

Biggest DDoS ever aimed at Cloudflare’s content delivery network

Network Time Protocol attack reached 400Gbps.

A distributed denial-of-service attack targeting a client of the content delivery network Cloudflare reached new highs in malicious traffic today, striking at the company’s data centers in Europe and the US. According to a Twitter post by Cloudflare CEO Matthew Prince, the full volume of the attack exceeded 400 gigabits per second—making it the largest DDoS attack ever recorded.

The attack used Network Time Protocol (NTP) reflection, the same technique used in recent attacks against gaming sites by a group called DERP Trolling. NTP is used to synchronize the time settings on computers across the Internet. The attack made fraudulent synchronization requests to NTP servers that caused them to send a flood of replies back at the targeted sites.

Reflection attacks have been a mainstay of DDoS tools and botnets, but the use of NTP in such attacks is relatively new. Last year’s attack on Spamhaus, which previously set the record for the largest DDoS ever, used a Domain Name Service (DNS) protocol attack—a much more common approach that takes advantage of the Internet’s directory service, forging requests for DNS lookups from the intended target and sending them to scores of open DNS servers. The size of the traffic directed back at the target from these requests far exceeds the size of the requests sent to the DNS servers, which is why the technique is often called a DNS amplification attack.

By comparison, NTP sends much smaller amounts of data in response to requests. But as efforts have been made to prevent DNS amplification attacks by reducing the number of open DNS servers available to attackers, there are over 3,000 active public time servers configured to reply to NTP requests, as well as many more time servers on smaller networks that may be open to outside requests.

Further, a recently discovered vulnerability in NTP allows for amplification attacks similar to those previously performed with DNS, exploiting a command in the protocol called “monlist” that sends the IP addresses of the last 600 devices connected to the server. These requests, sent via a packet with the forged address of the victim, send a torrent of data back at the targeted site. Like DNS reflection attacks, NTP attacks can be diminished in effectiveness by network operators if they configure firewalls to block external requests.

59 Reader Comments

This problem will not go away until routers start dropping packets with a return address that is not on the router's own subnet. It's as simple as that. Why on earth we haven't tuned the Internet to do this already is beyond me.

This is being done already, but its not mandatory, its part of "Best ISP´s" practices by IETF- Filter as close to edge as possible- Filter all private subnets- Let pass only subnets configured on clientsetc, etc, etc

Internet is a wild jungle, routing is not optimal and there is no 1 way path for a subnet therefore you cant use something like uRPF. With current internet configuration (BGP) there is no way of "tuning" it without losing capabilities/redundancy

You can use uRPF. That will look at your internal networks and the source IP of the packet heading out and say "Could this have come from this side?" and if it could, it sends it on. It also checks to see if it is an internet address that couldn't have come from that side at all and drop it if it is. For packets coming in from the internet, it will look and say "Could this actually have come from the internet?" and if it could, let it in. It doesn't really rely on routing tables being perfect of configuration of uRPF in any way. But not every device supports uRPF.

The ISPs should be doing this as the problem here is all about spoofed addresses.

The attacker sends a packet out with a forged 'from' address to an NTP server which respond with many more packets to that address which is a cloud flare one, so you see thata small amount of attacker bandwidth is multiplied into a huge attack.

The target firewall cannot do anything about this as it's too late when it gets it, the bandwidth is already used up and you can't just block these NTP packets as you usually want them and you can't tell good ones from these bad ones.

The only solution is for ISPs to implement egress filters on their bgp routers to discard any packet that does not come from their own network - it can't be a valid packet in this case. I read one anon who said such egress filters were #1 in his list. I can't see why the big ISPs don't do more of this to prevent such abuse of what is effectively their network, unless they get paid for the bandwidth that is used... but even then, you'd think they'd rather charge for Netflix traffic than the scammers.

I come from a technical background and, even to me, when I see numbers like this or the energy stored in the LHC magnets, it might as well be measured in units of chocolate per gram of water.

It is incomprehensibly huge. If you have a 20 megabit down connection in your home, a 500 gigabit attack would be theoretically enough to saturate your connection 25600 times over.

"It is incomprehensibly huge."

Then you'll not fully appreciate the 1pb/s fiber that they have working, but only about 50 miles right now. But you can enjoy 16tb/s fiber right now! For the low prices of $(NDA redacted). Well, it can be purchased and is production ready, but if you have to ask the price, you can't afford it.

We can transfer several factors more data over fiber than what they thought theoretically possibly only 10 years ago, and most of these advancements have happened in the past 3 years.

The only solution is for ISPs to implement egress filters on their bgp routers to discard any packet that does not come from their own network - it can't be a valid packet in this case. I read one anon who said such egress filters were #1 in his list. I can't see why the big ISPs don't do more of this to prevent such abuse of what is effectively their network, unless they get paid for the bandwidth that is used... but even then, you'd think they'd rather charge for Netflix traffic than the scammers.

...Because if I host a server, I want people on other ISPs to be able to access it?

The only solution is for ISPs to implement egress filters on their bgp routers to discard any packet that does not come from their own network - it can't be a valid packet in this case. I read one anon who said such egress filters were #1 in his list. I can't see why the big ISPs don't do more of this to prevent such abuse of what is effectively their network, unless they get paid for the bandwidth that is used... but even then, you'd think they'd rather charge for Netflix traffic than the scammers.

...Because if I host a server, I want people on other ISPs to be able to access it?

Why would you be hosting servers that send spoofed packets? What legal benefit would that give you?

You don't understand, if you host a server it gets an IP address. That IP will be assigned to you from one of the pool your ISP owns. So any packets it sends out will have your (ie one of their) IP addresses in its packet. So the ISP can tell that it's legit. If you forged the packet to pretend to be from a different IP then they know it's not from their network or is for nefarious purposes and can block it.

If you run a server you cannot just put any old address on it. Note that this is only outgoing packets, the ISP cannot tell if incoming ones are valid or not and have to take the routing information on trust, which is why these ddos attacks can happen.

Uh huh. And I got called out as paranoid that the end of XP support will degrade the health of the net even for those not running windows earlier today. We already have a problem - this is an illustration of just how bad the problem is - and if even a fraction of the existing boxes running XP is compromised the situation will get worse still.

You don't understand, if you host a server it gets an IP address. That IP will be assigned to you from one of the pool your ISP owns. So any packets it sends out will have your (ie one of their) IP addresses in its packet. So the ISP can tell that it's legit. If you forged the packet to pretend to be from a different IP then they know it's not from their network or is for nefarious purposes and can block it.

If you run a server you cannot just put any old address on it. Note that this is only outgoing packets, the ISP cannot tell if incoming ones are valid or not and have to take the routing information on trust, which is why these ddos attacks can happen.

Ah apologies, I took your comment to mean that an ISP should discard any packets on it's network with an origin from outside their address range.

What I want to know is if CloudFlare actually handled it or if the DDOS took down the site and/or a CF node.

Not according to their status page. 400Gbps is not much across all their locations, even the largest might have seen only 50Gbps or so, where CF have I believe 100-150Gbps at each location (well at the larger ones, some of the smaller regional ones are only 30-50Gbps capacity).

The real issue with 400Gbps is if somehow it manages to be from a few source tier1s, whereby it could easily flood some peering points (which usually only have 50-100Gbps capacity).

How is it possible? Assuming, on average, a residential consumer (most likely target to become part of a botnet) has a 1 Mbps uplink. That means, there have to be at least 20'000 'participating' hosts in this attack.

I think an average of 1 Mbps is far too low. Even before I got my current 1 Gbps connection, I was getting way more that. You only need 1in 10 people to have a 20 Mbps uplink to ensure the average is over 2 Mbps.

The people responsible for these kinds of attack don't face any negative consequences that I am aware of. With the NSA's massive network monitoring capacity surely they would be able to pinpoint the culprits and dispatch drones.

How awful would it be for someone like Cloudflare to "send some traffic back" as a way of getting the attention of the NTP server operators who are doing it wrong and allow such attacks?

Very wrong - it would make them as guilty of network abuse as the DDoS perpetrators - in any case those NTP servers are already getting hammered. What I hope Cloudflare did was to not all the NTP servers involved then after the fact contact their sysadmins and let them know how to fix it.

Cloudflare has this story, expounded in that other unsaid tech news aggregator, then a story on what a how they implement TLS, blah blah blah contradiction in security skills...

What do you think is more embarassing:1) Not knowing how to use your brand-spanking new Juniper perimeter security which causes an external botnet DDoS2) Lack of office space LAN firewalling so when a guy gets fired, with topology knowledge, deploys the botnet from workstations

You do not have to clog the bandwidth, which should be un-do-able with server farm security.You just have to forcefully convince a freaked out admin that the only immediate solution is hit the kill switch on the virtual server. Most of the time, it is less about your hardware throughput than the server/app capability. More importantly, over 75% of your security breaches and data integrity faults are internal sourced. If the story was not posted about some hacker commando at a coffee shop, we would be talking in detail about hypervisor security, not a broken protocol we have known about for a while.

I have to call Cloudflare on this one. Sue me for defamation. I would almost bet money they really fired the hacker. With either option above, how are they saving face? This is like Microsoft threatening 2048-bit pipe encryption after Snowden, when their OS sucks like a lung puncture, NTFS still exists with alternate data streams. People giving themselves jobs in security with holes they deploy as a sales pitch...