As an international organization that disrupts spam operators, the Spamhaus Project has made its share of enemies. Many of those enemies possess the Internet equivalent of millions of water cannons that can be turned on in an instant to flood targets with more traffic than they can possibly stand.

On Tuesday, Spamhaus came under a torrential deluge—75 gigabits of junk data every second—making it impossible for anyone to access the group's website (the real-time blacklists that ISPs use to filter billions of spam messages were never effected). Spamhaus quickly turned to CloudFlare, a company that secures websites and helps mitigate the effects of distributed denial-of-service attacks.

This is a story about how the attackers were able to flood a single site with so much traffic, and the way CloudFlare blocked it using a routing methodology known as Anycast.

While attacks of 100Gbps aren't unheard of, the 75Gbps assault was still massive and generally well beyond what most botnets are capable of generating. To magnify their limited amount of bandwidth, the attackers resorted to what's known as DNS (domain name system) amplification—a technique that allows attackers to multiply their junk traffic by as much as 100 fold. As Ars explained in October, DNS amplification attacks work because companies such as AT&T, GoDaddy, SoftLayer, and Pakistan Telecom allow open DNS servers to run on the networks they operate instead of limiting them to just paying customers. DDoS attackers have abused these open DNS resolvers for years in a way that severely aggravates the effects of their crippling assaults.

As many Ars readers know, DNS servers are the Internet directories that translate domain names such as arstechnica.com into IP addresses such as 50.31.151.33. But DNS servers can also be queried for the IP addresses of huge swaths of the Internet, putting the person listed as making the request on the receiving end of a massive response. In a blog post published Wednesday, CloudFlare CEO Matthew Prince said each DNS request sent by the Spamhaus attackers was likely only 36 bytes long, while each response was about 3,000 bytes. By spoofing the requests to make them appear as if they originated with Spamhaus, the attackers can turn the firepower of all those networks against their opponent, all but guaranteeing it won't be available to process legitimate traffic.

To get Spamhaus back online, CloudFlare relied on Anycast, a routing technique that distributes the same IP address across 23 data centers across the world. Internet traffic almost always chooses the shortest physical path. Anycast allows the geographically dispersed junk traffic to be absorbed by dozens of individual centers, where each packet is then inspected. When it bears signatures found in the attack traffic—for example, if it's a 3,000-byte response from an open DNS resolver—it is discarded in the CloudFlare data center. Only Legitimate Web requests are allowed to be forwarded to the Spamhaus data center.

"When there's an attack, Anycast serves to effectively dilute it by spreading it across our facilities," Prince wrote. "Since every data center announced the same IP address for any CloudFlare customer, traffic cannot be concentrated in any one location. Instead of the attack being many-to-one, it becomes many-to-many with no single point on the network acting as a bottleneck."

Anycast made it easy for CloudFlare to filter out other types of malicious traffic directed at Spamhaus. The attackers also flooded the anti-spam service with huge numbers of spoofed packets bearing the ACK flag, which is the second part of the multi-step handshake computers on the Internet follow to establish connections.

"In an ACK reflection, the attacker sends a number of SYN packets to servers with a spoofed source IP address pointing to the intended victim," Prince wrote. "The servers then respond to the victim's IP with an ACK. Like the DNS reflection attack, this disguises the source of the attack, making it appear to come from legitimate servers."

The attacks are significantly easier to block since there's no amplification effect. CloudFlare drops each unmatched ACK.

Ironically, when CloudFlare blocks these types of attacks it routinely hears from network operators who complain that the service is attacking their systems with abusive DNS queries or SYN floods. And therein shows the work that remains to get the DoS problem under control. As effective as Anycast is at lessening the effects of denial of service attacks, it's akin to cough medicine that treats the symptom while doing nothing to cure the cold that causes it in the first place. As Ars learned first-hand last week, just about anyone can wield a DoS club that can make it impossible for legitimate traffic to get through. Ridding the Internet of the scourge will require a combination of education and pressure on network providers to prevent their infrastructure from attacking innocent bystanders.

Promoted Comments

The underlying problem here is that a problem we've known of for more than a decade has not yet been solved. And it's not a complicated one.

Best practice if you run an ISP is to not let any packets out of your network with an IP that don't belong to you or your customers. All the attacks mentioned in this article rely on ISPs not doing this. If the ISP lets someone send a packet with a source IP that doesn't belong on their network (like say, the spamhaus web server's IP), then that ISP is enabling the spoofing required for either the DNS amplification track or the ACK trick attacks to work.

The open DNS recursor thing is an issue as well, but it would not be able to be abused without spoofing being made so easy by ISP noobs.

The Internet needs to come up with a solution to this problems, because huge companies can afford that bandwidth and DDOS protections. The rest can´t.

Small companies and individuals cannot pay this protections, and even if they can, they can only protect themselves vs very small attacks. This basically kills the Internet as we know it, where mom and dad can put their online business from home. It leaves an Internet only for huge corporations that can afford 100 Gbit connections or medium ones that can afford the appliances that costs thousands of dollars.

Having the same or more connection to protect an attack is just ridiculous because it means everyone is vulnerable. I think the big ISP and Telecoms in the world should unite to try to kill this on destinations, I think one of the best solution today is Arbor Peaks.

Attacks should be cut and stopped from the originating networks. I know its rather hard to detect and stop, in particular because most are infected botnets but the bigger Internet speeds and unlimited caps will just make things worst. At least when transfer was limited, a user would suspect something if suddenly his transfer is way over the normal. ISP should try to detect this types of anomalies as well.

Part of this problems is that there are so many computers and even servers being part of a botnet, that all this users are in part responsible for this things, for not securing their systems enough.

Attacked destination should inform originating destination, and they should cut traffic off from this systems to that destination. Sadly, even Arbor Networks is not affordable for smaller business. Something like this should be cheap or so affordable that it should power the Internet, stopping attacks from routers and switches that talk together, in a smart way. Otherwise there is a dark future for the Internet, in particular because there is not punishment for the attackers, nobody knows who controls the bots, and even kids making small attacks think its fun to do so.

If everyone starts to DOS everyone else, there would be no Internet, we would be all neutralizing the other party all the time. I don´t understand how even the persons that do this, don´t see this. The spammers, or illegal groups that collaborate to create botnets and ddos attacks are putting themselves out of business as well, as they show how effective this attacks are that other criminals start to use them as well, even against them, and so a killing spree of attacks go from one side to the other. This must stop. Its causing a huge damage the way Internet works, to the point we are going to need to redesign the Internet, even ISPs maybe even introduce back transfer limits, or caps, or the whole Internet is going to get more controlled and everyone will be punished eventually. This goes against free and open.

Do they (the dossers) really want an Internet only with 4 websites left? Like Google, Amazon, Microsoft, etc. Little guys will take their sites down, or move to some Facebook page, if things continue like this. On the end, more websites disappear because they cannot afford protection and the Internet shrinks. This is awful, even the bad guys seem not to realize they are collaborating to shrink the Internet like that, and since I suspect more of their illegal activities involve something online, this can only damage themselves as well eventually.

34 Reader Comments

I wonder what the intent of this attack was. Just a rage attack? Did they really think they'd shut down spamhaus? It probably didn't even interrupt any spam blockage, since most ISP's grab the blacklist once and use it for days or weeks at a stretch.

I wonder what the intent of this attack was. Just a rage attack? Did they really think they'd shut down spamhaus? It probably didn't even interrupt any spam blockage, since most ISP's grab the blacklist once and use it for days or weeks at a stretch.

People sophisticated enough to pull this off, generally have the self control not to rage indiscriminately. They had a purpose, we just don't know what it it yet.

I wonder what the intent of this attack was. Just a rage attack? Did they really think they'd shut down spamhaus? It probably didn't even interrupt any spam blockage, since most ISP's grab the blacklist once and use it for days or weeks at a stretch.

I believe the article actually said the blacklists were not effected. It was probably just another example someone, or some group, doing this out of anger. It is nice to see a quick and legal course of action for companies to turn to when this sort of thing happens.

Irony? Wasn't CloudFlare, the "good guys" in this tale, host to the TwBooter source of the Krebs and Ars DDoS attacks? Their CDN is mentioned in that article (linked in this story). I love irony but, if I'm right about that, it might bear a mention here. Even the good guys can fire their weapons in any direction, even if unwittingly.

Irony? Wasn't CloudFlare, the "good guys" in this tale, host to the TwBooter source of the Krebs and Ars DDoS attacks? Their CDN is mentioned in that article (linked in this story). I love irony but, if I'm right about that, it might bear a mention here. Even the good guys can fire their weapons in any direction, even if unwittingly.

Irony? Wasn't CloudFlare, the "good guys" in this tale, host to the TwBooter source of the Krebs and Ars DDoS attacks? Their CDN is mentioned in that article (linked in this story). I love irony but, if I'm right about that, it might bear a mention here. Even the good guys can fire their weapons in any direction, even if unwittingly.

No, as CloudFlare is not a host, they are a CDN. Anyone can sign up for their free service, even douches like TwBooter. And CF had nothing to do with the DDoS, as they're an inbound mirror only.

Several of my sites have been happily humming along with their free CDN service for little over a year now. I cannot complain. I've had my sites "appear offline" once or twice for a moment, but aside from that, I've had completely uninterrupted service (with the added benefit of them blocking attempted hacks and DoS).

Irony? Wasn't CloudFlare, the "good guys" in this tale, host to the TwBooter source of the Krebs and Ars DDoS attacks? Their CDN is mentioned in that article (linked in this story). I love irony but, if I'm right about that, it might bear a mention here. Even the good guys can fire their weapons in any direction, even if unwittingly.

No, as CloudFlare is not a host, they are a CDN. Anyone can sign up for their free service, even douches like TwBooter. And CF had nothing to do with the DDoS, as they're an inbound mirror only.

Ah. Thanks. Perhaps that exonerates them. Sort of. If it all went out through a network that they control ... well, never mind, I'm not up on CDNs and the like. I'll edit my post to link to yours.

Irony? Wasn't CloudFlare, the "good guys" in this tale, host to the TwBooter source of the Krebs and Ars DDoS attacks? Their CDN is mentioned in that article (linked in this story). I love irony but, if I'm right about that, it might bear a mention here. Even the good guys can fire their weapons in any direction, even if unwittingly.

No, as CloudFlare is not a host, they are a CDN. Anyone can sign up for their free service, even douches like TwBooter. And CF had nothing to do with the DDoS, as they're an inbound mirror only.

Ah. Thanks. Perhaps that exonerates them. Sort of. If it all went out through a network that they control ... well, never mind, I'm not up on CDNs and the like. I'll edit my post to link to yours.

Whatever TwBooter uses to perform their attacks (ahem, 'totally legitimate and authorized tests') is likely completely and totally unrelated to CloudFlare's CDN, seeing as a CDN wouldn't work that way. Ars or Krebs likely could look at the list of IPs and see which hosts the floods were originating from, and then get a good idea of who TwBooter was using to launch their attacks (they could have bought legitimate hosting, or they could have a bot net).

CloudFlare is pretty much equivalent to archive.org or google's cache pages (in a very high level, rough, vague sort of way).

Aggravate: 1) to make worse or more severe; intensify, as anything evil, disorderly, or troublesome: to aggravate a grievance; to aggravate an illness.

Why would you go out of your way to call me on misuse of a word, anyway? Even if you're right, the comment seems mean-spirited and off-topic. But when every English dictionary in the world shows I used the word correctly . . . well, it just reminds me why I don't participate more often in the Ars discussions.

The underlying problem here is that a problem we've known of for more than a decade has not yet been solved. And it's not a complicated one.

Best practice if you run an ISP is to not let any packets out of your network with an IP that don't belong to you or your customers. All the attacks mentioned in this article rely on ISPs not doing this. If the ISP lets someone send a packet with a source IP that doesn't belong on their network (like say, the spamhaus web server's IP), then that ISP is enabling the spoofing required for either the DNS amplification track or the ACK trick attacks to work.

The open DNS recursor thing is an issue as well, but it would not be able to be abused without spoofing being made so easy by ISP noobs.

Interestingly, I came to ars right now because a website I was trying to hit shows as down, with a cloud flare logo. I wondered if ars had a report of cloud flare being down again, and saw this article.

Interestingly, I came to ars right now because a website I was trying to hit shows as down, with a cloud flare logo. I wondered if ars had a report of cloud flare being down again, and saw this article.

Why would you go out of your way to call me on misuse of a word, anyway?

Sorry Dan, but some of us (including myself) are extreme pedants. We get anxious when the English language is misused, and try to point out the error. We generally fail in personal communication skills, and so our comments are often not phrased to communicate in the manner in which they are intended.

That said, your use in this article is correct. But please, never use the word "machine" after the acronym "ATM" when referring to banking equipment. I get these twitches.

The Internet needs to come up with a solution to this problems, because huge companies can afford that bandwidth and DDOS protections. The rest can´t.

Small companies and individuals cannot pay this protections, and even if they can, they can only protect themselves vs very small attacks. This basically kills the Internet as we know it, where mom and dad can put their online business from home. It leaves an Internet only for huge corporations that can afford 100 Gbit connections or medium ones that can afford the appliances that costs thousands of dollars.

Having the same or more connection to protect an attack is just ridiculous because it means everyone is vulnerable. I think the big ISP and Telecoms in the world should unite to try to kill this on destinations, I think one of the best solution today is Arbor Peaks.

Attacks should be cut and stopped from the originating networks. I know its rather hard to detect and stop, in particular because most are infected botnets but the bigger Internet speeds and unlimited caps will just make things worst. At least when transfer was limited, a user would suspect something if suddenly his transfer is way over the normal. ISP should try to detect this types of anomalies as well.

Part of this problems is that there are so many computers and even servers being part of a botnet, that all this users are in part responsible for this things, for not securing their systems enough.

Attacked destination should inform originating destination, and they should cut traffic off from this systems to that destination. Sadly, even Arbor Networks is not affordable for smaller business. Something like this should be cheap or so affordable that it should power the Internet, stopping attacks from routers and switches that talk together, in a smart way. Otherwise there is a dark future for the Internet, in particular because there is not punishment for the attackers, nobody knows who controls the bots, and even kids making small attacks think its fun to do so.

If everyone starts to DOS everyone else, there would be no Internet, we would be all neutralizing the other party all the time. I don´t understand how even the persons that do this, don´t see this. The spammers, or illegal groups that collaborate to create botnets and ddos attacks are putting themselves out of business as well, as they show how effective this attacks are that other criminals start to use them as well, even against them, and so a killing spree of attacks go from one side to the other. This must stop. Its causing a huge damage the way Internet works, to the point we are going to need to redesign the Internet, even ISPs maybe even introduce back transfer limits, or caps, or the whole Internet is going to get more controlled and everyone will be punished eventually. This goes against free and open.

Do they (the dossers) really want an Internet only with 4 websites left? Like Google, Amazon, Microsoft, etc. Little guys will take their sites down, or move to some Facebook page, if things continue like this. On the end, more websites disappear because they cannot afford protection and the Internet shrinks. This is awful, even the bad guys seem not to realize they are collaborating to shrink the Internet like that, and since I suspect more of their illegal activities involve something online, this can only damage themselves as well eventually.

Why would you go out of your way to call me on misuse of a word, anyway?

Sorry Dan, but some of us (including myself) are extreme pedants. We get anxious when the English language is misused, and try to point out the error. We generally fail in personal communication skills, and so our comments are often not phrased to communicate in the manner in which they are intended.

That said, your use in this article is correct. But please, never use the word "machine" after the acronym "ATM" when referring to banking equipment. I get these twitches.

<pedantry>How do you know for sure that "ATM machine" isn't refering to a machine that uses Asynchronous Transfer Mode to communicate with other machines on the bank's network?</pedantry>

The underlying problem here is that a problem we've known of for more than a decade has not yet been solved. And it's not a complicated one.

Best practice if you run an ISP is to not let any packets out of your network with an IP that don't belong to you or your customers. All the attacks mentioned in this article rely on ISPs not doing this. If the ISP lets someone send a packet with a source IP that doesn't belong on their network (like say, the spamhaus web server's IP), then that ISP is enabling the spoofing required for either the DNS amplification track or the ACK trick attacks to work.

The open DNS recursor thing is an issue as well, but it would not be able to be abused without spoofing being made so easy by ISP noobs.

If everyone starts to DOS everyone else, there would be no Internet, we would be all neutralizing the other party all the time. I don´t understand how even the persons that do this, don´t see this.

Do they (the dossers) really want an Internet only with 4 websites left? Like Google, Amazon, Microsoft, etc. Little guys will take their sites down, or move to some Facebook page, if things continue like this. On the end, more websites disappear because they cannot afford protection and the Internet shrinks. This is awful, even the bad guys seem not to realize they are collaborating to shrink the Internet like that, and since I suspect more of their illegal activities involve something online, this can only damage themselves as well eventually.

Why would you go out of your way to call me on misuse of a word, anyway?

Sorry Dan, but some of us (including myself) are extreme pedants. We get anxious when the English language is misused, and try to point out the error. We generally fail in personal communication skills, and so our comments are often not phrased to communicate in the manner in which they are intended.

That said, your use in this article is correct. But please, never use the word "machine" after the acronym "ATM" when referring to banking equipment. I get these twitches.

The one that I noticed was the non-word "effected" (not in the quoted sentence) when it should have been the verb "affected." "The blacklists were not affected" would be a correct sentence, as would "there was no effect / were no effects on the blacklists." It's one of those most-misused words in English.

But yes, it's largely pedantic.

I, for one, always like reading your articles, Dan. Some of the best reporting on the site these days.

EDIT:And in my own pedantry I made a mistake: "effected" is, in fact, a word. It's a subtle difference in usage / form that I've rarely seen outside of German. Definitely one of those evil, subtle words. For lack of a better way to explain, it seems like "effect" as a verb refers to direct action (to effect a change means I make the change) whereas "affect" is indirect (I didn't make the change, but whatever I did do influenced that change). http://www.diffen.com/difference/Affected_vs_Effected

The underlying problem here is that a problem we've known of for more than a decade has not yet been solved. And it's not a complicated one.

Best practice if you run an ISP is to not let any packets out of your network with an IP that don't belong to you or your customers. All the attacks mentioned in this article rely on ISPs not doing this. If the ISP lets someone send a packet with a source IP that doesn't belong on their network (like say, the spamhaus web server's IP), then that ISP is enabling the spoofing required for either the DNS amplification track or the ACK trick attacks to work.

The open DNS recursor thing is an issue as well, but it would not be able to be abused without spoofing being made so easy by ISP noobs.

I agree with you that engineers of large networks should be implementing appropriate security measures to mitigate DDoS attacks using spoofed source IPs. However, reverse path filtering isn't always the best solution. It's also not as easy as you make it sound, and blaming the problem on ISPs isn't accurate. Global networks with multiple ingress / egress points through multiple ISPs can have major problems with RPF. For that matter, if ISPs enabled it globally then they would probably break quite a few of their customers who are load balancing across multiple providers. Both of these scenarios (and many others that can cause problems) can usually be worked around, but the point is implementing RPF isn't always as easy as typing a command.

With that said, RPF should definitely be used when it can be used. However, as a network engineer I can say first hand that it's very, very hard to get CIOs to sign off on implementing things that can cause problems unless they will see a direct benefit. Unfortunately many execs don't view blocking an attack against someone ELSE to be a benefit to them. They weigh the risk of their infrastructure being used in a DDoS attack against the risk of having to answer to the CEO or board if there's a major outage, and you can guess which one they usually choose. They know that they'll probably never take flak for their infrastructure being used in a DDoS attack so if it can handle the extra load then they tend to look the other way. (To be fair to CIOs I know that it doesn't always come down to answering to the board. If they're a good CIO then their first priority is to keep their own infrastructure running properly and they're not going to take risks that they don't think are necessary.)

Fortunately there's more than one way to skin this cat. RPF isn't the only solution. IDS/IPS can catch and eliminate DNS amplification and other types of attacks. Firewalls doing deep packet inspection can block certain types of attacks as well. ACLs on routers can also play their part. The problem in my view is that many engineers just aren't concerned with fixing the problem. We ALWAYS have something on our plate so it's easy to push network security to the back burner. Should it be that way? Absolutely not. I preach network security all the time both on and off the job. The company that I work for now is concerned about it and that's great, but some of the companies I've worked for in the past just haven't been interested.

I agree with you that engineers of large networks should be implementing appropriate security measures to mitigate DDoS attacks using spoofed source IPs. However, reverse path filtering isn't always the best solution. It's also not as easy as you make it sound, and blaming the problem on ISPs isn't accurate. Global networks with multiple ingress / egress points through multiple ISPs can have major problems with RPF. For that matter, if ISPs enabled it globally then they would probably break quite a few of their customers who are load balancing across multiple providers.

uRPF has been out for over a decade. I don't think the problem is with the large providers either - they know how to run a network, and they understand the nuances involved when dealing with peering and multi-homed customers. It's a solved problem.

I suspect we're mostly talking about large hosting operations that are large enough to demand that their upstreams "don't filter me bro, we know what we're doing", yet still small enough to not have a really good network engineering group. The sad part being is that some random colo provider with two upstreams and aging 6500 series gear could totally mitigate their own mess with no impact - an isolated network with a handful of upstreams and no downstream transit customers is low hanging fruit...