As an international organization that disrupts spam operators, the Spamhaus Project has made its share of enemies. Many of those enemies possess the Internet equivalent of millions of water cannons that can be turned on in an instant to flood targets with more traffic than they can possibly stand.

On Tuesday, Spamhaus came under a torrential deluge—75 gigabits of junk data every second—making it impossible for anyone to access the group's website (the real-time blacklists that ISPs use to filter billions of spam messages were never effected). Spamhaus quickly turned to CloudFlare, a company that secures websites and helps mitigate the effects of distributed denial-of-service attacks.

This is a story about how the attackers were able to flood a single site with so much traffic, and the way CloudFlare blocked it using a routing methodology known as Anycast.

While attacks of 100Gbps aren't unheard of, the 75Gbps assault was still massive and generally well beyond what most botnets are capable of generating. To magnify their limited amount of bandwidth, the attackers resorted to what's known as DNS (domain name system) amplification—a technique that allows attackers to multiply their junk traffic by as much as 100 fold. As Ars explained in October, DNS amplification attacks work because companies such as AT&T, GoDaddy, SoftLayer, and Pakistan Telecom allow open DNS servers to run on the networks they operate instead of limiting them to just paying customers. DDoS attackers have abused these open DNS resolvers for years in a way that severely aggravates the effects of their crippling assaults.

As many Ars readers know, DNS servers are the Internet directories that translate domain names such as arstechnica.com into IP addresses such as 50.31.151.33. But DNS servers can also be queried for the IP addresses of huge swaths of the Internet, putting the person listed as making the request on the receiving end of a massive response. In a blog post published Wednesday, CloudFlare CEO Matthew Prince said each DNS request sent by the Spamhaus attackers was likely only 36 bytes long, while each response was about 3,000 bytes. By spoofing the requests to make them appear as if they originated with Spamhaus, the attackers can turn the firepower of all those networks against their opponent, all but guaranteeing it won't be available to process legitimate traffic.

To get Spamhaus back online, CloudFlare relied on Anycast, a routing technique that distributes the same IP address across 23 data centers across the world. Internet traffic almost always chooses the shortest physical path. Anycast allows the geographically dispersed junk traffic to be absorbed by dozens of individual centers, where each packet is then inspected. When it bears signatures found in the attack traffic—for example, if it's a 3,000-byte response from an open DNS resolver—it is discarded in the CloudFlare data center. Only Legitimate Web requests are allowed to be forwarded to the Spamhaus data center.

"When there's an attack, Anycast serves to effectively dilute it by spreading it across our facilities," Prince wrote. "Since every data center announced the same IP address for any CloudFlare customer, traffic cannot be concentrated in any one location. Instead of the attack being many-to-one, it becomes many-to-many with no single point on the network acting as a bottleneck."

Anycast made it easy for CloudFlare to filter out other types of malicious traffic directed at Spamhaus. The attackers also flooded the anti-spam service with huge numbers of spoofed packets bearing the ACK flag, which is the second part of the multi-step handshake computers on the Internet follow to establish connections.

"In an ACK reflection, the attacker sends a number of SYN packets to servers with a spoofed source IP address pointing to the intended victim," Prince wrote. "The servers then respond to the victim's IP with an ACK. Like the DNS reflection attack, this disguises the source of the attack, making it appear to come from legitimate servers."

The attacks are significantly easier to block since there's no amplification effect. CloudFlare drops each unmatched ACK.

Ironically, when CloudFlare blocks these types of attacks it routinely hears from network operators who complain that the service is attacking their systems with abusive DNS queries or SYN floods. And therein shows the work that remains to get the DoS problem under control. As effective as Anycast is at lessening the effects of denial of service attacks, it's akin to cough medicine that treats the symptom while doing nothing to cure the cold that causes it in the first place. As Ars learned first-hand last week, just about anyone can wield a DoS club that can make it impossible for legitimate traffic to get through. Ridding the Internet of the scourge will require a combination of education and pressure on network providers to prevent their infrastructure from attacking innocent bystanders.

Promoted Comments

The underlying problem here is that a problem we've known of for more than a decade has not yet been solved. And it's not a complicated one.

Best practice if you run an ISP is to not let any packets out of your network with an IP that don't belong to you or your customers. All the attacks mentioned in this article rely on ISPs not doing this. If the ISP lets someone send a packet with a source IP that doesn't belong on their network (like say, the spamhaus web server's IP), then that ISP is enabling the spoofing required for either the DNS amplification track or the ACK trick attacks to work.

The open DNS recursor thing is an issue as well, but it would not be able to be abused without spoofing being made so easy by ISP noobs.

The Internet needs to come up with a solution to this problems, because huge companies can afford that bandwidth and DDOS protections. The rest can´t.

Small companies and individuals cannot pay this protections, and even if they can, they can only protect themselves vs very small attacks. This basically kills the Internet as we know it, where mom and dad can put their online business from home. It leaves an Internet only for huge corporations that can afford 100 Gbit connections or medium ones that can afford the appliances that costs thousands of dollars.

Having the same or more connection to protect an attack is just ridiculous because it means everyone is vulnerable. I think the big ISP and Telecoms in the world should unite to try to kill this on destinations, I think one of the best solution today is Arbor Peaks.

Attacks should be cut and stopped from the originating networks. I know its rather hard to detect and stop, in particular because most are infected botnets but the bigger Internet speeds and unlimited caps will just make things worst. At least when transfer was limited, a user would suspect something if suddenly his transfer is way over the normal. ISP should try to detect this types of anomalies as well.

Part of this problems is that there are so many computers and even servers being part of a botnet, that all this users are in part responsible for this things, for not securing their systems enough.

Attacked destination should inform originating destination, and they should cut traffic off from this systems to that destination. Sadly, even Arbor Networks is not affordable for smaller business. Something like this should be cheap or so affordable that it should power the Internet, stopping attacks from routers and switches that talk together, in a smart way. Otherwise there is a dark future for the Internet, in particular because there is not punishment for the attackers, nobody knows who controls the bots, and even kids making small attacks think its fun to do so.

If everyone starts to DOS everyone else, there would be no Internet, we would be all neutralizing the other party all the time. I don´t understand how even the persons that do this, don´t see this. The spammers, or illegal groups that collaborate to create botnets and ddos attacks are putting themselves out of business as well, as they show how effective this attacks are that other criminals start to use them as well, even against them, and so a killing spree of attacks go from one side to the other. This must stop. Its causing a huge damage the way Internet works, to the point we are going to need to redesign the Internet, even ISPs maybe even introduce back transfer limits, or caps, or the whole Internet is going to get more controlled and everyone will be punished eventually. This goes against free and open.

Do they (the dossers) really want an Internet only with 4 websites left? Like Google, Amazon, Microsoft, etc. Little guys will take their sites down, or move to some Facebook page, if things continue like this. On the end, more websites disappear because they cannot afford protection and the Internet shrinks. This is awful, even the bad guys seem not to realize they are collaborating to shrink the Internet like that, and since I suspect more of their illegal activities involve something online, this can only damage themselves as well eventually.