The story behind the 300Gb/s attack on an anti-spam organization.

Over the last ten days, a series of massive denial-of-service attacks has been aimed at Spamhaus, a not-for-profit organization that describes its purpose as "track[ing] the Internet's spam operations and sources, to provide dependable realtime anti-spam protection for Internet networks." These attacks have grown so large—up to 300Gb/s—that the volume of traffic is threatening to bring down core Internet infrastructure.

The New York Times reported recently that the attacks came from a Dutch hosting company called CyberBunker (also known as cb3rob), which owns and operates a real military bunker and which has been targeted in the past by Spamhaus. The spokesman who the NYT interviewed, Sven Olaf Kamphuis, has since posted on his Facebook page that CyberBunker is not orchestrating the attacks. Kamphuis also claimed that NYT was plumping for sensationalism over accuracy.

Sven Olaf Kamphuis is, however, affiliated with the newly organized group "STOPhaus." STOPhaus claims that Spamhaus is "an offshore criminal network of tax circumventing self declared internet terrorists pretending to be 'spam' fighters" that is "attempt[ing] to control the internet through underhanded extortion tactics."

STOPhaus claims to have the support of "half the Russian and Chinese Internet industry." It wants nothing less than to put Spamhaus out of action, and it looks like it's not too picky about how that might be accomplished. And if Spamhaus won’t back down, Kamphuis has made clear that even more data can be thrown at the anti-spammers.

Escalation

Hating Spamhaus has a long history.

Spamhaus is a nonprofit organization based in London and Geneva that was started in 1998 as a way of combating the escalating spam problem. The group doesn't block any data itself, but it does operate a number of blacklist services used by others to block data.

The first of these was the Spamhaus Block List (SBL), a database of IP addresses known to be spam originators. E-mail servers can query the SBL for each incoming e-mail to see if the connection is being made from an IP address in the database. If it is, they can reject the connection as being a probable cause of spam.

SBL tended to be filled with machines that were, for one reason or another, operating as open relays. The protocol used for sending e-mail, SMTP (Simple Mail Transfer Protocol) has a feature that nowadays might be considered rather undesirable: in principle, any SMTP server can be used to send e-mail from any sender to any recipient. If the SMTP server isn't responsible for the message box that a mail is being sent to, it should look up the server that is responsible for the message box and forward the message on to that server, a process called "relaying," and servers that operate in this way are “open” relays.

This is great for spammers. They can use a bogus address for the sender and the victim's address for the recipient, then use any open relay to actually send that message. The open relay will then find the real recipient server and forward the message.

This is obviously undesirable, so most SMTP servers these days apply additional rules. For example, ISP-operated mail servers will often operate as relays, but with some restrictions: they'll only allow relaying if the connection is being made from an IP address that belongs to the ISP. Or they will require a username and password to access.

As awareness of the problem of open relays has grown and the number of useful open relays has dropped, spammers have moved to other approaches. Instead of sending mail through a relay, they more commonly send it from machines they control directly to the recipient's mail server.

Enlarge/ Spammers may hate them, but some people quite approve of the work Spamhaus does.

One way they do this is with compromised PCs organized in botnets. The command and control servers direct the PCs in the botnets to send spam, and so the spam originates from hundreds of thousands or millions of compromised home and office PCs. This is why the destruction of large botnets often results in a drop in the number of spam messages sent.

To counter this kind of thing, some blacklist operators operate blacklists of "client" IP addresses, addresses used by consumer-focused ISPs that, for the most part, shouldn't be directly sending mail at all (instead, they should be routing mail through their ISPs' respective mail relays). Spamhaus operates such a list, separate from the SBL, calling it the Policy Block List. Spamhaus also has a database of compromised machines, the Exploits Block List, that lists hijacked machines running spam-related malware.

Spamhaus has a number of criteria that can result in an IP address being listed in its database. The organization has a number of Spamtrap e-mail addresses; addresses which won't ever receive legitimate mail (because nobody actually uses them). This is the most obvious source of IP addresses, and probably the least controversial—if an IP address sends spam to an inbox, it's fair game to regard that IP address as a spam source.

Spamhaus also blocks "spam operations," which is to say companies it believes make a business of sending spam. It lists these in its Register of Known Spam Operations (ROKSO), and it will pre-emptively blacklist IP addresses used by these groups. (Spamhaus will blacklist "spam support services"—ISPs and other service operators known to be spam friendly, for example by offering Web hosting to spammers, hosting spam servers, or selling spam software.)

The organization's most severe measure is its DROP ("don't route or peer") list. The DROP list is a list of IP address blocks that are controlled by criminals and spammers. Routers can use these to block all traffic from these IP ranges. Rather than using DNS, this list is distributed as a text file, for manual configuration, and using the BGP protocol, for routers to use directly.

In addition to the lists it maintains and the different inclusion criteria, Spamhaus has one particularly important policy: escalation. Repeated infringement—such as an ISP that refuses to terminate the service of spammers on its network—will see Spamhaus move beyond blacklisting individual IP addresses and start blacklisting ranges. If behavior still isn't improved, Spamhaus will block ever-larger ranges.

Nice series of articles, Peter & Ars. Thanks for the coverage! If nothing else, the issues of open DNS and SMTP servers are huge ones that deserve far more public-facing coverage and consumer enlightenment. People in the industry often discuss such things, openly, but sometimes a bit of exposure and public outcry can work wonders.

To me Spamhaus is not evil at all. They maintain lists. One list has spammers, another spam supporters. The companies involved here intentionally choose on there own accord to end up on the spam supporter list. There is nothing inherently evil with publicizing that fact. One could even argue it is a free speech matter.

Lastly, there is no law that says you have to accept communications from a spam supporter. I would argue it is completely within your rights as a mail provider to ignore communication from such bad agents,

In short, if you do not want to be known as a spam supporter, don't support spam.

I have little sympathy for the spammers because what they are doing is extremely aggravating from a user's point of view. Cluttering up inboxes with garbage is the reason they getting black listed and companies that support them (tacitly or actively) should realize most do not want spam in their inbox, ideally no - ever.

Whether Spamhaus is heavy handed at times to make a point is not a big issue to most users. If being heavy handed at times is necessary then I suspect most users will side with Spamhaus.

STOPhaus claims to have the support of "half the Russian and Chinese Internet industry.

So... the spammers?

Exactly!

We use Spamhaus blacklist for our Exchange e-mail server infrastructure. It is rejecting an ungodly amount of messages in any given 24hr period. It's crazy.

Spamhaus is sort of an unsung hero around here. Users get to see messages rejected by content filters because those go to their junk mail folder. Vast majority of rejected spam is rejected at the perimeter thanks to the Spamhaus blacklist and no one ever gets to see that. It only shows up in my logs.

Good article, except for the parroting of the "Almost took down the internet" line. It looks like it originated at the anti-DDoS organization Spamhaus hired, and 300Gps is not enough to affect a global internet that regularly handles 2+ Tbps.

The capacity of the "global Internet" is basically irrelevant. Traffic doesn't go over "the global Internet" as if it were some single aggregated network. It goes over a whole bunch of point-to-point links. LINX, the IX that suffered serious problems (but conveniently prior to the 24 hours that Gizmodo looked at) uses predominantly 10gigE connections between peers (100gigE is out there, but for most people, prohibitively expensive). Saturating these connections, thereby functionally breaking the routes between various parts of the Internet, is within reach of DDoS attacks like this.

Indeed, it seems that's exactly what happened on Saturday, when LINX's connectivity and routing plummeted.

Of course, there are generally other routes available; that's a big part of the Internet's design. "routing around failure" and all that. But breaking IXes is a big deal.

Peter, so if I understand correctly, Spamhaus is the "de-facto" provider or email filters and blacklist to the ISP and other industries as a whole? Basically are they an industry adopted solution?

I have no problem with what Spamhaus does, and a well needed service it is, but I do worry when a third party might exercise that much control without a lawful review process. I don't endorse the DDoS in any way. But does the good of Spamhaus out weigh the bad? I can dig it if it does, but I would like some thoughts on this.

Good article, except for the parroting of the "Almost took down the internet" line. It looks like it originated at the anti-DDoS organization Spamhaus hired, and 300Gps is not enough to affect a global internet that regularly handles 2+ Tbps.

The capacity of the "global Internet" is basically irrelevant. Traffic doesn't go over "the global Internet" as if it were some single aggregated network. It goes over a whole bunch of point-to-point links. LINX, the IX that suffered serious problems (but conveniently prior to the 24 hours that Gizmodo looked at) uses predominantly 10gigE connections between peers (100gigE is out there, but for most people, prohibitively expensive). Saturating these connections, thereby functionally breaking the routes between various parts of the Internet, is within reach of DDoS attacks like this.

Indeed, it seems that's exactly what happened on Saturday, when LINX's connectivity and routing plummeted.

Of course, there are generally other routes available; that's a big part of the Internet's design. "routing around failure" and all that. But breaking IXes is a big deal.

Was going to post something similar. And even if a certain centre can handle 2.5tb/s another 300gb/s is quite a lot. And it's all going in one direction.

Do you mean open as in accepts queries for domains it is not authoritative for from clients outside of its network? I'm just looking for clarification, as in maybe you know something I don't and I want to know what that something is. I did the obligatory googling but the hits all seem to be related to public DNS services which I suspect are the problems you're pointing out.

Spamhaus asked A2B to cut off CyberBunker's network. A2B didn't, so in October 2011, Spamhaus added a range of 2,048 A2B-owned addresses to its blacklist.

Was this range limited to those IP addresses that had been spamming or did they block various legitimate users too?

If they are blocking legitimate users to force A2B to do their bidding without further justification then they are just abusing their power. I have no sympathy for spammers but damaging someone's business / blocking legitimate users just because someone doesn't do what you say right away is extortion in my opinion.

STOPhaus claims to have the support of "half the Russian and Chinese Internet industry.

So... the spammers?

Exactly!

We use Spamhaus blacklist for our Exchange e-mail server infrastructure. It is rejecting an ungodly amount of messages in any given 24hr period. It's crazy.

Spamhaus is sort of an unsung hero around here. Users get to see messages rejected by content filters because those go to their junk mail folder. Vast majority of rejected spam is rejected at the perimeter thanks to the Spamhaus blacklist and no one ever gets to see that. It only shows up in my logs.

Thank you for the info, I was writing my post before I saw yours. I wanted to get a feel on how Spamhaus was actually used and perceived by the folks using it on the ground. I still would like to see some Devil's advocate post though.

Peter, so if I understand correctly, Spamhaus is the "de-facto" provider or email filters and blacklist to the ISP and other industries as a whole? Basically are they an industry adopted solution?

I have no problem with what Spamhaus does, and a well needed service it is, but I do worry when a third party might exercise that much control without a lawful review process. I don't endorse the DDoS in any way. But does the good of Spamhaus out way the bad? I can dig it if it does, but I would like some thoughts on this.

That was my thought too. It is a little worrying for a single organization to have that much power without any over sight. On the other hand, they seem to be doing a good job so far and a massive internet crippling attack isn't a solution.

From what I can see, Spamhaus only provides a service. Just like any other service company. Nobody HAS to use their blacklists. But people CHOOSE to use them because of their successes at blocking spam.

I can't see how STOPhaus could have a valid case against them, legal or otherwise.

Spamhaus asked A2B to cut off CyberBunker's network. A2B didn't, so in October 2011, Spamhaus added a range of 2,048 A2B-owned addresses to its blacklist.

Was this range limited to those IP addresses that had been spamming or did they block various legitimate users too?

If they are blocking legitimate users to force A2B to do their bidding without further justification then they are just abusing their power. I have no sympathy for spammers but damaging someone's business / blocking legitimate users just because someone doesn't do what you say right away is extortion in my opinion.

Spamhaus does not BLOCK anyone. They only provide address lists of known and suspected spammers. It is up to local administrators whether or not they want to use that list.

I would like to encourage Ars to provide an in-depth article on the following:

Quote:

The specific issues are ISPs that allow forged traffic to leave their networks—something which has little good justification to permit

Specifically, let's push ISPs to implement such rules. At the very least, it would result in a decrease in traffic, which would reduce the load on their internal networks, which would mean they don't need to upgrade their equipment as rapidly and customers would experience better service.

I don't understand what you mean by this one though:

Quote:

—and open DNS servers that can be used to generate large responses in response to forged IP traffic

Do you mean people shouldn't be running DNS servers? Or that those DNS servers shouldn't be answering to the public? That orgs should outsource their DNS for their domains and provide DNS servers only for internal use?

I don't like Spamhaus's going nuclear all the time. It seems that they will block legitimate traffic in attempts to stop hosters to drop the spamming IP's.

They should not be able to do that as that is akin to extortion. So if I had a legitmate service hosted on A2B that was blocked due to spamhaus blocking A2B due to cyberbunker, I would be collateral damage. Many legit companies that use Spamhaus might not even know all the IP ranges that are blocked. I don't like the idea of Spamhaus blocking entire chunks of the internet and the companies using the lists would be none the wiser. They wield too much power with that list since they can decide who is on or off even though it is the local admin who decides to implement it or not (and maybe not knowing of the side effect).

I have no sympathy though for the spammers though but like the issue with civil rights, better stop the bigger power even if you have to side with the evil guys sometimes. They are still wrong to DDOS though

I see spamhaus's actions similar to black listing entire countries, leading to their sealing off. The correct way would be to selectively target and filter the traffic going through the border rather than a blanket blacklist.

Peter, so if I understand correctly, Spamhaus is the "de-facto" provider or email filters and blacklist to the ISP and other industries as a whole? Basically are they an industry adopted solution?

I have no problem with what Spamhaus does, and a well needed service it is, but I do worry when a third party might exercise that much control without a lawful review process. I don't endorse the DDoS in any way. But does the good of Spamhaus out weigh the bad? I can dig it if it does, but I would like some thoughts on this.

There are other DNS blacklists for stopping spam; Spamhaus is by no means a monopoly. There are fewer than there once were, as many organizations grew tired of the cost (and attacks) that resulted from operating such services. However, this still leaves dozens of operational blacklists.

I don't like Spamhaus's going nuclear all the time. It seems that they will block legitimate traffic in attempts to stop hosters to drop the spamming IP's.

Spamhaus never blocks any traffic.

Quote:

They should not be able to do that as that is akin to extortion. So if I had a legitmate service hosted on A2B that was blocked due to spamhaus blocking A2B due to cyberbunker, I would be collateral damage. Many legit companies that use Spamhaus might not even know all the IP ranges that are blocked. I don't like the idea of Spamhaus blocking entire chunks of the internet and the companies using the lists would be none the wiser. They wield too much power with that list since they can decide who is on or off even though it is the local admin who decides to implement it or not (and maybe not knowing of the side effect).

I have no sympathy though for the spammers though but like the issue with civil rights, better stop the bigger power even if you have to side with the evil guys sometimes. They are still wrong to DDOS though

Spamhaus only escalates after ISPs refuse to act on its single IP address blocks. In the A2B case, for example, three months passed between the initial single IP address block, and Spamhaus's escalation.

Do you mean open as in accepts queries for domains it is not authoritative for from clients outside of its network? I'm just looking for clarification, as in maybe you know something I don't and I want to know what that something is. I did the obligatory googling but the hits all seem to be related to public DNS services which I suspect are the problems you're pointing out.

Yes, open as-in, basically, misconfigured to accept and pass traffic that they shouldn't; "open" in the manner described in the article, the manner which enables such attacks:

Quote:

The specific issues are ISPs that allow forged traffic to leave their networks—something which has little good justification to permit—and open DNS servers that can be used to generate large responses in response to forged IP traffic.

Sorry if that wasn't clear. Here's a link to yesterday's article, in which "open" DNS servers and amplification attacks are further discussed, with a bit more detail:

As an email admin, I use seven different real time blocklist's on my mail gateway. The zen.spamhaus.org stops the majority of the spam coming in. I would hate to lose it. Spammers can die in a fire.

The SMTP protocol needs to be updated to enforce proper identity. There are plenty of solutions put out there to do this but none of them are official in the SMTP protocol. This is something that should have been done back in the late 90's.