Posted
by
samzenpus
on Friday March 29, 2013 @07:55AM
from the what's-to-blame dept.

CowboyRobot writes "Despite the headlines, the big denial of service attack may not have slowed the Internet after all. The argument against the original claim include the fact that reports of Internet users seeing slowdowns came not from service providers, but the DDoS mitigation service CloudFlare, which signed up Spamhaus as a customer last week. Also, multiple service providers and Internet watchers have now publicly stated that while the DDoS attacks against Spamhaus could theoretically have led to slowdowns, they've seen no evidence that this occurred for general Internet users. And while some users may have noticed a slowdown, the undersea cable cuts discovered by Egyptian sailors had more of an impact than the DDoS."

as usual, ArsTechnica does a much [arstechnica.com] better [arstechnica.com] job [arstechnica.com] of describing this, slashdot eds, take note please!

The best text-only (no ads!) reply [cluepon.net] though is from Richard A Steenbergen who responded to the gizmodo article. This guy works at one of the tier 1 providers and described the problem, particularly that the DDoS wasn't a big deal for them but that the attack on the INX exchanges might have been.. but turned out not to be after a little tweaking of their filters.

Nevertheless, the problem that I can see is that the internet is open to these kind of attacks. Now Spamhaus can get CloudFlare to handle these attacks on their behalf (for a lot of free advertising) but MyLittleSite.com cannot, and that leave them open to extortion attacks from the criminals who run these DDoSs. Surely a more appropriate response would not be "yeah, we're great, we can handle a poxy 300Gbps" but "we need to sort out this so the baddies cannot screw people with impunity". I'd prefer a technical resolution (eg ingress/egress filtering, rate limiting, non-recursive responses from outside your domain) to legal ones which is all there is at the moment it seems.

Not really, Two things would go a long way towards ending the ddos threat permanently. First, Implementing gateway sanity checks that already exist. If a provider is forwarding packets with spoofed IP address', then un-peer them until they fix their configurations.

Second, is an out of band feature which provides a mechanism where the recipient of a packet can flag that packet as malicious and ask the upstream connection to shut them down at their source. This feature should be recursive, and with the same sanity checks to make sure the requests are legitimate.

As a result of these two, a ddos begins: The recipient computer starts flagging IP address and requesting that their host provider shut off the flows from each IP as they are identified. Host provider filters everything from that IP. for a random interval between 10 minutes and a half hour. The host IP also passes the filter request upstream to the next link in the chain. This process continues until it backtracks all the way to each source machine, which finds itself disconnected for 10 minutes. If the owner is private, then they will call their ISP to find out why their connection sucks, at which point the ISP tells them, your machine is taking part in an illegal ddos. If you don't know how to fix it, take your computer to the local shop to have it cleaned, and have them explain internet security to you while they're at it. If the computer is institutional, then their IT department is going to have one heck of lot of explaining to do, being as they have compromised servers and had to be told by their ISP that they have a problem... Either way, no bot net operator will risk having their botnet dismantled automatically without a very long thought about what they are trying to accomplish. Additionally, no ddos would be effective for more than a few minutes as the requests filtered back upstream and shut it down at its distributed sources.

The biggest complaint I always hear about this plan, is what if someone spoofs shutdown requests to get someone disconnected. That kind of spoofing could only work if one of the intermediate nodes is compromised, or the IP validation is not enabled. either way, it requires that the network be broken in easily fixable ways, which presumably will be fixed as soon as discovered. Think of the whole system as an autoimmune reaction to infection. Terribly effective and largely automatic.

What if someone - say one of the millions of compromised computers out there - were to send a shutdown request against a large player - say spamhaus. By your example, spamhaus would be taken offline until someone at spamhaus' IT department called their ISP to get the block cleared. At which point another compromised computer sends a new shutdown request. Sure, you loose a few bots, but at the cost of one bot per 5 minutes downtime for a large vendor, you can get a pretty big bang out of a small bot network.

What if someone - say one of the millions of compromised computers out there - were to send a shutdown request against a large player - say spamhaus. By your example, spamhaus would be taken offline until someone at spamhaus' IT department called their ISP to get the block cleared. At which point another compromised computer sends a new shutdown request. Sure, you loose a few bots, but at the cost of one bot per 5 minutes downtime for a large vendor, you can get a pretty big bang out of a small bot network.

Kind of. In your example, what happens is that spamhouse looses the ability to send packets to that IP address The rest of their connection remains unaffected. Their ISP gets an indication that spamhaus is sending malicious traffic, and the routers have automatically blocked packets to the victims IP address. Everyone else is unaware that anything is going on. Spamhaus is relatively unaffected. It all depends what their ISP does with the information. Institutional services probably wont get disconnected rig

This was a smurf amplification attack, so no you won't be ending the ddos threat permanently. Or more accurately, a distributed smurf amplification attack. Your ideas might work to prevent the smurf amplification, but there is nothing which can be done to prevent a simple, direct DDoS. Except to have more bandwidth and processing capability than the attacker does.

Smurf attacks are easy to stop, and are not terribly effective anymore thanks to various changes made to the IP standard in the last 15 years. IP validation eliminates all possibility of smurf attacks, because smurf attacks only work if IP spoofing works. Properly configured networks are immune.

Yep, your solution is worse than the DDoS itself, because it only requires a few requests to take a server offline, not a massively sustained attack.

Can you explain to me how to progmatically tell the difference between your "spoof" shutdown request and a real one? If you can't do that, then you could effectively DDoS an entire ISP when all of their customers have their connections shut down, and they can't get through to support lines because everyone else is phoning up to get their line re-enabled, etc..

Can you explain to me how to progmatically tell the difference between your "spoof" shutdown request and a real one?

The sanity checks that prevent spoofing IP address will also prevent spoofed kill requests from making it to their destination. Even if the spoofed request makes it to the hosts routers, all it will do is shutdown their ability to send packets to the one ip address. the rest of their connection will remain unaffected. Again, if someone along the line is not doing spoof checking, this will highlight that very quickly, and they will fix it.

Second, is an out of band feature which provides a mechanism where the recipient of a packet can flag that packet as malicious and ask the upstream connection to shut them down at their source. This feature should be recursive, and with the same sanity checks to make sure the requests are legitimate.

LOL flag this packet as spam.. There is already a packet option for this in RFC 3514.

As a result of these two, a ddos begins: The recipient computer starts flagging IP address and requesting that their host provider shut off the flows from each IP as they are identified.

How do you identify an attack(er)?

Host provider filters everything from that IP.

Why should they trust your attack classification or anything you tell them? It sounds like a good way for them to get sued into oblivion.

which finds itself disconnected for 10 minutes. If

Why ten minutes? What if I have IPv6 and a number of IPs equal to 4 billion IPv4 Internets? Will it still target single hosts?

If the owner is private, then they will call their ISP to find out why their connection sucks

LOL I think most ISPs would pass unless your volunteering to sit there and pick up the phone for free.

LOL I think most ISPs would pass unless your volunteering to sit there and pick up the phone for free.

I think most of them will do this if the alternative is willfully ignoring a criminal act, which in most countries in the world is still conspiracy to commit... Answering the phone once in a while is a small price to pay, especially since, with these protections in place, ddos' will become practically non-existant: Because they would no longer work, no one would bother.

People have spent billions trying to classify email messages as spam vs legitimate and they are still no closer to getting that right.

That is because whether or not a particular e-mail is spam, is a very subjective analysis. One persons spam is another's monthly coupons. Personally, I would identify the entire Sunday paper coupon section as spam, but my wife reads it like the bible...

Malicious packets on the other hand are fairly easy to identify, and there isn't much gray area. Once is happenstance. Twice is coincidence. Three times, it's enemy action.

This is not something new. These attacks have been known for decades. The majority of existing protocols either are not subject to or have protections against this problem.

If you try and send SYN packets to start a TCP session using a spoofed source address the vast majority of currently deployed stacks will start requring cookies. If you are not able to receive the cookie your evil plot is foiled.

This problem really still only exists in a subset of clueless UDP protocols.

Now Spamhaus can get CloudFlare to handle these attacks on their behalf (for a lot of free advertising) but MyLittleSite.com cannot, and that leave them open to extortion attacks from the criminals who run these DDoSs.

Why not? CloudFlare has a free tier specifically designed for smaller sites. It's mostly used by bloggers and stuff to cache static content than for DDOS protection, but it offers the same level of functionality. The paid service they offer has extra features like SSL support and other options, but all levels of the service offer DDOS protection.

I wasn't thinking of blog sites and so, but commercial entities that are more likely to be sent an email telling them that unless they pay $10k on a special day (eg a card retailer on Valentine's day) they'll be knocked off the internet for a week.CloudFlare will handle these extortion attempts, but as the site taking orders will require SSL, its probably cheaper just to pay the criminals. and that's a bad state of affairs.

Umm.. I'm not sure I follow you. The DDoS was comprised of DNS Reflection. Trying to add filtering at layer 2/3 is absolutely pointless since you're saturated at layer 0. The physical hardware is overwhelmed trying to keep up with the packets coming in.

Umm.. I'm not sure I follow you. The DDoS was comprised of DNS Reflection. Trying to add filtering at layer 2/3 is absolutely pointless since you're saturated at layer 0. The physical hardware is overwhelmed trying to keep up with the packets coming in.

I would guess he means filtering at the sending end in for sending IP's that are not in-network. And rate limiting on the recursive DNS end.

These are the obvious solutions to this kind of DDOS. Unfortunately, these solutions require third parties that are contributing to but not suffering in any serious way from the DDOS, to configure their servers/routers properly. There is a lack of motivation.

try reading the links. One thing that can happen is for all ISPs to refuse to deliver packets that are sent with a source IP that isn't part of that ISP's network.

Like if I called a pizza company and ordered you a pizza, assuming you lived in New York, and the call from me showed a California number, the pizza place would think twice about filling the order. On the internet, they'd send you a pizza, as would all the other places I'd have called. (ok, not the best analogy, but you get the idea)

CloudFlare can protect you for free. You might not have as much control but they'll protect you from bots and such for free. Also, MyLittleSite.com probably isn't big enough to piss off anyone who can crank 300 Gbps of DDoS (even if they required only 1/100th of that since they exploited open DNS resolvers).

But, don't let the facts get in the way of sensationalist clickbait and media whoring. If nothing else, the clueless need something to get incensed about and start demanding legislative fixes to imaginary problems.

The problem was supposedly more severe in Europe but, FWIW, my response times in Madrid, Spain were completely normal. I realize that proves nothing, but it does make me skeptical of the Internet Brought to It's Knees claims.

It's definitely a way for SPAMHaus to make the headlines. Whether it is proper conduct, especially for a trust-based organisation like SPAMHaus, is the real question.

DNSBL is not the way to fight spam. I've worked for several large ESP's, and we've had more issues with false positives and various DNSBL's blocking regular, solicited email everytime some angry recipient with a vengeace decided to file a spam-report, instead of just opting-out from the mailing they opted-in for themselves.

But it's definitely sounds a bit 'shadey' to launch a misinformation-campaign for this, especially for an antispam-firm.

What part of "launch a misinformation-campaign" doesn't sound "shadey"? Well, aside from the accusation coming from from an anonymous poster who doesn't bother to provide a shred of supporting evidence for the claim. That part doesn't sound the least bit shady.

DNSBL is great for fighting spam -- *if* you find collateral damage acceptable.

As someone who had joe jobs cost a few thousand dollars years ago... I find this wholly appropriate. I *want* to opt in to DNSBL and hurt ISPs that host a spammer. If you affiliate with a ROKSO spammer -- I don't want my servers on your subnet. I don't want email from your subnet once you've had an opportunity to remove them and failed to do so.

Thing you have to understand is DNSBLs fight spam -- not "unlawful UCBE".

People like to hear that DNSBLs are a problem. And then they like to repeat the accusations. Not sure how folks have gotten attached to the idea, but I'm certain it's not from detailed investigation.

For one thing, don't conflate the mechanism with the implementations. Anyone can publish a DNSBL. You could. And you could make your list all false positives. It would be a bad idea for people to subscribe to your list. Caveat emptor, right?

And that's why you get false positives. You've chosen badly. And you're not using the lists for scoring — sounds like you're using them as final arbiters.

The "trick" to getting DNSBLs to work is to choose wisely. You have to do some research into how the lists are made, and since it's you who will be blocking emails based on the information provided by the lists, it's your responsibility to understand the nature of that information. What are the listing/delisting policies? If you don't know, you're not being a smart consumer. "... everytime some angry recipient with a vengeace decided to file a spam-report..." Hopefully you know better than to think that every DNSBL is made this way.

And the "smart" spam filters, so you know, are resource intensive. Instead, it's possible to eliminate lots of spam using extremely low resource checks. Validating the SMTP "HELO" (requiring they give FQDN, non-bare address literals, not your domain or IP, and a couple other checks as per RFC) will nix half of spam off the bat. And you can eliminate another third of spam (two-thirds the spam passing HELO checks) by using (well-chosen) DNSBLs. DNS lookups are cheap (and you can download zone files of you're worried about outages). That's 83% of spam cheaply nixed, all before you even get to "MAIL FROM:". If your "smart" checks are building Markov chains and feeding a naive Bayes classifier, that's gonna take time and effort in processing power, in disk resource, in procedures and staff attention/knowledge for maintenance.

DNSBLs are clearly a way to fight spam. But you have to know what they are and how to use them.

Shopping for DNSBLs takes effort, it's true. If you want to do a good job. Once upon a time, Al Iverson's http://www.dnsbl.info/ [dnsbl.info] was up-to-date and gave wonderful statistics on success rates of the various lists (using his (rather knowledgeable) measures). Doing the research now without such a resource is much more challenging.

I use Spamhaus's XBL and SpamCop's SCBL. That's it. Combined, those give me the aforementioned inexpensive 33% spam reduction. (If I used them before the HELO checks the reduction would probably be near 75%, my guess.) I vetted the lists for efficacy (true positives v. false positives), policy (how they're made, listing and delisting), and longevity/reputability. I've been using these guys for 5 years without a hiccup.

300Gb/s, what is that as a fraction of the total Internet bandwidth ? Without that number we don't know if it is a significant proportion of what is available.
Maybe we should be asking for that figure round/close-to the Spamhaus servers.

By total I mean the core internet routers, not including those in outlying backwaters.

Depending on how many decimal places you want to consider significant, it rounds to 0%.According to a resource [wikipedia.org], the attack would've consumed slightly more bandwidth than a OC-48 / STM-16 / 2.5G SONET optical carrier cable. Alternatively, it is about 1/13th the bandwidth of a OC-768 / STM-256 optical carrier cable.

So, I think you need more than 5 significant digits for 300Gb/s to round to any number other than 0% of total internet bandwidth, but if there was only one cable between the source machines and th

It works out to 30 or fewer average 10G Internet links. Depending on where it hits it could take out a good chunk of many smaller peering exchanges, but any of the Major (Tier 1) ISP's run 80Gbps+ between nodes with 100Gbps links becoming more common, and the larger peering fabrics run multiple Tbps across their peering fabrics. Basically, it is large to many individual sites, but tiny for Internet scale.

On Tuesday afternoon, GMT-6, I could do exactly zero of my job functions, as none of my remote server connections would stay up for longer than 5-7 seconds. Not knowing what was happening, I did hours of troubleshooting on my own connection, before finally just calling it quits for the day.

I was about ready to just walk away out of frustration before things just seemed to magically fix themselves the following morning. So yes, I think this did affect parts of the internet as a whole, and not others. I am

So if the spammers botnets are busy with a ddos attack, has there been any measurable decrease in spam on the internet? I haven't seen any internet slowdowns, but I haven't seen any slowdown in spam either...

The Internet connection speed for many is so slow already, that they would not even notice if the Internet speed as a whole dropped by 90%. In the evening, watching Netflix or any other video is a pain. That is why we still get DVDs in the mail.

Youtube for listening to music while I work is painful. It can't buffer at all and I have a FIOS connection. I had to reformat my computer and installing Office over the internet and patching the 3 gigs of data for SWTOR was capped at 300k even if I have a fiber connection.

I rebooted my router a few times and ran ipconfig/flushdns but to no avail.

However, none of my activity uses European servers or DNS so I highly doubt this is related at all. Google did say it was absorbing some of the traffic because th

I like CloudFlare, but it seems like they exaggerated the scope of this incident in order to get publicity. It's a Startup -- so any exposure seems like good exposure, and they have a lot of operating expenses (bandwidth/hardware/etc), so getting on some VC's radar for a second investment round might be a priority.
I'm in the network of the founders on LinkedIn and she shared the NY Times article about the incident asking (not directed towards europeans) all her contacts if the internet was running slow

Mine has been unbearably slow. I've called up my provider twice. Problem is, speed tests to their servers show I am gettign my advertised speed. If I do speed tests to nearby servers, I am seeing this, but if I go outside of my geographic area, speeds start taking a huge hit. Connecting to most speed test servers on the internet, I am seeing 1/20th of the speed I am paying for (I usually get close). I used to be able to stream HDX from Vudo no problem while surfing, but now, Amazon and Netflix SD buffer lik