Krebs on Security

In-depth security news and investigation

The New Normal: 200-400 Gbps DDoS Attacks

Over the past four years, KrebsOnSecurity has been targeted by countless denial-of-service attacks intended to knock it offline. Earlier this week, KrebsOnSecurity was hit by easily the most massive and intense such attack yet — a nearly 200 Gbps assault leveraging a simple attack method that industry experts say is becoming alarmingly common.

At issue is a seemingly harmless feature built into many Internet servers known as the Network Time Protocol (NTP), which is used to sync the date and time between machines on a network. The problem isn’t with NTP itself, per se, but with certain outdated or hard-coded implementations of it that attackers can use to turn a relatively negligible attack into something much, much bigger. Symantec‘s writeup on this threat from December 2013 explains the problem succinctly:

Similar to DNS amplification attacks, the attacker sends a small forged packet that requests a large amount of data be sent to the target IP Address. In this case, the attackers are taking advantage of the monlist command. Monlist is a remote command in older version of NTP that sends the requester a list of the last 600 hosts who have connected to that server. For attackers the monlist query is a great reconnaissance tool. For a localized NTP server it can help to build a network profile. However, as a DDoS tool, it is even better because a small query can redirect megabytes worth of traffic.

Matthew Prince, the CEO of Cloudflare — a company that helps Web sites stay online in the face of huge DDoS attacks — blogged Thursday about a nearly 400 Gbps attack that recently hit one of the company’s customers and leveraged NTP amplification. Prince said that while Cloudflare “generally [was] able to mitigate the attack, it was large enough that it caused network congestion in parts of Europe.”

“Monday’s DDoS proved these attacks aren’t just theoretical. To generate approximately 400Gbps of traffic, the attacker used 4,529 NTP servers running on 1,298 different networks,” Prince wrote. “On average, each of these servers sent 87Mbps of traffic to the intended victim on CloudFlare’s network. Remarkably, it is possible that the attacker used only a single server running on a network that allowed source IP address spoofing to initiate the requests. An attacker with a 1 Gbps connection can theoretically generate more than 200Gbps of DDoS traffic.”

NO TIME LIKE THE PRESENT

Prince suggests a number of solutions for cleaning up the problem that permits attackers to seize control over so many ill-configured NTP servers, and this is sound advice. But what that post does not mention is the reality that a great many of today’s DDoS attacks are being launched or coordinated by the same individuals who are running DDoS-for-hire services (a.k.a “booters”) which are hiding behind Cloudflare’s own free cloud protection services.

As I noted in a talk I gave last summer with Lance James at the Black Hat security conference in Las Vegas, a funny thing happens when you decide to operate a DDoS-for-hire Web service: Your service becomes the target of attacks from competing DDoS-for-hire services. Hence, a majority of these services have chosen to avail themselves of Cloudflare’s free content distribution service, which generally does a pretty good job of negating this occupational hazard for the proprietors of DDoS services.

Lance James, Yours Truly, and Matthew Prince.

Mr. Prince took strong exception to my remarks at Black Hat, which observed that this industry probably would destroy itself without Cloudflare’s protection, and furthermore that some might perceive a credibility issue with a company that sells DDoS protection services providing safe haven to an entire cottage industry of DDoS-for-hire services.

Prince has noted that while Cloudflare will respond to legal process and subpoenas from law enforcement to take sites offline, “sometimes we have court orders that order us to not take sites down.” Indeed, one such example was CarderProfit, a Cloudflare-protected carding forum that turned out to be an elaborate sting operation set up by the FBI.

In a phone interview today, Prince emphasized that he has seen no indication that actual malicious packets are being sent out of Cloudflare’s network from the dozens of booter service Web sites that are using the service. Rather, he said, those booter services are simply the marketing end of these operations.

“The very nature of what we are trying to build is a system by which any content can be online and we can make denial-of-service attacks a thing of the past. But that means that some controversial content will end up on our network. We have an attack of over 100 Gbps almost every hour of every day. If I really thought it would solve the problem, and if our network was actually being used in these attacks, that’s a no-brainer. But I can’t get behind the idea that we should deny service to a marketing site just so that it can be attacked by these other sites, and that this will somehow make the problem go away. I don’t think that’s right, and it starts us down a slippery slope.”

As a journalist, I’m obviously extremely supportive of free speech rights. But it seems to me that most of these DDoS-for-hire services are — by definition — all about stifling speech. Worse yet, over the past few months the individuals behind these offerings have begun to latch onto NTP attacks, said Allison Nixon, a researcher for NTT Com Security who spoke about DDoS protection bypass techniques at last year’s Black Hat.

“There is a growing awareness of NTP based attacks in the criminal underground in the past several months,” Nixon said. “I believe it’s because nobody realized just how many vulnerable servers are out there until recently. “The technical problem of NTP amplification has been known for a long time. Now that more and more attack lists are being traded around, the availability of DDoS services with NTP attack functionality is on the rise.”

(S)KIDS JUST WANNA HAVE FUN

The shocking thing about these DDoS-for-hire services is that — as I’ve reported in severalpreviousstories — a majority of them are run by young kids who apparently can think of no better way to prove how cool and “leet” they are than by wantonly knocking Web sites offline and by launching hugely disruptive assaults. Case in point: My site appears to have been attacked this week by a 15-year-old boy from Illinois who calls himself “Mr. Booter Master” online.

Prolexic Technologies, the company that has been protecting KrebsOnSecurity from DDoS attacks for the past 18 months, said the attack that hit my site this week clocked in just shy of 200 Gbps. A year or two ago, a 200 Gbps attack would have been close to the largest attack on record, but the general upswing in attack volume over the past year makes the biggest attacks timeline look a bit like a hockey stick, according to a blog post on NTP attacks posted today by Arbor Networks. Arbor’s writeup speaks volumes about the motivations and maturity of the individuals behind a majority of these NTP attacks.

Source: Arbor Networks

The NTP attack on my site was short-lived — only about 10 minutes in duration, according to Prolexic. That suggested the attack was little more than a proof-of-concept, a demonstration.

Indeed, shortly after the attack subsided, I heard from a trusted source who closely monitors hacker activity in the cybercrime underground. The source wanted to know if my site had recently been the subject of a denial-of-service attack. I said yes and asked what he knew about it. The source shared some information showing that someone using the nickname “Rasbora” had very recently posted several indicators in a private forum in a bid to prove that he had just launched a large attack against my site.

Rasbora’s posts on Hackforums.

Apparently, Rasbora did this so that he could prove his greatness to the administrators of Darkode, a closely guarded cybercrime forum that has been profiled at length in this blog. Rasbora was anxious to show what he could contribute to the Darkode community, and his application for membership there hinged in part on whether he could be successful in taking down my site (incidentally, this is not the first time Darkode administrators have used my site as a test target for vetting prospective members who apply based on the strength of some professed DDoS prowess).

Rasbora, like other youngAmerican kidsinvolved in DDoS-for-hire services, hasn’t done a great job of separating his online self from his real life persona, and it wasn’t long before I was speaking to Rasbora’s dad. His father seemed genuinely alarmed — albeit otherwise clueless — to learn about his son’s alleged activities. Rasbora himself agreed to speak to me, but denied that he was responsible for any attack on my site. He did, however, admit to using the nickname Rasbora — and eventually — to being consumed with various projects related to DDoS activities.

Rasbora maintains a healthy presence on Hackforums[dot]net, a relatively open forum that is full of young kids engaged in selling hacking services and malicious code of one kind or another. Throughout 2013, he ran a DDoS-for-hire service hidden behind Cloudflare called “Flashstresser.net,” but that service is currently unreachable. These days, Rasbora seems to be taking projects mostly by private contract.

Some of Rasbora’s posts prior to our phone call.

Rasbora’s most recent project just happens to be gathering, maintaining huge “top quality” lists of servers that can be used to launch amplification attacks online. Despite his insistence that he’s never launched DDoS attacks, Rasbora did eventually allow that someone reading his posts on Hackforums might conclude that he was actively involved in DDoS attacks for hire.

“I don’t see what a wall of text can really tell you about what someone does in real life though,” said Rasbora, whose real-life identity is being withheld because he’s a minor. This reply came in response to my reading him several posts that he’d made on Hackforums not 24 hours earlier that strongly suggested he was still in the business of knocking Web sites offline: In a Feb. 12 post on a thread called “Hiring a hit on a Web site” that Rasbora has since deleted, he tells a fellow Hackforums user, “If all else fails and you just want it offline, PM me.”

Rasbora has tried to clean up some of his more self-incriminating posts on Hackforums, but he remains defiantly steadfast in his claim that he doesn’t DDoS people. Who knows, maybe his dad will ground him and take away his Internet privileges.

And no doubt in a week or two will be back to his same stupid tricks… The mentality of people like that is generally more of a brick than a sponge, too macho to soak up useful life experiences and learn their lesson.

Ok so I’ve been involved with computers since the mid-nineties and at this point am running an IT department. I’m by no means ‘leet’ but I get by and can usually solve problems and even automate things here and there. What boggles my mind is how does someone who has been alive for less time than I have been learning and working with computers learn enough about how the fundamental structure of the internet works to be able to pull these kinds of things off? Disclaimer: yes I’m jealous – but that doesn’t quite explain it. He can’t have even been studying for those 15 years as he needed a few years to learn how to just READ didn’t he?

Maybe his parents are grounding him to a corner with technical manuals and a computer when he acts up?

Watch your mouth. And I broke privacy laws? How do you figure? The kid’s dad explicitly gave me permission to interview him. And what’s more, I don’t even name the little turkey, so it’s hardly an invasion of privacy.

Total Ignorance on your part When is illegal to make a phone call? If the parent said ” don’t call here anymore” and B.K. did then I would see a issue. But because he spoke to this kids legal guardians , I don’t see a issue here,

Probably was a waste of time either way… Guys like that are usually too dense to learn their lesson until some guy named Bubba diddles them in the butt for 5 years because of their ‘fun time’ on the computer. Maybe then it will sink in that they shouldn’t be breaking things they don’t own and cost a fortune just for fun.

I’d bet some good money that he doesn’t even bother to go… probably regularly misses classes and has a C or D average, the guy lining him up for law school is pretty funny, by the time this kid hits that age he will probably have a few felonies and a 2.0 GPA.

It really doesn’t require deep knowledge to use tools others have created to facilitate attack by means that others have discovered. It isn’t even very hard to take a cursory description of the NTP monlist attack and the source for a legitimate NTP client to hack together an attack tool. One might even be able to avoid any C work and just morph the packets in iptables…

Note that at no point would this skid need to learn “how the fundamental structure of the Internet works” OR much of the knowledge that a legitimate pro needs to run a modest-size IT department. As for where he found the docs, that’s easy: install any general-use Unix-like OS (Linux, FreeBSD, MacOS, whatever…) and one has all the tools and docs one could want for doing *anything* with TCP/IP. And of course there are piles of legitimate open source code to crib from if one is eager to code instead of just cobbling together existing tools.

Rasbora- I can only hope you read this. The world is going to need people like yourself, with a passion for net/comp. There’s tons of money to be made by boning up on core elements like AP dev. and so many other things. As you well know so much of our world is moving onto the net, ready or not. The next war wont be fought on the ground, and I think you know that. When they come for _your_ connection you really need to know where you stand morally. Take all of that anger, focus it really tight and write something that helps everybody out. The things you’ll learn about being a Dev. and yourself will be worth the trip.
Hope your able to understand,
-b

I’ve got a 14 year old at home who could easily do this sort of thing if 1) I let him spend every waking moment on the computer like he wants and 2) I didn’t pay attention to what he was doing on said computer. The Internet creates amazing opportunities for these kids; they can learn so much about tech. There’s also a lot of inappropriate information. It’s difficult enough for these kids to understand the consequences of their actions in the physical world. On the Internet, where most of their actions seem anonymous, this disconnect becomes even worse.

It’s the parents responsibility to keep an eye on their kids while on the computer/Internet. They should know what games they play, what sites they visit, etc. They should be blocking and monitoring web sites. You have to wonder if that was happening in this case.

I was wondering (worried) when I couldn’t get onto your site on Tuesday, but since you were only down for 10 minutes, maybe that was just my broke-down iPhone 4. Good to hear you are on the case as usual. What a tangled web the free Internet is…

Great post, but I think your analysis of DDoS-for-hire sites attacking one another, is static and therefore incomplete. Granted, at least initially, DDoS-for-hire sites might start to attack one another if kicked out from behind security networks. However, in the longer run, attacking each other is ultimately unprofitable, just as the Sopranos and Corleones don’t go on whacking one another forever. The weaker ones will get knocked out, but sooner or later they will achieve some truce, divvy up the territories, and start on more profitable criminal ventures. “You get North Jersey and I get every thing south of Mulberry Street.” At the end, you would be left with a Nash Equilibrium and a Darwinian outcome comprised of the most ruthless sites. Full disclosure: I work for CloudFlare.

But at least the politicians wouldn’t be looking the other way out of fear of retribution….

Of course, I know it is not so simplistic. And that freedom of speech is something we must uphold; even as it is abused by some doesn’t mean we fight fire with fire.

It would also be a large task to police(yes, police) every client, even if actions were taken only after they have acted badly. That costs money; money which would make a company look a lot different to investors who, by their nature, care about money.

But back to the politics – perhaps companies such as Cloudflare are hesitant to act because of the fear of recriminations. I can imagine that if one is associating with people who enjoy strongarm tactics, one would be silly to think they would be exempt if they started to not “play nice.”

Cloudfire knows everything that BK laid out here. So, it’s a good, educated guess that they’re about making money, and that as long as they’re making money and staying out of legal trouble (civil or criminal) they’ll stay with the current business model.

And yet I can’t remember ever have trouble getting to your site! Maybe this is why others posting here complain of lag time before their posts show up? Otherwise PFTT! – they be a figment of the imagination – go away figment! ]:)

Cloudflare saying that they’re not seeing any outbound activity is totally disingenuous, but technically true. Since they only handle requested traffic, not all outbound traffic, they only see connections that are initiated from outside. The root server could be sending out traffic and they’re be none the wiser. It’s even better if there’s more than one connection on the server.

I’m not a fan of CloudFlare. I had a problem accessing one of their client’s sites but the only way to contact CloudFlare is is to sign-on as a new client. I did that (it was free and only took a minute) and then filled out a “Tech. Support Ticket”, but when I tried to submit the ticket, the web-form was SO broken I had to give up and just remove the original site from my bookmarks.

I’m one of those people who thinks the inventors of the so-called Cloud were probably smart, while their clients definitely aren’t. But there don’t seem to be many of us who think this. Or maybe most of us can only speak Russian. Who knows? I imagine Russians laugh pretty hard about the cloud. Maybe THEY invented it. Maybe Mr. Kaspersky invented it. They invented Tetris, after all, and won the space race despite/while being a communist country: 1st space ship, 1st animal, 1st man, 1st woman in space. As for the moon, they just used telescopes. Brilliant!

Rasbora is harmless, honestly. I used to speak with him, we used to be quite close actually. Used to help him a bit with FlashStresser. Krebs, get your facts straight — Just because he posts some shit on HF does it mean he’s “right up there in the l33t z0ne of ub3r hax0rs?”. I think not, watch your mouth, son.

Lol’d for a second here.
Hi Fhoto & Rogue.
I used to hang around with Rusburra too, he’s cool. Never had a single problem with him. Besides that, if needed to produce around 200 gbps attacks, you’ll need by fare 20+ attack servers. Which are very expensive. So you say a 15 year old kid can have more then $1000?

Can you come up with a simple way to prevent attacks like this? How about RPF? Unfortunately, this needs to be done at the ISP level, and from my experience with ISPs, they don’t all have routing gurus on staff.

“As a journalist, I’m obviously extremely supportive of free speech rights. But it seems to me that most of these DDoS-for-hire services are — by definition — all about stifling speech.”

Can’t agree more and i told them numerous times, what they say is just a stupid technical explanation of an incredible marketing BS. Cloudflare should be prosecuted. There’s no free speech here, ddos is AGAINST free speech.

I hope cloudflare dies behind these attacks and loses all their customers, that’s what they deserve.

” Apparently, Rasbora did this so that he could prove his greatness to the administrators of Darkode, a closely guarded cybercrime forum ” . I found a little humor when I went to Norton Safe Web ( https://safeweb.norton.com/report/show?url=darkode.com ) ” Safe Web Report for: darkode.com Annoyance factors: 0.” Virustotal had just two mentions of Malicious site and one mention of Suspicious site. As expected, Web of Trust said “The site has a poor reputation based on user ratings, and it is not suitable for children.”

Maybe you could just by accident say…give me this young mans address. I can assure you he won’t touch a keyboard ever again after we “talk”…..please….I’ll pay money. I know I would make any I spent back on Youtube adverts…..even after the young man sees a proctologist.

mr. krebs thank you for contacting the boy’s father. having been 15 once i can assure everyone that if an act is destructive, dangerous, and really, really dumb it has been acted on or at least seriously considered. in a world rocketing to the “internet of things” a single (or more likely group of) pubescient kids can create havoc. even if this kid wasn’t hoping for $$, pure vandalism is equally challenging and attractive… IT security has consider snot nosed idiot kids as well as disgruntled employees and criminal/state run gangs…frankly i think teenage punks are going to be just as much trouble as the rest, just different. we’re not talking pellet guns and spray paint now.

“He said the company has a stated policy of not singling out one type of content over another, citing a fear of sliding down a slippery slope of censorship.”

But freedom of (anonymous) speech is only one side of the censorship coin. The other side is freedom of (anonymous) inquiry, and Cloudflare emphatically does NOT support that. Every single site under their umbrella blocks TOR users, regardless of if they are potentially posting comments or just there to read information (without exposing their reading habits to NSA or other third-party scrutiny).

The right to inquire and read without being monitored, identified, and profiled by the government, the website you’re reading, advertising partners of that website, and others is clearly not being respected by Cloudflare. And that’s a problem.

-Cloudflare protects booter site
-Cloudflare protects some random flower shop (just an example)
-Competitor pays 20$ to the booter site (protected by cloudflare) to boot off the shop site
-Cloudflare drops protection on shop when hit by the ddos
-Shop site backend has no protection and gets a direct ddos
-Shop site offline
-Competitor wins
-Booter wins

Lesson of the story:

Cloudflare doesn’t even protect their own customers from the booters they have.

But they will offer you the business or enterprise plan, starting at XXX$ per month.

Some would argue that only a government can censor. I guess Prince sees corporations as a “government” actor since a corporation is an individual, and individuals are, fundamentally, the establishers of governments?

You’re so amateur! I was Orig Anon back in the day on cb’s, yaesu and ham radios. you know nothing about self discipline or discretion. that’s why you will never amount to anything and ultimately end up in a club fed prison. grow up kid and stop bringing humiliation and embarrassment to your family name. you may be smart, But you’re not as smart as me or anyone else here. just fade to black. and that goes for all of you wanna be self proclaimed ‘1337’ derps.

“Earl: You’re still a drug (DDoS-for-hire) dealer. And I’m still the government of the United States. It’s a free market, Cake. Not a free world.”

Cloudflare protects the blackhat sites — but it also records all the traffic through their nameservers and responds to US Justice Dept. subpoenas. And traffic via known open proxies is automatically blocked unless the site’s administrators manually permit them. It seems colossally stupid for anyone in the US (or any country with an extradition treaty) to be doing illegal business on a site “protected” by Cloudflare.