Very interesting read. The thing about one IP address being outside of the US makes circumventing the constitution trivial since so much traffic gets routed through e.g. Canada to save money. Plus, so many websites I regularly visit aren't hosted in the US at all, which makes my communications with them fair game evidently.

I've said it before and I'll say it again, the third party doctrine is bullshit and needs to die in a fire.

As others have said, this is a very interesting read. In fact, it touches on a lot of similar facts that were exposed in a relatively recent Frontline documentary From 9/11 to Boston. It doesn't go into detail with regards to the NSA's technical efforts. Rather, it outlines just how the U.S. government ramped up its efforts on the "War on Terror", the massive amounts of money and new powers that were handed out, how the CIA, the DoD, NSA, Homeland Security, etc. all started scrambling for that power and how it's never ramped back down.

Working at CERN, it is remarkable how familiar all of this sounds... The strategic hierarchies are remarkably similar save for the inverted nature of the distributed collection nodes. There is only one ATLAS, so we have to distribute our data out to the world from a central collection point. These guys are just flipping that, distributed collection nodes and mostly "centralized" analysis directives.

Hmm...

Dear Mid-Level NSA Analyst/Developer/Technician,

I bet you have a post-graduate degree in a STEM field right?

I just want to encourage you. I bet you are a little embarrassed now, 'cause you probably wasted no time in telling your folks and friends that you landed a job at the NSA, I sure would. Don't take it too hard, people are justifiably upset with the lengths to which our grand social-contract has been circumvented by supposedly trusted leaders of every stripe. The system has let them down, and you, as a conveniently accessible cog in that system will see the brunt of the frustration. In fact, if you find yourself in my neck of the woods, I would love to hear your side of the story over a ice-cold Dr. Pepper.

In any case, if you would like to solve interesting problems using your above average ability in multi-variate data-mining, get in touch with your local university's High Energy Physics group. The work is challenging, your work will be appropriately attributed, and on top of all that, you wont be violating the sacred trust between the Government for and by the people and those very same people.

Sincerely,~AffineParameter

Ps. I am thinking about shorting a few stocks over the next couple of weeks, feel free to send me any interesting emails from these corporations... I'll leave the list in an unopened email in my Gmail, subject line: "Greenback Jihad"... just kidding. I'm a grad-student, what the hell are stocks anyway?

I was thinking about all this, and realised that I've basically accepted the “local internet sovereignty movement” as the best course of action. ICANN and such should probably be wrested away from US control now. We can't do much about your spying, but here in Europe we'll have better control over our secrets than if all this wasn't known and we still trusted yank corporation.

Yes, ICANN in international control probably means that countries like China and Russia have more say for nefarious purposes, but it's not like the US isn't the same, is it? What the UK and Germany and France can do to protect our industrial secrets is to have local data storage, and then we'll just have to fight off NSA spies as they attempt to penetrate the security. Now we know they care nothing for our rights and secrets. That's good to know too.

At this point, I don't really care if everything implodes in a great big ball of fire. Actually, while I have sympathy for the yanks I know and respect, since the US government have gone down this path and have made its choices, I sort of hope the whole hierarchy burns to the ground (Hi, NSA! I'm British, so this isn't actually treasonous talk, you know. Be a good sport; see you on the internet.).

The Global War On Terror, or GWOT as it was known around DC's beltway, opened up the purse strings for everything on the intelligence, surveillance, and reconnaissance (ISR) shopping list. The NSA's budget is hidden within the larger National Intelligence Program (NIP) budget. But some estimates suggest that the NSA's piece of that pie is between 17 and 20 percent—putting its cumulative budget from fiscal year 2006 through 2012, conservatively, at about $58 billion.

The Global War On Terror, or GWOT as it was known around DC's beltway, opened up the purse strings for everything on the intelligence, surveillance, and reconnaissance (ISR) shopping list. The NSA's budget is hidden within the larger National Intelligence Program (NIP) budget. But some estimates suggest that the NSA's piece of that pie is between 17 and 20 percent—putting its cumulative budget from fiscal year 2006 through 2012, conservatively, at about $58 billion.

Meanwhile, NASA has been left basically starving.

Actually, NASA's budget is considerably larger than NSA's. Over that same 2006-2012 period, we spent $121 billion on NASA (according to Wikipedia). I'd still like to see that number be higher, but it does sort of put the NSA is perspective. And Google had nearly the entire 2006-2012 NSA budget quoted above as income in a single year. If there is a vast, limitless panopticon, it must be the most efficient government operation ever

ICANN and such should probably be wrested away from US control now. We can't do much about your spying, but here in Europe we'll have better control over our secrets than if all this wasn't known and we still trusted yank corporation.

... What the UK and Germany and France can do to protect our industrial secrets is to have local data storage, and then we'll just have to fight off NSA spies as they attempt to penetrate the security. Now we know they care nothing for our rights and secrets. That's good to know too.

... I sort of hope the whole hierarchy burns to the ground (Hi, NSA! I'm British, so this isn't actually treasonous talk, you know. Be a good sport; see you on the internet.).

Only one handpicked group of nations is excluded -- countries that the NSA has defined as close friends, or "2nd party," as one internal document indicates. They include the UK, Australia, Canada and New Zealand. A document classified as "top secret" states that, "The NSA does NOT target its 2nd party partners, nor request that 2nd parties do anything that is inherently illegal for NSA to do."

Interviewer: What are some of the big surveillance programs that are active today and how do international partners aid the NSA?

Snowden: In some cases, the so-called Five Eye Partners go beyond what NSA itself does. For instance, the UK's General Communications Headquarters (GCHQ) has a system called TEMPORA. TEMPORA is the signals intelligence community's first "full-take" Internet buffer that doesn't care about content type and pays only marginal attention to the Human Rights Act. It snarfs everything, in a rolling buffer to allow retroactive investigation without missing a single bit. Right now the buffer can hold three days of traffic, but that's being improved. Three days may not sound like much, but remember that that's not metadata. "Full-take" means it doesn't miss anything, and ingests the entirety of each circuit's capacity. If you send a single ICMP packet and it routes through the UK, we get it. If you download something and the CDN (Content Delivery Network) happens to serve from the UK, we get it. If your sick daughter's medical records get processed at a London call center … well, you get the idea.

At this point, I don't really care if everything implodes in a great big ball of fire. Actually, while I have sympathy for the yanks I know and respect, since the US government have gone down this path and have made its choices, I sort of hope the whole hierarchy burns to the ground (Hi, NSA! I'm British, so this isn't actually treasonous talk, you know. Be a good sport; see you on the internet.).

On the day of 9/11, after seeing the images of people jumping and firemen rushing into a hopeless situation, the videos of support from the English people touched me very deeply.

On the day of 9/11, after seeing the images of people jumping and firemen rushing into a hopeless situation, the videos of support from the English people touched me very deeply.

That hasn't changed.

Like we went through with The Troubles, we have cultural experience of terrorism targeting our shopping centres and hotels and underground stations.

But there's a difference between having sympathy for the victims of terror, to submitting to the panopticon state as described in this article we're talking about.

My government is up in its eyeballs in that. So are many other European governments. It will take time to fix that, but I can be fixed. Unfortunately for relations between the governments, it means that we Europeans will have to strengthen the anti-american forces that exist in all our countries. Over the coming years, I expect parties that define themselves against the US will strengthen because of this.

Then the relationship will strain and our accessories to surveillance will be defunded or directed elsewhere. Distrust is already there now. No, it doesn't require massive shifts of public opinion. Even in my country, there are seven parties in parliament. The Greens or someone else would benefit. Most countries have some sort of proportional representation in their parliaments, and what is needed is just enough of a shift to make these parties part of the calculations for government coalitions.

I'm sorry about that, but you basically leave us no choice if we're going to continue to remain free and independent societies.

Your country wasn't forced into it's uber-close, long-term intelligence sharing relationship with the US. Your government feels it's tremendously valuable and critical to the UK's own interests.

My country is mostly interested in having piss-ups down at the pub, watching X-Factor and the Ashes to see if we can beat the aussies. My government may have decided it was so. But my government is not my country.

That would be the work of F6, the Special Collections Service, and NSA's Tailored Access Operations unit. They have either tapped into networks covertly by splicing into undersea cables or have compromised facilities of telecom companies in those countries in order to get access.

... the NSA used a small internal startup-like organization made up of NSA personnel and contract help from companies such as defense contractor SAIC to build and maintain XKeyscore. The XKeyscore product team used many of the principles of "agile" development and the so-called "devops" approach to running a Web operation—shipping code early and often, having support staff and developers work alongside each other, and reacting quickly to customer demands with new (and sometimes experimental) features.

Some of the "dotcom" dysfunctionalities also accompany this, and parallels to how such issues led to the "dotcom crash" can be helpful:

1) Perpetual beta mentality - endless tweaking and prospects for improvement. "This is just the beginning," "the next revision is better and around the corner," "we know what needs to be done, we just need more funding." This supports a "do no wrong" mentality - any failures (such as successful terrorist attacks) are simply because of product immaturity and inadequate resources to realize what is already envisioned (Total Information Awareness). False positives are not "wrong."

2) If you add computers to anything, it is inherently better. "Yes, I said bricks.com. But we're not just selling bricks, we're selling them on the Internet!"

3) Computational mysticism, mainly espoused by marketing (although there are plenty of developers amazed by the Matrix [mathematics]). It's just like how Amazon & Netflix generate suggestions, or how Google generates search results, and you know how successful they are. Just takes computer clusters and skillz. Artificial intelligence = black magic (No one has a clue how this neural net works, it just does (5% of the time, with the right training data), so Don't Touch It! Alternate: amazing results via genetic algorithms and/or simulated annealing.)

4) No responsibility when a computer causes a disaster (see "flash crashes"). We're just refining our understanding of the problem space, and just need to adjust some coefficients and add some more special cases to the code. Now we have more market intelligence. Today's lesson: no more automated drone strikes against suspicious looking people on Halloween (though this could be over-special-casing) - it's understandable this happened since the software was originally developed for the Middle Eastern market - another live & learn snafu, right?).

5) Look what else we can do! With our massive data set and existing code base, targeted advertising practically pops out! Sure, we launched our product touting we were ad-free, but the ads are "unobtrusive," yet so effective. A 0.5% click-thru/interrogation rate is almost unheard of!

6) If an algorithm spits out anything that was better in any way than a random guess, it's gold. You know something about the world - monetize it! We'll decide the color for our web portal statistically.

With these kinds of modern business ideas in hand, you can keep stringing investors (or Congresscritters) along for quite a while. Mix in the intrigue of spies & national security, and irrational levels of fear or terrorist attacks, and you're set. A hundred times more so when you're running in "stealth mode," just keep pointing to your successes (whatever they may be), and keep the not-so-great parts under wraps.

The idea that the system can't hold more than 3 days of full packet caps seems wrong to me.

A few years ago I assembled the underpinnings of a scalable system such as this one using snort and a custom DAQ. With a combination of dedupe and aggregation, they should be able to compress and store all the world's unencrypted traffic. I would imagine that this sort of system is probably being rolled out now.

Anyway many of the underlying tools are open source. There's even a couple of Linux distros that come bundled with them out of the box. You just install the distro on some decent hardware, walk up to a distribution switch in your network fabric, plug into the span / tap port, and off you go.

Seeing how everything we write is now automatically copyrighted, can't we file a massive class-action lawsuit against the NSA for copyright violation? Even non-commercial copyright infringement is punishable, reads the warning.

Seeing how everything we write is now automatically copyrighted, can't we file a massive class-action lawsuit against the NSA for copyright violation? Even non-commercial copyright infringement is punishable, reads the warning.

It's a little bit more complicated than that. The US has far weaker automatic copyright than the rest of the world. If the copyright is not registered within three months, you can only seek actual damages, which in this case would be zero. Also, there's often a number of exceptions to these laws for the government There's also fair use factors which would probably short circuit your case..

Seeing how everything we write is now automatically copyrighted, can't we file a massive class-action lawsuit against the NSA for copyright violation? Even non-commercial copyright infringement is punishable, reads the warning.

It's a little bit more complicated than that. The US has far weaker automatic copyright than the rest of the world. If the copyright is not registered within three months, you can only seek actual damages, which in this case would be zero. Also, there's often a number of exceptions to these laws for the government There's also fair use factors which would probably short circuit your case..

My emails are being copied the same day they're sent, well within the three month window. And I'd be willing to start registering them to sue the NSA for 150k/email.

Seeing how everything we write is now automatically copyrighted, can't we file a massive class-action lawsuit against the NSA for copyright violation? Even non-commercial copyright infringement is punishable, reads the warning.

It's a little bit more complicated than that. The US has far weaker automatic copyright than the rest of the world. If the copyright is not registered within three months, you can only seek actual damages, which in this case would be zero. Also, there's often a number of exceptions to these laws for the government There's also fair use factors which would probably short circuit your case..

My emails are being copied the same day they're sent, well within the three month window. And I'd be willing to start registering them to sue the NSA for 150k/email.

Fair enough, although it's worth noting that one element of getting a registered copyright is sending a copy to the Library of Congress. So you are paying $40 a pop to give your information to the government, so you better have your legal strategy well worked out beforehand or else it will backfire spectacularly.

Amazing stuff. With the pace of technological innovation, imagine where we'll be ten years from now. I wonder if the rule of law will have caught up by then...

It won't and it can't by design. The governments of most "free" countries were explicitly set up to be inefficient. It takes time and will to make something happen. That tends to cause the laws and rules put in place to be better than ones that are enacted as a knee-jerk response to something happening.

USA PATRIOT is an example of the kinds of terrible, sweeping laws that are rife for abuse when that ineffciency is cut through and something happens quickly because the knee-jerked hard enough to make it happen. If it had taken a decade for the US government (my government) to come up with USA PATRIOT it's arguable that it would have been far more limited than it was in the wake of 9/11.

Hollywood really is the modern day prophet. The internet had not yet been invented when said by the Outer Limit's control voice:There is nothing wrong with your television set. Do not attempt to adjust the picture. We are controlling transmission. . . . For the next hour we will control all that you see and hear.

If the technology existed to record everybody's thoughts and then cull out the interesting stuff and funnel it into a searchable database, does anyone think that our governments wouldn't do it?

Of course they would. Not just capture thoughts, control them. The U.S. military has been interested in mind control projects for quite some time now. If they could control people's free will, there wouldn't be any problems!

Just think, they started LSD tests in Cali back in the 60's to determine if it could be used for this purpose. Unfortunately for them, LSD is pretty much the red pill from the Matrix (exactly the opposite of what they were looking for). They are not going to make this mistake again.

"Where" we will be has been foreseen by Michel Foucault and J-F Lyotard during the 70's. Essentially, the modification of our behaviour will have extended the realm of surveillance to our own consciousness.

"Hence the major effect of the Panopticon: to induce in the inmate a state of conscious and permanent visibility that assures the automatic functioning of power. So to arrange things that the surveillance is permanent in its effects, even if it is discontinuous in its action; that the perfection of power should tend to render its actual exercise unnecessary; that this architectural apparatus should be a machine for creating and sustaining a power relation independent of the person who exercises it; in short, that the inmates should be caught up in a power situation of which they are themselves the bearers. — Michel Foucault. "Discipline and Punish" (1975)

I wonder if I turn off every collection devices around the globe Internet will become suddenly faster?Not only they have to analyze everything (slowing down a bit) but also to transmit some information so a big source of traffic would be missing...

The Global War On Terror, or GWOT as it was known around DC's beltway, opened up the purse strings for everything on the intelligence, surveillance, and reconnaissance (ISR) shopping list. The NSA's budget is hidden within the larger National Intelligence Program (NIP) budget. But some estimates suggest that the NSA's piece of that pie is between 17 and 20 percent—putting its cumulative budget from fiscal year 2006 through 2012, conservatively, at about $58 billion.

I wonder if I turn off every collection devices around the globe Internet will become suddenly faster?Not only they have to analyze everything (slowing down a bit) but also to transmit some information so a big source of traffic would be missing...

My understanding, from previous stories, is that they basically split the fiber-optic signal at the collection point, and the monitoring equipment just passively listens to the stream. So I don't think there's much impact to network performance as nothing is passed downstream from the analyzers.

The Global War On Terror, or GWOT as it was known around DC's beltway, opened up the purse strings for everything on the intelligence, surveillance, and reconnaissance (ISR) shopping list. The NSA's budget is hidden within the larger National Intelligence Program (NIP) budget. But some estimates suggest that the NSA's piece of that pie is between 17 and 20 percent—putting its cumulative budget from fiscal year 2006 through 2012, conservatively, at about $58 billion.

I can be optimistic and imagine a strong enough reaction from the public to at least hinder the surveillance state in this country.. but, what I can't imagine is reversing all this infrastructure.

They've got all these toys -- all these employees -- these buildings and hardware and software... there's no way it's going away. And any law that defeats the designed purpose of these devices would end up circumvented or ignored out of perceived necessity.

You have a car.. you're gonna drive it.. got a gun.. you'll end up shooting it.. have billions of dollars worth of internet surveillance equipment, you aren't just going to put a curtain over it and say "welp, looks like they passed a law that says we can't use this."

I can be optimistic and imagine a strong enough reaction from the public to at least hinder the surveillance state in this country.. but, what I can't imagine is reversing all this infrastructure.

They've got all these toys -- all these employees -- these buildings and hardware and software... there's no way it's going away. And any law that defeats the designed purpose of these devices would end up circumvented or ignored out of perceived necessity.

You have a car.. you're gonna drive it.. got a gun.. you'll end up shooting it.. have billions of dollars worth of internet surveillance equipment, you aren't just going to put a curtain over it and say "welp, looks like they passed a law that says we can't use this."

The Russian site is believed to be in a US Embassy (several European sites are) and the Chinese is a compromised server NSA has control of. You can find information on Cryptome and other Guardian and Foreign Policy articles.