A top secret National Security Agency program allows analysts to search with no prior authorization through vast databases containing emails, online chats and the browsing histories of millions of individuals, according to documents provided by whistleblower Edward Snowden.

The NSA boasts in training materials that the program, called XKeyscore, is its "widest-reaching" system for developing intelligence from the internet.

[T]raining materials for XKeyscore detail how analysts can use it and other systems to mine enormous agency databases by filling in a simple on-screen form giving only a broad justification for the search. The request is not reviewed by a court or any NSA personnel before it is processed.

Greenwald isn't kidding about the "broad justification." The slides tout the breadth of the search program, which provides results other programs can't. As is stated in the opening slides, XKeyscore allows agents to pull up tons of data (in search of "anomalies") and work backward to refine the results. The justification for these broad searches is available via a pulldown menu, as can (sort of) be seen in this screenshot, which gives agents a variety to choose from. (From the list, it appears that anything ending with "outside the US" is fair game.)

XKeyscore utilizes a variety of plugins to allow searches, including email addresses, phone numbers, IP addresses, full logs of every DNI session and machine-specific cookies. This gives agents an advantage other surveillance programs don't.

The purpose of XKeyscore is to allow analysts to search the metadata as well as the content of emails and other internet activity, such as browser history, even when there is no known email account (a "selector" in NSA parlance) associated with the individual being targeted.

Analysts can also search by name, telephone number, IP address, keywords, the language in which the internet activity was conducted or the type of browser used.

One document notes that this is because "strong selection [search by email address] itself gives us only a very limited capability" because "a large amount of time spent on the web is performing actions that are anonymous."

The slides warn that the data collected will be too large to parse (or even store for a great length of time). It recommends harvesting first and "selecting" second, in order to refine the results (using a "Strong Selector"). Agents are directed to look for "anomalous events," some of which seem a bit troubling.

E.g., Someone whose language is out of place for the region they are in

Someone who is using encryption

Someone searching the web for suspicious stuff

These "anomalies" are common enough that plenty of non-terrorists will be getting a second look from agents utilizing this program. And again we see the NSA's instant distrust of anyone using encryption. This is one of the hazards of "collecting it all" and then working backwards. It's easy to make common behavior look suspicious if you start at an end assumption and connect the dots in reverse.

Also troubling are some of the suggested applications of the search program shown in the slide deck, including "show me all the VPNs startups in Country X" and "show me all exploitable machines in Country X."

On top of this, there's the sheer breadth of the program.

The quantity of communications accessible through programs such as XKeyscore is staggeringly large. One NSA report from 2007 estimated that there were 850bn "call events" collected and stored in the NSA databases, and close to 150bn internet records. Each day, the document says, 1-2bn records were added.

The XKeyscore system is continuously collecting so much internet data that it can be stored only for short periods of time. Content remains on the system for only three to five days, while metadata is stored for 30 days. One document explains: "At some sites, the amount of data we receive per day (20+ terabytes) can only be stored for as little as 24 hours."

Because of the massive size of the data haul, metadata is retained and stored longer while more specific data is released. This still allows agents to perform broad searches to gather as much data as possible while relying on the stored metadata to put other connections together. Once they have the connections, the shallow search can be better utilized with the "strong selectors."

The data harvested isn't solely relegated to foreign communications, no matter what the pulldown menu says. The power of the database pretty much guarantees the inadvertent collection of data on American citizens. This is exacerbated by the fact that some web traffic will be indeterminate in origin or termination. This leads to violations of the few laws that do pertain to NSA data collection, something the NSA documents admit is a problem. Of course, as Snowden pointed out, there's always a solution.

In recent years, the NSA has attempted to segregate exclusively domestic US communications in separate databases. But even NSA documents acknowledge that such efforts are imperfect, as even purely domestic communications can travel on foreign systems, and NSA tools are sometimes unable to identify the national origins of communications.

Moreover, all communications between Americans and someone on foreign soil are included in the same databases as foreign-to-foreign communications, making them readily searchable without warrants.

Some searches conducted by NSA analysts are periodically reviewed by their supervisors within the NSA. "It's very rare to be questioned on our searches," Snowden told the Guardian in June, "and even when we are, it's usually along the lines of: 'let's bulk up the justification'."

Speaking of "justification," the slides claim that over 300 terrorists have been caught using XKeyscore. And the NSA has responded to the Guardian's leak with the usual claims that everything here is legal and audited, etc., which, again, doesn't make it right or even constitutional. It just makes it what it is: the end result of more than a decade's worth of expansion, secret law interpretations and compliant administrations.

Re: Re: Words Fail Me

The problem is that the absolute power has so corrupted them that they still believe they are right in what they are doing, that they genuinely are serving the greater good. Delusion that deep is frightening.

Re: Re: Re: Re: Words Fail Me

"Most of them probably are serving the greater good"

No most of them probably THINK they are serving the greater good. Neo-con ideology really started to take hold out of trying to prevent another Pearl Harbor. Prior to that we felt our best policy was to mostly stay out of foreign conflicts apart from simply providing our allies with some requested support. When that sort policy failed twice to keep us safe, the thinking shifted to keeping tabs on and manipulation of foreign affairs as a means of minimizing the ability of situations where we could be attacked. In a nutshell, it's applying the theory of "the best defense is a good offense" to foreign affairs. The initial reasons behind it are still to keep Americans safe. The major downsides to it are doozies: 1. It tends to make a lot of enemies out of people that wouldn't otherwise consider you an enemy. 2. It fosters a us against the world mentality where you have to constantly overcome the collective strength of practically everyone else to make it work and keep it up in the long term. 3. If you can make it work, then those with the power to control the machine that manage it tend to become corrupted by the power that they have undermining all of the nobleness behind the initial ideology, which is where we are today.

Re: Words Fail Me

Unfortunately, I'm afraid this has not much to do with power but with mere bureaucratic estranged conscience. The same that has turned an engineer into a mass executer. I say "unfortunately" because if this were power, it wouldn't last long, victim to its own blind ambitions, and would somehow quickly come out as obvious (e.g. Nixon). The second brand however is silent, unnoticed and, well, just about average.
Overall, what stands out as characteristics seems to me to be stupidity and an appalling lack of elegance. Typical Bush-era stuff perpetuated by the current weak administration.

Re: Lying with facts

Besides you can't really "wiretap just anyone" anymore. Many people don't use or even have POTS lines anymore, making "wiretaps" not as prevalent. You can't really tap a wire on a wireless phone anyway now can you?

Re:

Those documents are 5-6 years old. I am sure that by now (if their programmers are worth a shit) that they have streamlined the process to automatically include the "justification" based on the results of the query.

AGAIN without The Google.

It's amazing how Mike can dodge around mentioning the MAJOR source of "vast databases containing emails, online chats and the browsing histories of millions of individuals".

So I repeat again. Emphasis added:

'Greenwald told ABC News’ George Stephanopoulos. “And what these programs are, are very simple screens, like the ones that supermarket clerks or shipping and receiving clerks use, where all an analyst has to do is enter an email address or an IP address, and it does two things. It searches that database and lets them listen to the calls or read the emails of everything that the NSA has stored, or look at the browsing histories or Google search terms that you’ve entered, and it also alerts them to any further activity that people connected to that email address or that IP address do in the future.”

Re: Re: Re: AGAIN without The Google.

Re: AGAIN without The Google.

Google is not necessarily involved in this particular program. Remember that the NSA has some 15-20 monitoring points at telecom centers across the U.S. HTTP traffic can be collected, filtered, and indexed in those places. Though, as HTTPS becomes more commonly used then searches using Google will have to be monitored with Google's help.

Re: AGAIN without The Google.

Re: AGAIN without The Google.

blue, again, Google has nothing to do with implementing this other than having to do what the government requests in order to keep from causing themselves a world of hurt at the hands of the government. If you want to rant about something wrong Google is doing (all though I admit it is off topic for this particular post) here, I'll throw you a bone...

Back in 1986 I worked on a project that the Marine Corps wanted, adapting the precursor to the Sniffer tool (a network protocol analyzer) so they could monitor their own LAN. I have realized for a long time the theoretical capability of monitoring everything on the Internet. It now scares me to realize this is actually happening. XKEYSCORE takes the cake. I am officially paranoid.

Things that struck me from the power point presentation.

-If you are going to encrypt you're emails, chats, or phone calls, you're drawing attention to yourself. It behooves you to take all possible precautions with the rest of your internet activities. Don't go halfway!

-There are MAC addresses in Excel documents?

-It looks like the NSA analyzes HTTP headers and does browser fingerprinting. This can help to identify your computer even while going through a proxy.

HTTPS everywhere

Someone will pipe up and say: "but I use HTTPS everywhere" extension so that web connections that can use Https, will use Https.

This lulls people into a false sense of security.

Let me point no further than the immediately preceding TechDirt article about how the NSA is in bed with American business, and how this hurts American business.

But first, let me digress. Remember sometime back all the controversy and outrage when Mozilla revoked the SSL signing certificates from a company that had issued root certificates to a third party? In that case, the third party was a company that made border routers for large networks. Those devices could then issue you a genuine signed certificate for, oh, let's just say, Amazon.com, and your web browser would believe it really was talking to Amazon.com. In reality, the intermediate router was what your browser was talking to. Then the router talked to Amazon.com on your behalf. This allowed the intermediate router to intercept, monitor, log or do anything else with your private traffic between you and Amazon.com.

At the time, the end result was that a lot of people began to wonder about just how much SSL and that green reassuring logo in your address bar should be trusted. If you want to Amazon.com, and your browser had a green trust logo, and you clicked it to inspect the certificate, and it was signed by, let's just say, Honest Achmed's Trusty SSL Certificates of Tehran Iran, would you believe that Amazon had purchased their SSL certificates from there?

Now back from my digression to the topic at hand.

Do you suppose that the NSA might secretly make secret arrangements with American certificate authorities (CA's) so that their secret private signing keys and or root certificates are secretly sent to the NSA so that the NSA can secretly play MITM (maniacal monster in the middle) games with your supposedly secure SSL traffic?

I would laugh myself silly if a subsequent leak revealed exactly that.

The entire underlying trust model of supposedly secure traffic on the internet would be broken. Who could trust anything over SSL? Who in other countries could trust American businesses ever again?

Re: Re: HTTPS everywhere

They could only do this with selected targets as it is computationally prohibitive on a large scale.

I don't see any reason why this couldn't be done on a large scale. The computational requirements aren't terribly prohibitive (large, yes, but not prohibitively so). The main constraint would be bandwidth, not CPU cycles, and that's easy to mitigate by scattering your servers across the globe.

Re: Re: Re: HTTPS everywhere

Possibly. That was a seat-of-my-pants estimate. I do know from working on a small router that adding encryption dropped the throughput by an order of magnitude. My estimate may not accurately take into account the gains, since then, provided by GPUs or other hardware that is encryption/decryption specific. Mark Klein described Narus machines filling the small room at the SF ATT center. At that time HTTPS usage was limited. Remember, that with MITM every packet is decrypted and re-encrypted and this is for traffic in both directions. I am wondering why the NSA would go this route rather than demanding more PRISM-like co-operation from the server endpoints.

Re: Re: Re: Re: Re: HTTPS everywhere

This was a low-end software based router. Very cheap, but the CPU was just equivalent of a high-end PC for 2006. In this system, the CPU was the choke point for throughput. Adding encryption/decryption without hardware accelerators made it even more of a choke point. So, that's why I am not sure how much current CPUs with multiple cores, along with GPUs or other specialized hardware would be affected. Still I think this is problematic. Read my next comment.

Re: Re: Re: Re: Re: Re: HTTPS everywhere

Well, in my own home network, I use a cheap consumer grade, 5-year-old router that I've replaced the software on. It is a VPN server and all traffic over my network flows through it and is encrypted (including things like Netflix). My network is under moderately heavy load most of the time.

I did not do benchmarks when I installed the crypto, so I can't give exact figures -- but whatever slowdown the crypto is causing was low enough that it was unnoticeable in practical usage.

Re: Re: Re: HTTPS everywhere

The more I think about this the less likely I think large scale MITM could happen. It is probably theoretically possible. The NSA would have to redesign all the telecom centers where they currently have taps into communication. Instead of just receiving a one-way feed of mirrored traffic they would have to insert themselves into the core routers. Normally core routers don't have hardware accelerators for encryption/decryption. That functionality is left for edge routers that must implement protocols like VPN. So, those routers would have to be a custom job and at the same time as quick and reliable as the equipment they are replacing. I am sure the current core router complement in these centers is nowhere near capable of handling the extra computational requirement for mass MITM functionality.

Re: Re: HTTPS everywhere

I'm not sure why you are saying that MITM can not be done on a mass scale?

Suppose I had secretly obtained the root signing certificate from a CA? For the following, I will use a fictional CA and call it VeriSlime.

Here is what my MITM device would need to do.

When you connect to Amazon.com, I first check if I have ever created a certificate for Amazon.com. If so, then I just use that fake cert to accomplish my MITM between you and Amazon.com.

But what if there is a cache miss? You connect to your small town bank site. I don't have that cert in my cache. So I make a connection to your small town bank site just to obtain its cert. I create a new cert with all the same properties, and sign it with VeriSlime's root cert key. That does not take very long to accomplish. And it only must be done once for that cert. I am not breaking any crypto -- merely performing some routine operations. Then using the new cert, I complete your original connection to your small town bank, but doing MITM, using the new cert.

Unless you are alert you might not notice that your bank certificates used to be signed by another CA, and now are signed by VeriSlime.

But if I was the NSA, I might have the root signing certs for every American CA. Then I could sign my impostor cert for your small town bank using the root cert from the same CA that your bank uses.

Now suppose that even though now I use the right CA to sign all my fake certs, you still notice the thumbprint has changed and might be suspicious. Or suppose the bank could insert JavaScript code in their page to check the cert and insure it is what they expected to see? Or like Google, the Chrome browser checks Google certificates to be sure that they really are what they should be?

Well, if I were the NSA, I might simply require every CA to give me a duplicate of any signing certificates that they issue to their customers. So (new application here...) when Microsoft buys a certificate (but this time instead of SSL, let's say a code signing certificate) the CA will issue me a copy of the certificate. That way I can sign any binary code I want it it REALLY IS signed by Microsoft. Now I can impersonate Microsoft's update servers, and have you do a Windows Update to my MITM server, and I could install any freakin' code I want onto your computer and it would be trusted!

What is so difficult to do on a large scale here if I had, say, a twenty person team of experts working on it, starting, say, five years ago?

One thing, using the Microsoft example, that Microsoft could do is to NOT use code signing certs issued by a CA. Set up their own internal CA that creates root certs, signs code, and put your trusted certs into your products (Windows, Office, etc) so that they only trust your own root certs and no third party is involved. But oh, wait -- Microsoft was working with the NSA either willingly or unwillingly. And Google. And everyone else.

Re: Re: Re: HTTPS everywhere

A MITM operation requires that every packet received is decrypted and then re-encrypted before re-transmitting. That is computationally expensive. Also, the NSA listening points at telecom centers were just mirroring received traffic. If MITM were a consideration, the NSA would have to insert themselves into the switching points and not just get traffic fed to them through a branch from the switches. If I were designing this I would say the NSA is better off expanding their PRISM capabilities.

Signing code is an entirely different matter. I have always wondered if Microsoft wasn't allowing the FBI/CIA/NSA use their update capability to install code on targeted machines. The, recently publicized, fact that Microsoft was selling or providing security vulnerabilities/exploits to the government undermines that suspicion.

Re: Re: Re: Re: HTTPS everywhere

Encrypting / decrypting for SSL is not that expensive. An Apache Tomcat server (java) on a decent hardware*, can easily handle SSL without any outside software / hardware assistance. Yes, I know a distributor in front of it could do the SSL, or a hardware card, or even another web server (like Apache) in front of Apache Tomcat could remove the SSL burden from the Tomcat server (written in Java). (tcnative is also part of this configuration)

*by decent hardware, I mean like a server with Xeon X5560. That CPU chip costs upward of $1000, (but it includes a heat sink! :-) ) Then buy enough of those chips to fill all the sockets on the motherboard. For only thousands of dollars a server with no special hardware assistance can easily handle a lot of SSL traffic without even breaking a sweat. I promise. That includes serving even static resources (graphics, js, css, etc over SSL) And if I ever need to offload the SSL onto other hardware, this is easy to do in several different ways, and totally transparent to the application. And it would also be very easy to move static resources to another server software (Apache or other), or even another server hardware. But in terms of economics, if it is not even breaking a sweat today, why bother until necessary.

Re: Re: Re: HTTPS everywhere

Bluffdale is just a storage center, that is still not capable of handling the raw feed of all the voice/internet traffic that is routed through a distributed set of telecom switching centers. It's scary enough what that data center will be capable of holding but it's not everything and it's not with the additional requirement of acting as MITM to all that traffic.

Re: Re:

Re:

What we need to do, is take the time to have our address and phone number represent something, and therefore be a creative expression and then shut down the collection of our metadata on copyright grounds.

To use the logic of the company that created bank routing numbers and the trolls who defended them.

The vastness of the data collected is simply mind boggling. It gets worse each day with Govt lies being exposed as they are fresh in the public collective minds. This is getting epic proportions. I wonder how much damage the Govt or US tech companies will take before they finally start doing something to fix the issue...

Re:

As far as the router/switch companies go. I worked for a major competitor of Cisco and they did not have anything in the software that secretly allowed for general monitoring. CALEA conformance was basically assigning the use of a general purpose mirroring port which was not controlled separately from full administrative control of the router or switch involved. Any code residing in ASICS or FPGAs had to interact with the software. I can't speak for Cisco's routers, but it would be hard to keep backdoors secret. Usually, any software engineer that worked on a router has access to all the code for that router.

Re: Re: Re:

I had access to all the code. I did go through a lot of it but not everything (really a lot of code). The stuff I did not go through was not handling the majority of traffic flow in that they implemented specialized protocols. I was privy to all the low-level interaction with the hardware and I knew everything about configuring a mirroring port.
Can you point me to a discussion of backdoors for Cisco? Also, what other router companies are you thinking of?

Re: Re: Re: Re:

The backdoors would not be in the code proper, they would be in the hardware and require no interaction with code outside the chip.

I cannot give you a list of the routers that I know about personally right now, but a quick web search turns up a list of usual suspects, including Cisco.

Unless you're doing packet analysis of the traffic to and from a router while the back door is actually in use or attempt a known exploit and find that it succeeds, it's almost impossible to be sure that the router is not compromised -- even if you can guarantee that the higher-level code isn't.

This is one reason why I don't use commercial routers at all between my network and the internet (although I do use them for internal routing).

Re: Re: Re: Re:

you may think that you had access to all the code, but it could be easy to hide things where you wouldn't expect to look. Did you have access to the build machines? Do you know the ins, outs, and innards of the build system? I can tell you for a fact, the build process provided by microsoft for visual studio has open insertion points where I could add more source files/change the compiler/change the linker/completely replace the whole build process with just a text file and setting an environment variable.

Re: Re: Re: Re: Re:

I was very familiar with the build process and sometimes helped them when problems arose. Both engineering and build used the same source database. No, I didn't have actual access to the build machine but if there were added source there would be a difference in size of the binary between what they built and what was on my development machine. Believe me, I checked this often just to ensure no foul-ups occurred.
I did a lot of testing of throughput including accounting for every single packet received on a port and where it went. These counts occurred in standard industry hardware outside of our proprietary ASICs and FPGA code. I would have noticed a discrepancy. If there was a backdoor in an ASIC it would still have to be triggered or configured by software. Even if there was a secret configurable register, there needed to be software that handled reads or writes to that specific interface. I knew all the low-level software. The only possibility I can see is if the compiler itself had been altered to add secret code to all the builds. I just find that hard to believe the company would go to that degree of trouble and risk screwing up any logic that would be impossible for most of the developers to debug.

Re: Page 24

Were I a spook I would've been using/encouraging known exploits (especially those "you heard it here first" special edition deliveries) to exploit any and every machine on a wire, period. Wire tapped, key logged, private locks and private doors. select x-morecontent from theworld where z-content in(kill, bomb, jihad, '%yourgirlfriend%', selfie);

it was quite obvious before this why Rogers was condemning what Snowden was saying. i thik he anticipates making a nice little bonus from the security companies his family are associated with. now add in the fact that he didn't want it known for sure just how big a liar he is, and you can understand why he was shouting so loud. now this tidbit is out, it's blatantly obvious why he was shouting! isn't it funny how the most guilty, shouts the loudest to try to deflect from them on to anyone else. i sure hope that someone is keeping a list of those that need replacing come voting time. he definitely needs to lose his job!!

Every day/week/month something new is coming out about the NSA and the dastardly deeds enabled by a free for all in spying that doesn't take any consideration about what the grounding laws actually say.

We the public have been lied to so often and frequently that trust is no longer possible. It is far and beyond time to end the Patriot Act and other laws put on the books that enabled this sort of massive spying. I'm not sure that just defunding NSA is enough.

What I am sure of is I don't recognize this country as the one I served in the military for. This country as it is being revealed begins to look more and more like Russia or China in it's keeping track of the populace. With absolutely no justifications beyond 'it might' qualifying. Give 'it might' to the paranoid and it becomes a certainly even through it is never proven to actually be so.

Pg. 28

"Over 300 terrorists have been captured using intelligence from XKeyscore"

Perhaps their claims that "we've stopped terrorist attacks with this surveillance!" might have a little credibility.

Of course, we'd be taking a leap of faith and assuming that all the terrorists the NSA has helped stop are actually terrorists who were planning to, well, cause terror among the general populace (American or otherwise), and not some unlucky bastard who got a nasty case of "mistaken identity" and was dragged in with the real threats because the government couldn't risk letting the guy go because he'd make a big fuss about everything.

Re: Pg. 28

Captured isn't the same as convicted (or even charged with a crime).
As part of their story, I'd like to know where these captured terrorists are (Bagram? Abu Ghraib?) and who did the capturing (US or another nation).

Of course, since its only cost effective if you need to store data for at least 600 years, the NSA is the primary candidate to implement DNA DATA STORAGE. They may decide to retain your personal data for a longer period of time, to see if your grandgrandgrandgrandgrandgrandsons arent a threat.

NASA Spying

"If people can't trust not only the executive branch, but also don't trust Congress and don't trust federal judges to make sure that we're abiding by the Constitution, due process and rule of law, then we're going to have some problems here," ...Obama
Well, when the Executive Branch, Congess and the courts provide a reason as to why we should trust them on this issue, then maybe we won't have a problem. Implicit in Obama's statement is that the American people should "trust us, we have your interests at heart". The fact of the matter is that since 9/11 the Surveillance State has grown exponentially, with little or no dialogue on the part of the Executive branch, Congress (with the exception of Senators Wyden and Udall) and the courts with the American people, regarding the tradeoff between civil liberties and the role of surveillance in 21st Century America. Furthermore, without Edward Snowden this conversation wouldn't be taking place, even now.