"Primary objective" of document dump is to allow public to scour them for stories.

Share this story

Several years ago, nobody would have believed you if you said that a secret US court was ordering the nation's telecoms to forward the metadata for all telephone calls coming to and from the United States to the National Security Agency. It would have sounded like fiction from some deranged person wearing a tinfoil hat. But it was true. Edward Snowden, a former contractor for the NSA, turned over internal government documents in 2013 that illustrated just that reality. Future document releases would underscore that the United States had been spying on its populace and the world at large to a breathtaking extent.

Snowden, now living in Russia, handed over the documents to reporter Glenn Greenwald, who published many of the juiciest disclosures at the Guardian. Greenwald left the Guardian and took the documents with him to The Intercept, which announced Monday that it is beginning a public document dump of the goods provided by Snowden. Today, The Intercept is releasing its first batch of many classified documents—166 articles of the NSA's internal newsletter called SIDtoday. The site explained:

The Intercept’s first SIDtoday release comprises 166 articles, including all articles published between March 31, 2003, when SIDtoday began, and June 30, 2003, plus installments of all article series begun during this period through the end of the year. Major topics include the National Security Agency’s role in interrogations, the Iraq War, the war on terror, new leadership in the Signals Intelligence Directorate, and new, popular uses of the internet and of mobile computing devices.

Greenwald encouraged "journalists, researchers, and interested parties" to sift through these and forthcoming document dumps "to find additional material of interest. Others may well find stories, or clues that lead to stories, that we did not. (To contact us about such finds, see the instructions here.) A primary objective of these batch releases is to make that kind of exploration possible."

Greenwald added that he and other staff members had redacted documents "consistent with the requirements of our agreement" with Snowden in a bid to prevent "serious harm" to "innocent individuals."

Staff technologist Micah Lee was responsible for putting this online newsletter into a readable format, from "messy HTML to clean PDF." The Intercept explained:

In its original home at the NSA, SIDtoday is a website, but, unlike normal websites that are accessible from the internet, this one is only accessible from computers that are connected to an internal spy agency network. This means SIDtoday articles, as provided by Snowden, were in Hypertext Markup Language, or HTML, the native format of the web. To be fully readable, most webpages require HTML files as well as associated images, stylesheets, and other files, but unfortunately those other elements were not included in the Snowden archive. This meant the original webpages looked rather chaotic when viewed in a web browser. They also included broken links to the internal NSA network.

Staff technologist Micah Lee prepared software that parsed the original SIDtoday HTML files and extracted the usable content. This content was then placed into a template with minimalist design and converted to PDF format. PDF files, unlike HTML files, could be edited by multiple staffers using widely available, easy-to-use software tools with native redaction facilities.

The PDF files don’t look the same way the original SIDtoday website looked—they don’t use the original layout or style elements—but they do contain the same text content. They also include the original clip art-style SIDtoday logo.

Share this story

David Kravets
The senior editor for Ars Technica. Founder of TYDN fake news site. Technologist. Political scientist. Humorist. Dad of two boys. Been doing journalism for so long I remember manual typewriters with real paper. Emaildavid.kravets@arstechnica.com//Twitter@dmkravets

Staff technologist Micah Lee prepared software that parsed the original SIDtoday HTML files and extracted the usable content. This content was then placed into a template with minimalist design and converted to PDF format. PDF files, unlike HTML files, could be edited by multiple staffers using widely available, easy-to-use software tools with native redaction facilities.

I thought the point of Snowden leaking everything to journalists rather than the general public was to prevent unnecessary declassification of sensitive information. Doesn't this kind of go against that principle?

Edit: I guess they redacted parts. But it is also implied they might not have looked through it closely.

Several years ago, nobody would have believed you if you said that a secret US court was ordering the nation's telecoms to forward the metadata for all telephone calls coming to and from the United States to the National Security Agency.

Unless they had taken an interest in the NSA and read James Bamford's The Puzzle Palace, published in 1983. http://www.amazon.com/Puzzle-Palace-Nat ... 0140067485The mechanisms are updated and the court cover they has use has changed over the years, but what they do has barely changed since Corderman had people photographing every telegram going in and out of the country starting in 1945.

Come on people, isn't this story ever going to die? I mean, there's other important things and stuff. Trump and a centerfold might be your new leaders next year, and Twinkies, remember Twinkies? Well, they're back! All kinds of rilly, rilly important crappola is going on. And all you people are just sitting around gossiping about old news like some old ladies at the salon.

Several years ago, nobody would have believed you if you said that a secret US court was ordering the nation's telecoms to forward the metadata for all telephone calls coming to and from the United States to the National Security Agency. It would have sounded like fiction from some deranged person wearing a tinfoil hat.

The Wikipedia article on ECHELON has 66 references to publications that describe/discuss the electronic comms gathering sytem that's been in place since 1971. And those are just the most recent references. I first learned about it in 1980, and anyone, and I mean anyone, who's taken even a casual interest in the subject in the last 40 years knows about it.

And about the court - it's not a "secret" court if it's public knowledge.

Several years ago, nobody would have believed you if you said that a secret US court was ordering the nation's telecoms to forward the metadata for all telephone calls coming to and from the United States to the National Security Agency.

Unless they had taken an interest in the NSA and read James Bamford's The Puzzle Palace, published in 1983. http://www.amazon.com/Puzzle-Palace-Nat ... 0140067485The mechanisms are updated and the court cover they has use has changed over the years, but what they do has barely changed since Corderman had people photographing every telegram going in and out of the country starting in 1945.

Exactly what I was thinking, though the Mark Klein reveal of the Narus fiber optic splitters installed at AT&T was what came to mind - obviously they were sending everything going through the main trunk network to some other location, and the only reason to do that was NSA snooping - and not just metadata, either, the content of emails, voice-transcripts of calls, all of it seems to have been collected and searched with applications like TRAFFICTHIEF, then dumped into various depositories if of 'interests'.

Yes, that's searching the private communications of American citizens without a warrant, in violation of the U.S. Constitution.

The NSA is as lazy as the FBI. And it is all for naught. While they try to poke around in the contents of messages, they would find a clearer way to analyze the metadata. But, alas, we live in the era of the Bozo.

My biggest concern is that... not a lot of other Americans seem concerned any more. This was a horrid defamation of everything the United States and its Constitution stands (or stood) for, and its citizens don't really seem to give a fuck.

Have they become that jaded? Has the lead in the water finally dropped the collective IQ to the point that the sight of a Kardashian tit blows all other rational thought from their heads? It honestly rattles me. It shows the government that they really CAN do anything, and the people just won't care.

Several years ago, nobody would have believed you if you said that a secret US court was ordering the nation's telecoms to forward the metadata for all telephone calls coming to and from the United States to the National Security Agency. It would have sounded like fiction from some deranged person wearing a tinfoil hat.

The Wikipedia article on ECHELON has 66 references to publications that describe/discuss the electronic comms gathering sytem that's been in place since 1971. And those are just the most recent references. I first learned about it in 1980, and anyone, and I mean anyone, who's taken even a casual interest in the subject in the last 40 years knows about it.

And about the court - it's not a "secret" court if it's public knowledge.

Please, give the breathless drama voice a rest.

A court with secret hearings is a secret court.

Is that right? So then every grand jury ever impaneled in the US has been a secret court?

Yes, derp. The fuck did you think a court with secret hearings, secret witnesses, a prosecutor presenting only evidence (secret evidence!) from their side, etc, was?

It's one of the biggest arguments for grand juries being dispensed with, although one can make the argument that, since a grand jury necessarily feeds into the regular court system, it's significantly less damaging.

[quote]In its original home at the NSA, SIDtoday is a website, but, unlike normal websites that are accessible from the internet, this one is only accessible from computers that are connected to an internal spy agency network.[/quote

Staff technologist Micah Lee prepared software that parsed the original SIDtoday HTML files and extracted the usable content. This content was then placed into a template with minimalist design and converted to PDF format. PDF files, unlike HTML files, could be edited by multiple staffers using widely available, easy-to-use software tools with native redaction facilities.

Redaction tools that "black-bar" til you look at the source.

I ran one of the redacted PDFs through a couple text extraction tools. As far as I can tell, the text isn't there anymore.

In its original home at the NSA, SIDtoday is a website, but, unlike normal websites that are accessible from the internet, this one is only accessible from computers that are connected to an internal spy agency network.[/quote

Seems like an oddly long way of saying it's on the NSA intranet.

Not everyone knows what "intranet" means. Heck, I know people who regularly work with their company's intranet and some just call it the "internet" because it requires the Google Chrome to use. Of course, this is Ars...

Several years ago, nobody would have believed you if you said that a secret US court was ordering the nation's telecoms to forward the metadata for all telephone calls coming to and from the United States to the National Security Agency. It would have sounded like fiction from some deranged person wearing a tinfoil hat.

The Wikipedia article on ECHELON has 66 references to publications that describe/discuss the electronic comms gathering sytem that's been in place since 1971. And those are just the most recent references. I first learned about it in 1980, and anyone, and I mean anyone, who's taken even a casual interest in the subject in the last 40 years knows about it.

And about the court - it's not a "secret" court if it's public knowledge.

Please, give the breathless drama voice a rest.

A court with secret hearings is a secret court.

Is that right? So then every grand jury ever impaneled in the US has been a secret court?

Yes, derp. The fuck did you think a court with secret hearings, secret witnesses, a prosecutor presenting only evidence from their side, etc, was?

It's one of the biggest arguments for grand juries being dispensed with, although one can make the argument that, since a grand jury necessarily feeds into the regular court system, it's significantly less damaging.

Thanks for the thoughtful and reasoned response, jackass.

The grand jury is a way to judge the evidence against a not-yet-accused in order to provide multiple protections for the not-yet-accused:1. Their identity is not released to the public.2. The potential accusation is taken out of the hands of a potentially politically-driven prosecutor's office.

Clearing this up for you, my point was that public knowledge of the existence of a court means it's existence is not a secret. I'll go further and say that the sealed proceedings of a review process, provided over by a judge, makes it a court but does not mean the proceedings are necessarily proper fodder for public consumption.

Well, good to know that there's judges in secret courts acting in the public's interest. Sure seems reasonable to keep a legal holdover from medieval times as a secret court rubber stamping for prosecutors, instead of just using a regular indictment, and I hope you enjoyed your semantic quibbling and bitching over one word, used correctly, in the original story.

Several years ago, nobody would have believed you if you said that a secret US court was ordering the nation's telecoms to forward the metadata for all telephone calls coming to and from the United States to the National Security Agency. It would have sounded like fiction from some deranged person wearing a tinfoil hat.

The Wikipedia article on ECHELON has 66 references to publications that describe/discuss the electronic comms gathering sytem that's been in place since 1971. And those are just the most recent references. I first learned about it in 1980, and anyone, and I mean anyone, who's taken even a casual interest in the subject in the last 40 years knows about it.

And about the court - it's not a "secret" court if it's public knowledge.

Please, give the breathless drama voice a rest.

A court with secret hearings is a secret court.

Is that right? So then every grand jury ever impaneled in the US has been a secret court?

Yes, derp. The fuck did you think a court with secret hearings, secret witnesses, a prosecutor presenting only evidence from their side, etc, was?

It's one of the biggest arguments for grand juries being dispensed with, although one can make the argument that, since a grand jury necessarily feeds into the regular court system, it's significantly less damaging.

Thanks for the thoughtful and reasoned response, jackass.

The grand jury is a way to judge the evidence against a not-yet-accused in order to provide multiple protections for the not-yet-accused:1. Their identity is not released to the public.2. The potential accusation is taken out of the hands of a potentially politically-driven prosecutor's office.

Clearing this up for you, my point was that public knowledge of the existence of a court means it's existence is not a secret. I'll go further and say that the sealed proceedings of a review process, provided over by a judge, makes it a court but does not mean the proceedings are necessarily proper fodder for public consumption.

Children, please. "Secret" court doesn't need its existence to be secret. Nor are proceedings held in confidence automatically suspect.

What was, and is troubling is the one-sided nature of the decisions that come out of this court, with no apparent recourse to oversight or appeal.

My biggest concern is that... not a lot of other Americans seem concerned any more. This was a horrid defamation of everything the United States and its Constitution stands (or stood) for, and its citizens don't really seem to give a fuck.

Have they become that jaded? Has the lead in the water finally dropped the collective IQ to the point that the sight of a Kardashian tit blows all other rational thought from their heads? It honestly rattles me. It shows the government that they really CAN do anything, and the people just won't care.

I can think of a few reasons why people might not give a fuck.

Giving a fuck about these documents being leaked requires someone who is actively seeking this information. It's not going to be anywhere on the news during this election cycle, and even if it were, it would be a short 1-3 minute piece.

All of this is now highly politicized, and just like we have people who are convinced that waterboarding 'terrorists' is necessary and useful to national security we have people that will espouse things like "Snowden should be in jail for treason", but wouldn't be able to write an index card's worth of knowledge about the topic.

My biggest concern is that... not a lot of other Americans seem concerned any more. This was a horrid defamation of everything the United States and its Constitution stands (or stood) for, and its citizens don't really seem to give a fuck.

Have they become that jaded? Has the lead in the water finally dropped the collective IQ to the point that the sight of a Kardashian tit blows all other rational thought from their heads? It honestly rattles me. It shows the government that they really CAN do anything, and the people just won't care.

I can think of a few reasons why people might not give a fuck.

Giving a fuck about these documents being leaked requires someone who is actively seeking this information. It's not going to be anywhere on the news during this election cycle, and even if it were, it would be a short 1-3 minute piece.

All of this is now highly politicized, and just like we have people who are convinced that waterboarding 'terrorists' is necessary and useful to national security we have people that will espouse things like "Snowden should be in jail for treason", but wouldn't be able to write an index card's worth of knowledge about the topic.

This is why I'm glad for John Oliver (and Jon Stewart before him). They've both covered these topics. Of course, the fact that they're satirists doing a better job of journalism than the news media is an entirely different topic for commentary.

Come on people, isn't this story ever going to die? I mean, there's other important things and stuff. Trump and a centerfold might be your new leaders next year, and Twinkies, remember Twinkies? Well, they're back! All kinds of rilly, rilly important crappola is going on. And all you people are just sitting around gossiping about old news like some old ladies at the salon.

Look, Kettle chips flavor: Zesty Ranch, now that's a real story!

Well, and then there is Hillary out-Trumping Trump and specifically advocating expanding the NSA's capabilities. Except when it comes to her own Blackberry - then it's OK to violate the NSA's rules and run it off her own server.

Unfortunately, it looks like we may be stuck with two equally bad candidates.

Several years ago, nobody would have believed you if you said that a secret US court was ordering the nation's telecoms to forward the metadata for all telephone calls coming to and from the United States to the National Security Agency.

Unless they had taken an interest in the NSA and read James Bamford's The Puzzle Palace, published in 1983. http://www.amazon.com/Puzzle-Palace-Nat ... 0140067485The mechanisms are updated and the court cover they has use has changed over the years, but what they do has barely changed since Corderman had people photographing every telegram going in and out of the country starting in 1945.

Exactly what I was thinking, though the Mark Klein reveal of the Narus fiber optic splitters installed at AT&T was what came to mind - obviously they were sending everything going through the main trunk network to some other location, and the only reason to do that was NSA snooping - and not just metadata, either, the content of emails, voice-transcripts of calls, all of it seems to have been collected and searched with applications like TRAFFICTHIEF, then dumped into various depositories if of 'interests'.

Yes, that's searching the private communications of American citizens without a warrant, in violation of the U.S. Constitution.

Question for journalists and readers alike: Where is the line between acceptable and unacceptable disclosure?

The point of whistleblowing (as I understand it) is to shed light on things that are unjust or unethical but were kept in the dark either by corporate or governmental power, and the people doing the whistle blowing are generally considered to be doing a public service.

For example, Private Manning releasing "collateral murder", a video where US pilots laughed as they murdered what appear to be civilians from a gunship, is whistleblowing. Private Manning releasing millions of classified of Dept of State cables he didn't take the time to read because "theres some good bits in them that expose our leaders as hypocrits" is espionage/treason.

Snowden releasing classified information about NSA constitutional overreach to the press, whistleblowing. The press releasing classified information about NSA "hoping someone can find some good bits in them" is what exactly?

My apologies if the question is loaded. I really would like to understand where people on both sides are coming from. I personally see the discussion of overreach and the revelations of such as positive for the country, but I think bulk dumps of "maybe theres something in here" is irresponsible and unethical given the content.

My biggest concern is that... not a lot of other Americans seem concerned any more. This was a horrid defamation of everything the United States and its Constitution stands (or stood) for, and its citizens don't really seem to give a fuck.

Have they become that jaded? Has the lead in the water finally dropped the collective IQ to the point that the sight of a Kardashian tit blows all other rational thought from their heads? It honestly rattles me. It shows the government that they really CAN do anything, and the people just won't care.

The combination of citizen apathy and slow dissolution of privacy over the years has left naught but a shell of Democracy. Plus, toss in a Kardashian tit or two, and there you go.Can't fight tits.

Based on the redactions, that would imply it's not being released rough shod; there is a distinction to be made between completely untouched documents, and ones that have been passed through to remove improper information from civvies' hands.

My biggest concern is that... not a lot of other Americans seem concerned any more. This was a horrid defamation of everything the United States and its Constitution stands (or stood) for, and its citizens don't really seem to give a fuck.

Have they become that jaded? Has the lead in the water finally dropped the collective IQ to the point that the sight of a Kardashian tit blows all other rational thought from their heads? It honestly rattles me. It shows the government that they really CAN do anything, and the people just won't care.

They're quite literally the proles from 1984. Don't know, don't care, as long as they have their bread and circuses.

My biggest concern is that... not a lot of other Americans seem concerned any more. This was a horrid defamation of everything the United States and its Constitution stands (or stood) for, and its citizens don't really seem to give a fuck.

Have they become that jaded? Has the lead in the water finally dropped the collective IQ to the point that the sight of a Kardashian tit blows all other rational thought from their heads? It honestly rattles me. It shows the government that they really CAN do anything, and the people just won't care.

They're quite literally the proles from 1984. Don't know, don't care, as long as they have their bread and circuses.

Several years ago, nobody would have believed you if you said that a secret US court was ordering the nation's telecoms to forward the metadata for all telephone calls coming to and from the United States to the National Security Agency. It would have sounded like fiction from some deranged person wearing a tinfoil hat.

The Wikipedia article on ECHELON has 66 references to publications that describe/discuss the electronic comms gathering sytem that's been in place since 1971. And those are just the most recent references. I first learned about it in 1980, and anyone, and I mean anyone, who's taken even a casual interest in the subject in the last 40 years knows about it.

And about the court - it's not a "secret" court if it's public knowledge.

Please, give the breathless drama voice a rest.

A court with secret hearings is a secret court.

They also make secret decisions and set secret precedents. The names of the judges is also a secret.