NSA wiretapping — a 4th Amendment violation? Blake Norvell at TEDxSMU

John Brennan, Obama involved in bugging Trump Campaign during election

EDWARD SNOWDEN Everything about Donald Trump

NSA Whistleblower: Everyone in US under virtual surveillance, all info stored, no matter the post

William Binney – The Government is Profiling You (The NSA is Spying on You)

United States of Secrets: William Binney

NSA Whistleblower William Binney: The Future of FREEDOM

AUTHOR: BRIAN BARRETT.BRIAN BARRETT SECURITY

IF THE FEDS DID WIRETAP TRUMP TOWER, IT’S NOT OBAMA WHO SHOULD WORRY

Trump Tower in New York City, NY.JABIN BOTSFORD/GETTY IMAGES

IT STARTED, LIKE so many eruptions these days, with a tweet.

Early Saturday morning, President Trump fired off a series of tweets accusing, without evidence, former President Barack Obama of wiretapping Trump Tower in the month before the election. Trump compared the alleged snooping to “Nixon/Watergate,” and intimated legal action.

What makes the broader allegation so extraordinary isn’t that it is new. Quite the contrary. Various reports that the Foreign Intelligence Surveillance Court granted Justice Department investigators a warrant to probe the Trump campaign’s ties with Russia surfaced in November. What makes Trump’s Twitter tirade so striking is what prompted it, and what it might imply if it’s true.

Anatomy of an Allegation

Baffling as it may be, it appears Trump’s accusation stems from a recent article published on Breitbart, the conservative news outlet formerly run by White House senior adviser Stephen Bannon.

“This is a somewhat stunning, in so far as the president of the United States doesn’t need to get his information about classified activity from Breitbart,” says Cato Institute fellow Julian Sanchez.

That story, “Mark Levin to Congress: Investigate Obama’s ‘Silent Coup’ Vs. Trump,” rehashes comments the titular conservative radio host made Thursday equating the previously reported FISA warrant with a “police state,” and accuses Obama of a politically motivated, covert attempt to undermine Trump and his associates.

It’s unclear just what prompted Levin’s rant, or why Trump glommed onto it. Although no one has confirmed a FISA investigation, or wiretaps in Trump Tower, several news outlets have reported the former’s existence. The most detailed account thus far, from the BBC in January, provided a timeline: The Justice Department sought a FISA warrant in June to intercept communications from two Russian banks suspected of facilitating donations to the Trump campaign. The judge reportedly rejected the warrant, as well as a narrower version sought in July. A new judge granted the order in mid-October, according to the BBC.

None of this necessarily makes Trump’s allegations true. Even if a FISA warrant exists, it does not mean Trump Tower is tapped or that Trump specifically is the target. Further complicating things, the existence of a wiretap would not necessarily confirm the existence of a FISA warrant. Almost half of the building’s 58 floors are dedicated to commercial and office space, and any one of them—not to mention the building’s residents—could be the target of an investigation unrelated to international espionage or election tampering.

“If he has evidence that he was wiretapped without a proper FISA order being sought, that would be a huge scandal, and he should produce whatever evidence he’s got,” says Sanchez. “It’s a pretty serious claim, and it’s striking he would make it without anything solid to back it up.”

Republican Senator Ben Sasse called on the president to clarify his claims, stating that “we are in the midst of a civilization-warping crisis of public trust.” Obama spokesperson Kevin Lewis strongly denied extra-judicial surveillance of any US citizens to Politico in response to the claims..

Tower of FISA

If nothing else, Trump’s tweets show he doesn’t understand how the FISA system works. If he did, he may have limited himself to tweeting about Arnold Schwartzenegger quitting The Apprentice this morning.

“While the order would have been requested by some part of the executive branch, Obama can’t order anything. Nor can Trump,” says former NSA lawyer April Doss, who stresses that her comments are based only on public information. “The order has to come from the court, and the court operates independently.”

FISA court judges serve seven-year appointments, so the court’s composition doesn’t ebb and flow with the political tides. What’s more, specific laws adopted in the wake of Watergate prevent the very activity Trump accuses Obama of.

“You can’t tap the phones of a political candidate for political purposes,” says Doss.

What you could tap them for? Acting as a foreign power, or as an agent of a foreign power. In other words, spying against US interests with both knowledge and intent.

Clearing that bar is difficult, by design. FISA warrants don’t allow for broad wiretaps of, say, every call going in and out of a specific office in a 58-story Manhattan skyscraper. Federal authorities must demonstrate not just probable cause, but that a given phone line serves primarily to undermine US interests. It’s difficult, for instance, to obtain a warrant to wiretap a shared office, for fear of picking up innocent third-party conversations.

“I have high confidence that a FISA court judge would not have authorized any warrant unless it met all the requirements under the statute,” says Doss.

Trump’s wiretap claims, then, carry presumably inadvertent implications. First, based on previous reporting and the nature of FISA courts, any wiretaps within Trump Tower would be legal. And they would stem from overwhelming evidence that the Trump campaign, or someone within it, has unsavory ties to Russia or another foreign power. Otherwise, it’s unlikely those wiretaps would exist at all.

If federal authorities did have cause to listen in on Trump Tower, though, and they provided enough evidence for a FISA court to approve the snooping, Obama is not the one who ought to worry.

With additional reporting by Andy Greenberg.

This story has been updated to include responses from Obama spokesperson Kevin Lewis and GOP Senator Ben Sasse, and to reflect that FISA court judges serve seven-year terms, not lifetime tenure.

Trump asks Congress to probe alleged illicit campaign investigations

US President Donald Trump pictured during a meeting with parents and teachers at Saint Andrew Catholic School in Orlando, Florida, on March 3, 2017 (AFP Photo/Nicholas KAMM)

More

Washington (AFP) – President Donald Trump is asking Congress to probe “potentially politically motivated investigations” during the 2016 campaign, the White House said Sunday.

The announcement came one day after Trump took to Twitter to accuse his predecessor Barack Obama of tapping his phones ahead of the November election, without providing evidence of the explosive charge.

“President Donald J. Trump is requesting that as part of their investigation into Russian activity, the congressional intelligence committees exercise their oversight authority to determine whether executive branch investigative powers were abused in 2016,” Spicer said.

He added that there would be no more comment on the matter from Trump or the White House.

Trump leveled his charges against Obama early Saturday, at the end of a week in which his administration was battered by controversy over communications between Russian officials and some of his senior aides including Attorney General Jeff Sessions.

“I’d bet a good lawyer could make a great case out of the fact that President Obama was tapping my phones in October, just prior to Election!” Trump wrote.

“How low has President Obama gone to tapp (sic) my phones during the very sacred election process. This is Nixon/Watergate. Bad (or sick) guy!” he wrote in another tweet, referring to the political scandal that toppled president Richard Nixon in 1974.

Trump, Offering No Evidence, Says Obama Tapped His Phones

President Barack Obama and President-elect Donald J. Trump on Inauguration Day. Mr. Trump has praised Mr. Obama repeatedly since taking office. But on Saturday, he called his predecessor a “bad (or sick) guy.”CreditDamon Winter/The New York Times

WEST PALM BEACH, Fla. — President Trump on Saturday accused former President Barack Obama of tapping his phones at Trump Tower the month before the election, taking to Twitter to call his predecessor a “bad (or sick) guy.”

Without offering any evidence or providing the source of his information, Mr. Trump fired off a series of Twitter messages claiming that Mr. Obama “had my ‘wires tapped.’ ” He likened the supposed tapping to “Nixon/Watergate” and “McCarthyism.”

A spokesman for Mr. Obama said any suggestion that the former president had ordered such surveillance was “simply false.”

Mr. Trump’s aides declined to clarify whether the president’s explosive allegations were based on briefings from intelligence or law enforcement officials — which could mean that Mr. Trump was revealing previously unknown details about an investigation — or on something else, like a news report.

His decision to lend the power of his office to such a charged claim against his predecessor — without offering any initial proof — was remarkable, even for a leader who has repeatedly shown himself willing to make assertions that are false or based on dubious sources.

It would have been difficult for federal agents, working within the law, to obtain a wiretap order to target Mr. Trump’s phone conversations. It would have meant that the Justice Department had gathered sufficient evidence to persuade a federal judge that there was probable cause to believe he had committed a serious crime or was an agent of a foreign power, depending on whether it was a criminal investigation or a foreign intelligence one.

Former officials pointed to longstanding laws and procedures intended to ensure that presidents cannot wiretap a rival for political purposes.

“A cardinal rule of the Obama administration was that no White House official ever interfered with any independent investigation led by the Department of Justice,” said Kevin Lewis, a spokesman for Mr. Obama. “As part of that practice, neither President Obama nor any White House official ever ordered surveillance on any U.S. citizen.”

But a senior White House official said that Donald F. McGahn II, the president’s chief counsel, was working on Saturday to secure access to what Mr. McGahn believed was an order issued by the Foreign Intelligence Surveillance Court authorizing some form of surveillance related to Mr. Trump and his associates.

The official offered no evidence to support the notion that such an order exists. If one does, it would be highly unusual for a White House to order the Justice Department to turn over such an investigative document, given the traditional independence of law enforcement matters.

It has been widely reported that there is a federal investigation, which began during the 2016 presidential campaign, into links between Trump associates and the Russians. That issue has dogged Mr. Trump for months.

In one message, which Mr. Trump sent from his Palm Beach, Fla., estate at 6:35 a.m., the president said he had “just found out” that his phones had been tapped before the election. Mr. Trump’s reference to “wires tapped” raised the possibility that he was referring to some other type of electronic surveillance and was using the idea of phone tapping loosely.

Two people close to Mr. Trump said they believed he was referring to a Breitbart News article, which aides said had been passed around among his advisers. Mark Levin, a conservative radio host, had also embraced the theory recently in a push against what right-leaning commentators have been calling the “deep state.”

The Breitbart article, published on Friday, claimed that there was a series of “known steps taken by President Barack Obama’s administration in its last months to undermine Donald Trump’s presidential campaign and, later, his new administration.”

If Mr. Trump was motivated to take to Twitter after reading the Breitbart article or listening to Mr. Levin, he was using a presidential megaphone to spread dark theories of a broad conspiracy aimed at undermining his presidential ambitions, and later his presidency.

Even with the Breitbart article circulating, several of Mr. Trump’s advisers were stunned by the president’s morning Twitter outburst. Those advisers said they were uncertain about what specifically Mr. Trump was referring to; one surmised that he may also have been referring to a months-old news report about a secret surveillance warrant for communications at his New York offices.

One senior law enforcement official from the Obama administration, who has direct knowledge of the F.B.I. investigation into Russia and of government wiretapping, said that it was “100 percent untrue” that the government had wiretapped Mr. Trump. The official, who asked for anonymity to discuss matters related to investigations and intelligence, said the White House owed the American people an explanation for the president’s allegations.

Ben Rhodes, a former top national security aide to Mr. Obama, said in a Twitter message directed at Mr. Trump on Saturday that “no president can order a wiretap” and added, “Those restrictions were put in place to protect citizens from people like you.”

The House and Senate Intelligence Committees are moving forward with their own investigations into Russia’s efforts to influence the election, and they have said they will examine links between Mr. Trump’s associates and the Russians.

Senator Chris Coons, Democrat of Delaware, said on Friday that he believed there were “transcripts” that would help document those contacts, though he said he had not yet seen them.

Photo

Mr. Trump claimed the Obama administration ordered the phoned at his building in New York tapped.CreditVictor J. Blue for The New York Times

“There are transcripts that provide very helpful, very critical insights into whether or not Russian intelligence or senior Russian political leaders — including Vladimir Putin — were cooperating, were colluding, with the Trump campaign at the highest levels to influence the outcome of our election,” Mr. Coons told Andrea Mitchell on MSNBC. “I believe they exist.”

In a written statement on Saturday, a spokesman for Mr. Coons said that the senator “did not imply that he is aware of transcripts indicating collusion between the Trump campaign and the Russians.” The spokesman, Sean Coit, said Mr. Coons “simply stated that a full review of all relevant transcripts and intelligence intercepts is necessary to determine if collusion took place.”

The New York Times reported in January that among the associates whose links to Russia are being scrutinized are Paul Manafort, Mr. Trump’s onetime campaign chairman; Carter Page, a businessman and foreign policy adviser to the campaign; and Roger Stone, a longtime Republican operative who said he was in touch with WikiLeaks at one point before it released a trove of emails from John D. Podesta, Hillary Clinton’s campaign chairman, last August. Mr. Stone later said he had communicated with WikiLeaks through an intermediary.

Mr. Trump appeared on Saturday to suggest that warrants had been issued by the Foreign Intelligence Surveillance Court. He claimed that the Obama administration had once been “turned down by court” in its supposed efforts to listen in on conversations by Mr. Trump and his associates.

In the fall, the F.B.I. examined computer data showing an odd stream of activity between a Trump Organization server and Alfa Bank, one of Russia’s biggest banks, whose owners have longstanding ties to Mr. Putin. While some F.B.I. officials initially believed that the computer activity indicated an encrypted channel between Moscow and New York, the bureau ultimately moved away from that view. The activity remains unexplained.

There is no confirmed evidence that the F.B.I. obtained a court warrant to wiretap the Trump Organization or was capturing communications directly from the Trump Organization.

During the transition, the F.B.I. — which uses FISA warrants to eavesdrop on the communications of foreign leaders inside the United States — overheard conversations between the Russian ambassador to the United States and Michael T. Flynn, whom Mr. Trump had named national security adviser.

Mr. Trump has pointedly and repeatedly questioned in conversations how it was that Mr. Flynn’s conversations were recorded, and wondered who could have issued a warrant.

After The Washington Post reported that Mr. Flynn and the ambassador, Sergey I. Kislyak, had discussed sanctions that the Obama administration had just imposed on Russia, Mr. Flynn was pushed out of his post by the White House because he had lied to Vice President Mike Pence about the nature of the calls.

The Breitbart article cited mainstream news reports and concluded — going beyond the public record — that the Obama administration had “obtained authorization to eavesdrop on the Trump campaign; continued monitoring the Trump team even when no evidence of wrongdoing was found; then relaxed the N.S.A. rules to allow evidence to be shared widely within the government.”

Mr. Levin, a day earlier, railed about what he called a “much bigger scandal,” claiming — again with no evidence — that Mr. Obama and his aides had used “the instrumentalities of the federal government, intelligence activity, to surveil members of the Trump campaign and put that information out in the public.”

Several senior members of Mr. Trump’s White House staff, including his spokesman, Sean Spicer, did not respond to an email requesting on-the-record responses to more than a half-dozen questions about Mr. Trump’s Twitter posts.

Representative Adam B. Schiff, the top Democrat on the House Intelligence Committee, denounced the “willingness of the nation’s chief executive to make the most outlandish and destructive claims without providing a scintilla of evidence to support them.”

Even some Republican lawmakers questioned Mr. Trump’s accusations. Senator Ben Sasse of Nebraska issued a statement demanding that the president reveal everything he knows about any wiretaps or warrants.

“The president today made some very serious allegations, and the informed citizens that a republic requires deserve more information,” Mr. Sasse said, adding that “we are in the midst of a civilization-warping crisis of public trust.”

Taping calls seems to hold a spot in Mr. Trump’s consciousness. He spent many years taping his own phone calls as a businessman. During the campaign, Mr. Trump’s staff members told reporters they feared that their offices were being bugged.

The current president has frequently spoken about how much he admires Mr. Obama for the gracious way he handled the transition. But since taking office, Mr. Trump has frequently clashed with the intelligence agencies over the Russia inquiries, including efforts to examine the attempts by that country to influence the presidential election and the contacts between Mr. Trump’s aides and the Russian government.

In recent days, the president has appeared increasingly angry about leaks of information that he believes are coming from law enforcement and intelligence officials who are holdovers or recently departed from Mr. Obama’s administration.

People close to Mr. Trump have described him as determined to stop those people from sabotaging his administration. One adviser said on Friday that the president had been discussing a possible plan to try to prevent leaks from occurring. The adviser declined to elaborate on what the plan might entail.

Two senior administration officials said Mr. Trump had tried for two days to find a way to be on an offensive footing against the news articles resulting from leaks; one person close to Mr. Trump said his explosive claim was a result of that.

Mr. Trump’s mood was said to be volatile even before he departed for his weekend in Florida, with an episode in which he vented at his staff. The president’s ire was trained in particular on Mr. McGahn, his White House counsel, according to two people briefed on the matter.

Mr. Trump was said to be frustrated about the decision by Jeff Sessions, his attorney general, to recuse himself from participating in any investigations of connections between the Trump campaign and Russia. Mr. Trump has said there were no such connections. Mr. Trump, who did not learn that Mr. Sessions was recusing himself until after the decision was made, told aides that it gave an opening to his critics on the Russia issue.

Michael D. Shear reported from West Palm Beach, and Michael S. Schmidt from Washington. Matt Apuzzo and Charlie Savage contributed reporting from Washington, and Maggie Haberman from New York.

Trump Faces Furor Over Unsubstantiated Claim Obama Wiretapped Him

President adds meeting with attorney general and homeland security secretary as allies and foes react to his claims on Twitter

Trump Faces Furor Over Unsubstantiated Claim Obama Wiretapped Him

President adds meeting with attorney general and homeland security secretary as allies and foes react to his claims on Twitter

PALM BEACH, Fla.—President Donald Trump called together several top advisers Saturday, including Attorney General Jeff Sessions and political strategist Steve Bannon, as the White House faced a growing furor over the president’s early morning claims he had been wiretapped by his predecessor.

Mr. Trump, who is spending the weekend at his private Mar-a-Lago club here, added a meeting with Mr. Sessions and Homeland Security Secretary John Kelly to his calendar on Saturday, as political allies and foes of the president continued to react to the president’s claim on Twitter.

Mr. Trump tweeted early Saturday morning that former President Barack Obama had tapped his phones at Trump Tower, where Mr. Trump lived and worked in New York during the campaign, but provided no evidence for what amounted to an extraordinary claim of illegal activity by a former president.

The tweets used strong language and compared Mr. Obama to Joseph McCarthy and Richard Nixon. “Terrible! Just found out that Obama had my “wires tapped” in Trump Tower just before the victory,” Mr. Trump tweeted at 6:35 a.m. “Nothing found. This is McCarthyism!”

Jeff Sessions Used Political Funds for Republican Convention Expenses

Campaign-finance records show attorney general used campaign account, not official funds, for expenses to Cleveland, where he met Russia’s ambassador at an event.

CLICK TO READ STORY

Trump Jr. Was Likely Paid at Least $50,000 to Speak to Pro-Russia Group

President Donald Trump’s eldest son Donald Trump Jr. was likely paid at least $50,000 for an appearance late last year before a French think tank whose founder and wife are allies of the Russian government in efforts to end the war in Syria.

CLICK TO READ STORY

ADVERTISEMENT

In Short Supply in Trump’s Cabinet: Lawyers

President Trump’s preference for business and military leaders has marginalized a group long at the capital’s levers of power: lawyers. Just three of his 16 cabinet picks have law degrees, a sharp drop from the four previous administrations.

CLICK TO READ STORY

Russian Ambassador Kislyak Was Avid Networker in D.C.

Sergei Kislyak, the Russian official at the center of the furor around the Trump administration, was active in the Washington political circuit.

His postings also referred to Mr. Obama as a “bad (or sick) guy,” compared any such phone tapping to the Watergate scandal, and suggested that “a good lawyer could make a great case” of the matter.

Public officials, including some of Mr. Trump’s fellow Republicans, reacted sharply to the allegation. A spokesman for Mr. Obama said “neither President Obama nor any White House official ever ordered surveillance on any U.S. citizen. Any suggestion otherwise is simply false.”

The White House didn’t provide clarification on what Mr. Trump may have been referring to or what evidence he had for his claims, though a recent article on the Breitbart website made similar allegations about the Obama administration. The furor threatened to overshadow a crucial upcoming week for Mr. Trump, in which he is scheduled to release a revised executive order on immigration and Republican-led congressional committees are set to begin producing health-care legislation.

Mr. Trump had been scheduled to dine Saturday evening with Commerce Secretary Wilbur Ross at Mar-a-Lago. In an update, the president’s staff said the two also would be joined by Messrs. Sessions, Bannon and Kelly, as well as policy adviser Stephen Miller and the White House counsel, Don McGahn, turning the dinner into a session of the president’s inner circle.

The meeting came as some prominent Republicans joined Democrats in calling on the president to explain his claims. Sen. Ben Sasse (R., Neb.), a member of the Senate Judiciary Committee, said in a written statement Saturday that Mr. Trump should “explain what sort of wiretap it was and how he knows this.”

Mr. Sasse said the president’s statement suggests he may have been illegally tapped, or the courts may have approved a legal wiretap out of national security concerns, which would raise its own set of questions. “We are in the midst of a civilization-warping crisis of public trust, and the president’s allegations today demand the thorough and dispassionate attention of serious patriots,” Mr. Sasse said.

Earlier in the day, Sen. Lindsey Graham (R., S.C.), told an audience at Clemson University that Mr. Trump’s claim of having been illegally wiretapped, if true, would be “the biggest scandal since Watergate.” He added that he was also “very worried if in fact the Obama administration was able to obtain a warrant lawfully about Trump campaign activity with foreign governments.”

Rep. Justin Amash (R., Mich.), also called on Mr. Trump to share any evidence for his claim, saying on Twitter that the president “should provide more info to Congress immediately to assess constitutionality and legality.”

Messrs. Sasse, Graham and Amash have been among the Republicans most willing to publicly criticize Mr. Trump in the past. Still, few if any Republicans defended his comments Saturday.

Mr. Trump’s assertion followed several days of negative news coverage, as attention focused on interactions between Mr. Trump’s campaign advisers, family, and political allies with representatives of the Russian government. Saturday’s tweets marked a return to the caustic tone that has often characterized the president’s remarks, a tone he departed from during a well-received speech to a joint session of Congress earlier in the week.

Under the law, presidents can’t legally order wiretaps. In a national security probe, investigators seeking a wiretap must convince a judge there is probable cause that a target for surveillance is an agent of foreign intelligence, and that the main purpose of the surveillance is to obtain foreign intelligence information. Investigators sometimes face a higher bar if the target is an American citizen.

In a criminal probe, investigators must show probable cause that a crime has been committed for electronic surveillance to be approved.

The conservative media outlet Breitbart, whose former chairman, Mr. Bannon, is Mr. Trump’s political strategist, published an article Friday based on the claims of a right-wing radio host that intelligence agencies were conducting a “silent coup” against Mr. Trump.

The host, Mark Levin, and the Breitbart article assert the Obama administration received authorization to conduct surveillance on the Trump campaign. Mr. Levin said that intelligence-gathering on the Trump campaign and its surrogates—rather than the communications between Mr. Trump’s allies and Russian officials—should be investigated by Congress.

Democrats reacted angrily to what they called baseless claims by Mr. Trump, including the president’s description of his predecessor as “bad” or “sick.”

“The President offered no evidence to support this spectacularly reckless allegation,” said Rep. Adam Schiff (D., Calif.), the top Democrat on the House Intelligence Committee. “If there is something bad or sick going on, it is the willingness of the nation’s chief executive to make the most outlandish and destructive claims without providing a scintilla of evidence to support them.”

This isn’t the first time Mr. Trump has offered allegations of election-related misdeeds without evidence. He has also claimed that he lost the popular vote to Democrat Hillary Clinton because “millions” of votes were cast illegally in the 2016 presidential election, including those by illegal immigrants. There is no evidence for that, and election officials, including Republicans, say it didn’t happen.

Mr. Sessions said Thursday that he would recuse himself from investigations involving the Trump campaign. He added that his testimony was an accurate reflection of his understanding of the question posed at his hearing, since he had no ongoing contact with any Russian officials.

The Wall Street Journal has reported that intelligence officials were examining contacts involving Russian ambassador Sergei Kislyak, Mr. Sessions and others as part of a wide-ranging counterintelligence investigation.

Mr. Sessions was on the president’s mind Saturday morning: The president’s first tweet, at 6:26 a.m., asserted that Mr. Sessions’ first meeting with Mr. Kislyak was part of an “education program” arranged by the Obama administration for a large group of ambassadors. Mr. Sessions has said that his contact with Mr. Kislyak was in the context of his position as a senator from Alabama, and not in his capacity as an adviser to the Trump campaign, but he acknowledged that the two discussed Ukraine.

In a later tweet, Mr. Trump said that the Russian ambassador had visited the Obama White House 22 times, and four times last year. That number of visits by the ambassador for a major foreign power wouldn’t be unusual.

By 8:19 a.m., Mr. Trump had turned his sights to Arnold Schwarzenegger, the actor and former California governor who succeeded Mr. Trump as host of the reality TV show “The Apprentice,” and who has sparred with Mr. Trump over the president’s policies on immigration.

‘Unverified and potentially unverifiable’ reports published by BuzzFeed

Donald Trump and his inner circle ‘have received a regular flow of intelligence from the Kremlin, including on his Democratic and other political rivals’, a report dated June 2016 alleges. Photograph: Rex/Shutterstock

This article is 1 month old

Shares

Senator John McCain passed documents to the FBI director, James Comey, last month alleging secret contacts between the Trump campaign and Moscow and that Russian intelligence had personally compromising material on the president-elect himself.

The Guardian has not been able to confirm the veracity of the documents’ contents, and the Trump team has consistently denied any hidden contacts with the Russian government.

A spokesman for the Russian president, Vladimir Putin, on Wednesday denied Russia has collected compromising information on Trump and dismissed news reports as a “complete fabrication and utter nonsense”. Dmitry Peskov insisted that the Kremlin “does not engage in collecting compromising material”.

Trump’s transition team did not immediately respond to a request for comment, but late on Tuesday, Trump tweeted: “FAKE NEWS – A TOTAL POLITICAL WITCH HUNT!” He made no direct reference to the allegations.

An official in the US administration who spoke to the Guardian described the source who wrote the intelligence report as consistently reliable, meticulous and well-informed, with a reputation for having extensive Russian contacts.

Some of the reports – which are dated from 20 June to 20 October last year – also proved to be prescient, predicting events that happened after they were sent.

One report, dated June 2016, claims that the Kremlin has been cultivating, supporting and assisting Trump for at least five years, with the aim of encouraging “splits and divisions in western alliance”.

It claims that Trump had declined “various sweetener real estate deals offered him in Russia” especially in developments linked to the 2018 World Cup finals but that “he and his inner circle have accepted a regular flow of intelligence from the Kremlin, including on his Democratic and other political rivals.”

Most explosively, the report alleges: “FSB has compromised Trump through his activities in Moscow sufficiently to be able to blackmail him.” The president-elect has not responded to the allegations.

CNN reported on Tuesday that the FBI was still investigating the credibility of the documents but added that the intelligence chiefs had included a summary of the material in a secret briefing on Russian interference in the election delivered last week to Barack Obama and Donald Trump.

The emergence of the documents is potentially explosive, 10 days before Trump’s inauguration and on the eve of his first planned press conference since July last year.

Despite glowing references from US and foreign officials who have worked with the source, there are some errors in the reports. One describes the Moscow suburb of Barvikha as “reserved for the residences of the top leadership and their close associates”, but although it is a very expensive neighbourhood, there are no restrictions on who can own property there. The document also misspells the name of a Russian banking corporation.

The FBI does not normally make any comment on ongoing counter-intelligence investigations but was under increasing pressure from Democrats and some Republicans to act before the inauguration, particularly because of Comey’s announcement of a continuing investigation into Hillary Clinton’s email server 11 days before the election, which many of her supporters believe cost her the presidency.

The reports were initially commissioned as opposition research during the presidential campaign, but its author was sufficiently alarmed by what he discovered to send a copy to the FBI. It is unclear who within the organisation they reached and what action the bureau took. The former Democratic Senate leader, Harry Reid, has lambasted Comey for publicising investigations into Hillary Clinton’s private server, while allegedly sitting on “explosive” material on Trump’s ties to Russia.

Another Democratic senator, Ron Wyden, questioned Comey insistently at a Senate intelligence committee hearing on Tuesday on whether the FBI was pursuing leads on Trump campaign contacts with Russia.

“Has the FBI investigated these reported relationships?” Wyden asked.

Comey replied: “I would never comment on investigations … in a public forum.

The Guardian can confirm that the documents reached the top of the FBI by December. Senator John McCain, who was informed about the existence of the documents separately by an intermediary from a western allied state, dispatched an emissary overseas to meet the source and then decided to present the material to Comey in a one-on-one meeting on 9 December, according to a source aware of the meeting. The documents, which were first reported on last year by Mother Jones, are also in the hands of officials in the White House.

McCain is not thought to have made a judgment on the reliability of the documents but was sufficiently impressed by the source’s credentials to feel obliged to pass them to the FBI.

The Senate armed services committee, which Senator McCain chairs, launched an inquiry last week into Russian cyber-attacks during the election.

McCain was reluctant to get involved, according to a colleague, for fear the issue would be dismissed as a personal grudge against Trump. He pushed instead for the creation of a special Senate committee to look into connections between campaign staff and Moscow, but the proposal was blocked by the Republican leadership.

McCain told the NBC programme Meet the Press on Sunday: “I would like to see a select committee. Apparently that is not in agreement by our leadership. So we will move forward with the armed services committee and I’m sure foreign relations and intelligence committee will as well.”

But the senator added: “It is possible if enough information comes out, that that decision could be reversed. I still think it’s the best way to attack the issue.”

Asked on the same programme on whether an investigation was ongoing into campaign links to Moscow, Senator Lindsey Graham, another conservative Republican said: “I believe that it’s happening.”

According to the report passed to Comey, Russian intelligence allegedly gathered compromising material during Trump’s stay in Moscow in November 2013, when he was in the city to host the Miss Universe pageant.

Another report, dated 19 July last year said that Carter Page, a businessman named by Trump as one of his foreign policy advisers, had held a secret meeting that month with Igor Sechin, head of the Rosneft state-owned oil company and a long-serving lieutenant of Vladimir Putin. Page also allegedly met Igor Divyekin, an internal affairs official with a background in intelligence, who is said to have warned Page that Moscow had “kompromat” (compromising material) on Trump.

Two months later, allegations of Page’s meetings surfaced in the US media, attributed to intelligence sources, along with reports that he had been under FBI scrutiny.

The Guardian has learned that the FBI applied for a warrant from the foreign intelligence surveillance (Fisa) court over the summer in order to monitor four members of the Trump team suspected of irregular contacts with Russian officials. The Fisa court turned down the application asking FBI counter-intelligence investigators to narrow its focus. According to one report, the FBI was finally granted a warrant in October, but that has not been confirmed, and it is not clear whether any warrant led to a full investigation.

A month after Trump’s surprise election victory, Page was back in Moscow saying he was meeting with “business leaders and thought leaders”, dismissing the FBI investigation as a “witch-hunt” and suggesting the Russian hacking of the Democratic Party alleged by US intelligence agencies, could be a false flag operation to incriminate Moscow.

Another of the reports compiled by the former western counter-intelligence official in July said that members of Trump’s team, which was led by campaign manager Paul Manafort (a former consultant for pro-Russian politicians in Ukraine), had knowledge of the DNC hacking operation, and in return “had agreed to sideline Russian intervention in Ukraine as a campaign issue and to raise US/Nato defence commitments in the Baltics and Eastern Europe to deflect attention away from Ukraine”.

A few days later, Trump raised the possibility that his administration might recognise Russia’s annexation of Crimea and openly called on Moscow to hack Hillary Clinton’s emails.

In August, officials from the Trump campaign intervened in the drafting of the Republican party platform, specifically to remove a call for lethal assistance to Ukraine for its battle against Moscow-backed eastern rebels.

Manafort stepped down in August as campaign manager and the campaign steadily distanced itself from Page. However, Trump’s praise of Putin and defence of Moscow’s actions in Ukraine and Syria remained one of the few constants in his campaign talking points.

Manafort has denied secret links with Moscow calling the allegation “an outrageous smear being driven by Harry Reid and the Clinton campaign”.

Since then, Trump has consistently cast doubt on Russian culpability for hacking the Democratic National Committee, defying a consensus of 17 national intelligence agencies. After Obama deported 35 Russian diplomats in retaliation for Moscow’s intervention, Trump praised Putin for not carrying out tit-for-tat deportations of US diplomats. “I always knew he was very smart,” he tweeted.

NSA Whistleblower: Everyone in US under virtual surveillance, all info stored, no matter the post

Rand Paul Causes A Vicious Senate Cat Fight Over Patriot Act

Rand Paul’s Freedom Act Filibuster

The Senate has approved the USA Freedom Act, which will alter the way U.S. agencies conduct surveillance and gather data. A final vote on the bill came late Tuesday afternoon, after amendments to the bill failed.

Update at 9:30 p.m. ET: Obama’s Signature

Following an expedited enrollment process, President Obama signed the bill into law late Tuesday.

Enactment of this legislation will strengthen civil liberty safeguards and provide greater public confidence in these programs,” Obama said in a statement after the vote. “I am gratified that Congress has finally moved forward with this sensible reform legislation.”

Update at 4:30 p.m. ET: The Bill Has Passed

In the final tally of the vote, 67 senators were in favor of the measure and 32 against. The legislation needed a simple majority to pass.

Last November, the Freedom Act failed in the Senate after not receiving enough support to avoid a filibuster. Its critics say the act doesn’t go far enough to curtail surveillance programs that can access huge databases of information about Americans.

Sen. Rand Paul, R-Ky., voted against the measure today, as he did last fall. Also voting against the bill Tuesday was independent Sen. Bernie Sanders, who is seeking the Democratic presidential nomination.

The lead sponsor of the bill in the House, Rep. Jim Sensenbrenner, R-Wis., promises it will “rein in the dragnet collection of data” by the NSA and others, and “increase transparency of the Foreign Intelligence Surveillance Court.”

Calling today’s passage “a milestone,” ACLU Deputy Legal Director Jameel Jaffer says, “This is the most important surveillance reform bill since 1978, and its passage is an indication that Americans are no longer willing to give the intelligence agencies a blank check.”

Our original post continues:

The vote comes two days after controversial provisions of the Patriot Act expired because the Senate was unable to “overcome parliamentary maneuvers by Sen. Rand Paul,” as Eyder reported Sunday night, “and let three controversial provisions of the Patriot Act expire at midnight.”

The House of Representatives approved the Freedom Act on May 13. The legislation would remain in effect until Dec. 15, 2019.

“We worked for two years across the aisle and across the Capitol,” said Sen. Patrick Leahy, a champion of the bill. He said it would bring much-needed reform to America’s intelligence-gathering.

Leahy and his chief ally on the bill, Sen. Mike Lee, R-Utah, spent their Tuesday in the Senate fighting against amendments to the USA Freedom Act that were put forth — and defeated.

Tuesday’s vote on the Freedom Act comes less than a month after a federal appeals court ruled that the National Security Agency’s practice of collecting bulk data about Americans’ phone calls violates the Constitution.

Before the vote, a displeased Majority Leader Sen. Mitch McConnell criticized the policies of President Obama and said that the Freedom Act weakens America’s ability to protect itself. He also cited an AP article that called the turn of events in the surveillance and spying field as “a victory for Edward Snowden,” the former NSA contractor who released secret information about U.S. spying in June of 2013 (Snowden discussed the Patriot Act Tuesday).

After McConnell spoke, Minority Leader Sen. Harry Reid responded by saying that if McConnell is worried about making America look weaker, “he should look in the mirror.”

Reid accused the majority leader of trying to deploy distractions from the real issues and said that McConnell had also implicitly criticized the House of Representatives.

With or Without the Patriot Act, Here’s How the NSA Can Still Spy on Americans

June 1, 2015,Jason M. Breslow

While it may only be temporary, the National Security Agency on Monday lost its authority to collect Americans’ phone records in bulk after the Senate failed to extend provisions of the Patriot Act authorizing the controversial domestic surveillance program.

For now, the stall in the Senate means the NSA can’t collect any newly created telephone records. Under the now-lapsed Section 215 of the Patriot Act, the NSA gathered metadata such as who called whom, the time the call was placed and how long the conversation lasted.

Also lapsed are provisions of the law that allowed for wiretap orders on “lone wolf” terrorism suspects; that permitted roving wiretaps that follow suspects from device to device as they change phones; and that compelled businesses to turn over records deemed relevant to a national security investigation.

Under an entirely separate law, the 2008 FISA Amendments Act, the government still has the authority to access the communications of users of popular Internet sites such as Facebook, Google, Microsoft and Yahoo. Section 702 of the law, which does not expire until 2017, gives the government the ability to collect the content of an Internet user’s actual communications — not just metadata.

The law is geared towards non-citizens outside of the U.S., but as privacy advocates argue, it is inevitable that the communications of U.S. citizens and those of non-citizens lawfully living in the U.S. are swept up by the program.

“The phone records program under Section 215 is really just one piece of a much larger puzzle,” said Stephen Vladeck, a professor of law at the American University Washington College of Law. “They’re targeted at non-citizens but the way the technology works there is just no way for the vacuum cleaner to distinguish between the particles of dirt.”

An even older and more obscure Regan-era law, Executive Order No. 12333, provides U.S. intelligence with nearly identical surveillance capabilities to intercept overseas communications, Vladeck said, with the same implications for privacy.

“The way the government is intercepting communications under these authorities,” said Vladeck, referring to Section 702 and Executive Order 123333, “it cannot tell at the point of collection whether the actual sender or recipient is or is not a U.S. citizen.”

Also unaffected by the sunset of Section 215 is the use of National Security Letters, which since 9/11 have helped to dramatically expand the government’s ability to collect information about Americans directly from phone companies and Internet providers. Any FBI office can issue one, without a court’s review and with a gag order. In the past 10 years, more than 300,000 National Security Letters have been issued, according to the Electronic Frontier Foundation, and until 2013, no major Internet or phone company is known to have questioned the constitutionality of one.

Meanwhile, it’s not clear that all surveillance conducted under the Patriot Act has officially come to a close. As The New York Times noted, all three aspects of the law that expired Monday “contained a so-called grandfather clause that permits their authority to continue indefinitely for any investigation that had begun before June 1.”

Of course, by the end of the week, that may not matter. After having failed to extend the expiring Patriot Act provisions on Sunday, the Senate appears poised to pass a House bill, the USA Freedom Act, that would restore the lapsed Patriot Act powers into law. The one critical difference in the new law is that bulk phone records would stay in the hands of phone companies, rather than with the government.

In this two-part, Peabody Award-winning series, FRONTLINE explores how the U.S. government came to monitor and collect the communications of millions of people around the world — and here at home — and the lengths to which officials tried hide the massive surveillance from the public.

USA Freedom Act

To reform the authorities of the Federal Government to require the production of certain business records, conduct electronic surveillance, use pen registers and trap and trace devices, and use other forms of information gathering for foreign intelligence, counterterrorism, and criminal purposes, and for other purposes.

Acronym

USA Freedom Act, a backronym for “Uniting and Strengthening America by FulfillingRights and Ending Eavesdropping,Dragnet-collection and Online Monitoring Act“

The USA Freedom Act is a law which was originally introduced in both houses of the U.S. Congress on October 29, 2013. Following the expiration of several provisions of the Patriot Act, the act was passed on June 2, 2015.[3][4] The title of the act is a ten-letter backronym (USA FREEDOM) that stands for “Uniting and Strengthening America by Fulfilling Rights and Ending Eavesdropping, Dragnet-collection and OnlineMonitoring Act.”

When the bill was re-introduced in the 114th Congress (2015-2016), it was described by the bill sponsors as “a balanced approach”[5] while being questioned for extending the Patriot Act through the end of 2019.[6]Supporters of the bill said that the House Intelligence Committee and House leadership[7] would insist on reauthorizing all Patriot Act powers except bulk collection under Section 215 of the Patriot Act[8]. Critics assert that mass surveillance of the content of Americans’ communication will continue under Section 702 of FISA which does not expire until 2017[9][10] and Executive Order 12333[9][11] due to the “unstoppable surveillance-industrial complex”[12] despite the fact that a bipartisan majority of the House had previously voted to close backdoor mass surveillance.[7]

Purpose

According to supporters of the USA Freedom Act, the USA Freedom Act[13][full citation needed] was meant to end the bulk collection of Americans’ metadata by the NSA, end the secret laws created by the FISA court, and introduce a “Special Advocate” to represent public and privacy matters.[14][15][16] However, the USA Freedom Act does allow the bulk collection of Americans’ metadata by phone companies, which is then accessible by the NSA; it also does not address other laws which have purportedly challenged Americans’Fourth Amendment rights.[17] Other proposed changes included limits to programs like PRISM, which retains Americans’ Internet data,[18] and greater transparency by allowing companies such as Google andFacebook to disclose information about government requests for information.[19]

Purpose

Representative Jim Sensenbrenner, who introduced the bill, stated that its purpose was:

To rein in the dragnet collection of data by the National Security Agency (NSA) and other government agencies, increase transparency of the Foreign Intelligence Surveillance Court (FISC), provide businesses the ability to release information regarding FISA requests, and create an independent constitutional advocate to argue cases before the FISC.[14][28]

According to the bill’s sponsors, their legislation would have amended Section 215 of the Patriot Act to ensure that any phone records obtained by the government were essential in an investigation that involved terrorism or espionage, thereby ending bulk collection,[8] while preserving “the intelligence community’s ability to gather information in a more focused way.”[29]

Background

Many members of Congress believed that in the wake of the Snowden disclosures, restoration of public trust would require legislative changes.[30] More than 20 bills have been written since the disclosures began with the goal of clarifying government surveillance powers.[18]

Sensenbrenner, who introduced the USA PATRIOT Act (H.R. 3162) in 2001 following the September 11 terrorist attacks to give more power to US intelligence agencies, and who has described himself as “author of the Patriot Act,”[31] declared that it was time to put the NSA’s “metadata program out of business.” With its bulk collection of Americans’ phone data, Sensenbrenner asserted that the intelligence community “misused those powers,” had gone “far beyond” the original intent of the legislation, and had “overstepped its authority.”[30][32]

An opinion piece by Leahy and Sensenbrenner, published in Politico, described the impetus for proposed changes,[33]saying:

The intelligence community has failed to justify its expansive use of [the FISA and Patriot Act] laws. It is simply not accurate to say that the bulk collection of phone records has prevented dozens of terrorist plots. The most senior NSA officials have acknowledged as much in congressional testimony. We also know that the FISA court has admonished the government for making a series of substantial misrepresentations to the court regarding these programs. As a result, the intelligence community now faces a trust deficit with the American public that compromises its ability to do its job. It is not enough to just make minor tweaks around the edges. It is time for real, substantive reform.[29]

Markup in House Judiciary Committee

In May 2014, the U.S. House Judiciary Committee posted a “Manager’s Amendment” on its website. Title VII of the Amendment read “Section 102(b)(1) of the USA Patriot Improvement and Reauthorization Act of 2005 (50 U.S.C. 1805 note) is amended by striking “June 1, 2015” and inserting “December 31, 2017,” extending the controversial USA PATRIOT Act through the end of 2017.[34] The Electronic Privacy Information Center (EPIC) has criticized the Patriot Act as unconstitutional, especially when “the private communications of law-abiding American citizens might be intercepted incidentally”.[35] James Dempsey, of the CDT, believes that the Patriot Act unnecessarily overlooks the importance of notice under the Fourth Amendment and under a Title III wiretap,[36] while the American Library Associationbecame so concerned that they formed a resolution condemning the USA PATRIOT Act, and which urged members to defend free speech and protect patrons’ privacy against the Act.[37]

The Guardian wrote “civil libertarians on the Judiciary Committee had to compromise in order to gain support for the act. Significantly, the government will still be able to collect phone data on Americans, pending a judge’s individualized order based on ‘reasonable articulable suspicion‘ – a standard preferred by the NSA – of wrongdoing, and can collect call records two degrees or ‘hops’ of separation from the individual suspected.”[23] Kara Brandeisky of ProPublica said, “some worry that the bill does not unequivocally ban bulk collection of American records. Again, a lot depends on how the Foreign Intelligence Surveillance Court interprets the statute.”[38]

The Electronic Frontier Foundation (EFF) stated it remained “concerned that this bill omits important transparency provisions found in the (original 2013) USA FREEDOM Act, which are necessary to shed light on surveillance abuses”. In addition, the EFF said it believed “this bill should do more to address mass surveillance under Section 702 of Foreign Intelligence Surveillance Amendments Act, a section of law used to collect the communications of users worldwide.”[40]The Open Technology Institute commented “several other key reforms—such as provisions allowing Internet and phone companies to publish more information about the demands they receive, which OTI and a coalition of companies and organizations have been pressing for since last summer—have been removed, while the bill also provides for a new type of court order that the President has requested, allowing for continuous collection by the government of specified telephone records.”[41]

Despite the criticism from civil liberties groups, Mike Rogers, a defender of the NSA‘s surveillance practices and the chairman of the House Intelligence Committee, praised the amendments. Rogers, who had his own bill which would codify the NSA’s surveillance practices in to law, called the proposed amendments a “huge improvement.” Foreign Policy wrote “any compromise to the Judiciary bill risks an insurrection from civil libertarians in Congress. Michigan Republican Justin Amashled such a revolt last year when he offered an NSA amendment to a defense appropriations bill that would have stripped funding for the NSA’s collection program.” “Just a weakened bill or worse than status quo? I’ll find out,” Representative Amash said.[42]

After the marked up bill passed the House Judiciary Committee USA Freedom Act co-author and Senate Committee on the Judiciary Chairman Patrick Leahycommented that he “remain concerned that the legislation approved today does not include some of the important reforms related to national security letters, a strong special advocate at the FISA Court, and greater transparency. I will continue to push for those reforms when the Senate Judiciary Committee considers the USA Freedom Act this summer.”[25]

Passage in House of Representatives

The House of Representatives passed on May 22, 2014 the USA Freedom act by 303 votes to 121.[43] Because the House version was weakened by lawmakers loyal to the intelligence establishment it lost support of important House Judiciary members like Republicans Darrell Issa, Ted Poe and Raul Labrador and Democrat Zoe Lofgren who previously voted for the act.[44] “The result is a bill that will actually not end bulk collection, regrettably,” said Rep. Zoe Lofgren who voted against the bill.[45] The act would shift responsibility for retaining telephonic metadata from the government to telephone companies. Providers like AT&T and Verizon would be required to maintain the records and let the NSA search them in terrorism investigations when the agency obtains a judicial order or in certain emergency situations.[46] The USA Freedom Act demands that the NSA get approval for a search from the Foreign Intelligence Surveillance Court before demanding that the telecoms hand over metadata. However, no “probable-cause” Fourth Amendment standard is required to access the database[45] While an allowable search under the original USA Freedom Act was defined as “a term used to uniquely describe a person, entity, or account”, but under the House version a database search inquiry is now allowed if it is “a discrete term, such as a term specifically identifying a person, entity, account, address, or device.”[45] Provisions that were dropped from the bill included requirements to estimate the number of Americans whose records were captured under the program, and the creation of a public advocate to challenge the government’s legal arguments before the Foreign Intelligence Surveillance Court.[47][48]

The passed House version[49] was criticised by U.S. senators, tech firms like Google, Apple, Microsoft, Facebook and Twitter, as well as civil liberties groups.[44][45][46][47][50] Senator Sen. Patrick Leahy, chair of the Senate Judiciary Committee and lead Democratic author of the Freedom Act, criticized the House version by saying in a statement: “Today’s action in the House continues the bipartisan effort to restore Americans’ civil liberties. But I was disappointed that the legislation passed today does not include some of the meaningful reforms contained in the original USA Freedom Act. I will continue to push for these important reforms when the Senate judiciary committee considers the USA Freedom Act next month.”[50] And Senator Ron Wyden stated he was “gravely concerned that the changes that have been made to the House version of this bill have watered it down so far that it fails to protect Americans from suspicionless mass surveillance.”[50]Major U.S. tech firms like Google, Apple, Microsoft, Facebook, and Twitter joined together in the Reform Government Surveillance coalition which called the House version a move in the wrong direction. The Reform Government Surveillance released a statement on June 5, stating: “The latest draft opens up an unacceptable loophole that could enable the bulk collection of Internet users’ data … While it makes important progress, we cannot support this bill as currently drafted and urge Congress to close this loophole to ensure meaningful reform.”[51] Mark Jaycox, a legislative analyst with the Electronic Frontier Foundation, said: “The bill is littered with loopholes. The problem right now, especially after multiple revisions, is that it doesn’t effectively end mass surveillance.”[44][44] Zeke Johnson, director ofAmnesty International USA’s security and human rights program, accusing the House for failing to deliver serious surveillance reform said: “People inside and outside the U.S. would remain at risk of dragnet surveillance. The Senate should pass much stronger reforms ensuring greater transparency, robust judicial review, equal rights for non-U.S. persons, and a clear, unambiguous ban on mass spying. President Obama need not wait. He can and should implement such safeguards today.” The White House however endorsed the bill. “The Administration strongly supports House passage of H.R. 3361, the USA Freedom Act. … The Administration applauds and appreciates the strong bipartisan effort that led to the formulation of this bill, which heeds the President’s call on this important issue,” the White House said in a statement.[51] “The bill ensures our intelligence and law enforcement professionals have the authorities they need to protect the Nation, while further ensuring that individuals’ privacy is appropriately protected when these authorities are employed. Among other provisions, the bill prohibits bulk collection through the use of Section 215, FISA pen registers, and National Security Letters.”[46][52]

Civil rights groups and scholars said the new language allowing the NSA to search meta data handed over from telephone companies was vague and perhaps would allow the NSA to ensnare the metadata of broad swaths of innocent people in violation of their constitutional rights. “In particular, while the previous bill would have required any request for records to be tied to a clearly defined set of ‘specific selection terms,’ the bill that just passed leaves the definition of ‘specific selection terms’ open. This could allow for an overly broad and creative interpretation, which is something we’ve certainly seen from the executive branch and the FISA Court before,” said Elizabeth Goitein, a co-director of the Brennan Center’s Liberty and National Security Program.[45] “The new definition is incredibly more expansive than previous definitions … The new version not only adds the undefined words “address” and “device,” but makes the list of potential selection terms open-ended by using the term “such as.” Congress has been clear that it wishes to end bulk collection, but given the government’s history of twisted legal interpretations, this language can’t be relied on to protect our freedoms,” said the Electronic Frontier Foundation in a press release.[51][53]

Defeat in the Senate

Negotiations among intelligence agencies, the White House, lawmakers and their aides, and privacy advocates in the summer of 2014 led to a modified bill (S. S.2685)[54] in the U.S. Senate. This bill version addressed most privacy concerns regarding the NSA program that collects records of Americans’ phone calls in bulk and other issues.[55]

Under the bill the NSA would no longer collect those phone records. Instead, most of the records would have stayed in the hands of the phone companies, which would not have been required to hold them any longer than they already do for normal business purposes, which in some cases is 18 months. The bill would require the NSA to request specific data from phone companies under specified limits i.e. the NSA would need to show it had reasonable, articulable suspicion that the number it is interested in is tied to a foreign terrorist organization or individual. The proposed legislation would still have allowed analysts to perform so-called contact chaining in which they trace a suspect’s network of acquaintances, but they would been required to use a new kind of court order to swiftly obtain only those records that were linked, up to two layers away, to a suspect — even when held by different phone companies. It would also require the federal surveillance court to appoint a panel of public advocates to advance legal positions in support of privacy and civil liberties, and would expand company reporting to the public on the scope of government requests for customers’ data. This USA Freedom Act version thus gained the support of the Obama Administration, including the director of national intelligence and attorney general, as well as many tech companies including Apple, Google, Microsoft and Yahoo as well as a diverse range of groups, including the National Rifle Association and the American Civil Liberties Union.[26][55]

Following the 2014 Congressional elections, the Senate voted on November 18, 2014, to block further debate of the measure during the 113th United States Congress. Fifty-four Democrats and four Republicans who supported consideration failed to muster the 60 votes required.[56] Senator Patrick Leahy, who drafted the bill, blamed its defeat on what he called fear-mongering by opponents, saying, “Fomenting fear stifles serious debate and constructive solutions.” Senator Mitch McConnell, the Republican leader, argued that the NSA’s bulk collection of Americans’ metadata was a vital tool in the fight against terrorism. “This is the worst possible time to be tying our hands behind our backs,” he said.[26]

114th Congress (2015-2016)

The USA Freedom Act was re-introduced in the House Judiciary Committee and Senate Judiciary Committee in late April 2015 based upon a modified version of the one which failed in the Senate in the 113th Congress.

The 2015 USA Freedom Act[57] version is described by its sponsors as “a balanced approach that would ensure the NSA maintains an ability to obtain the data it needs to detect terrorist plots without infringing on Americans’ right to privacy.”[5] Human rights groups believed the bill’s transparency and court oversight provisions are less robust than would have been required in a previous version of the bill, with more limited reporting requirements and a more narrowly defined role for external court advocates.[58]

The bill received a mixture of reaction, ranging from support from national security and computer trade groups, skepticism or moderate objection from civil liberties groups, to outright opposition from former NSA whistle blowers. The editorial board of the New York Times ran an editorial against the bill which “will be weakened further in the Senate by the majority leader” and advised readers to “get used to the protections of your civil liberties being minimally viable”.[59]

Passage out of House Judiciary Committee

The bill passed out of the House Judiciary Committee on April 30, 2015.[60] The proposed bill would end the NSA’s bulk collection under Section 215 by requiring the government to seek records from companies using a “specific selection term” that identifies a specific person, account or address and “is used to limit . . . the scope” of records sought. The term may not be a phone or Internet company.[5]

Amendments to strengthen the bill were voted down during Committee markup. One would have offered a constitutional advocate and failed by voice vote,[61] while another would have offered protection for whistle blower complaints.[62] Representative Jordan unsuccessfully argued for another amendment with the following dialog, “It’s not a vote to blow up the deal. It’s a vote for the Fourth Amendment. Plain and simple. All the Gentleman says in his amendment is, if you’re going to get information from an American citizen, you need a warrant.”[63] The bill ultimately received 25 votes in support (64%), 12 abstentions (31%), and 2 in opposition (5%).[64]

House Judiciary Chairman Bob Goodlatte said “the USA Freedom Act reforms our nation’s intelligence-gathering programs to ensure they operate in a manner that reflects core American values … We urge both the House and Senate to move expeditiously on this legislation so that we rein in government overreach and rebuild trust with the American people”.[65]

Representative Ted Poe was one member to vote against the bill. “Between the Committee vote and the House floor the bill was changed and it now confused what should have been clarified. The version of the USA Freedom Act that passed the House today leaves room for different interpretations, potentially giving NSA the ability to continue to act outside the intent of Congress and the Constitution. I could not support a bill that may allow abuses of the fourth amendment to continue,” he said.[66]

Reaction

National security and trade groups

The Center for National Security supports the USA Freedom Act introduced on April 28, 2015 to end bulk collection of Americans’ telephone metadata under the so-called “section 215” program.[67]

The Software Alliance sponsored the legislation saying “in reforming government surveillance practices, it is critical that legislation strikes the right balance between securing our nation and its citizens and improving privacy protections for the public. The FISA reforms in the USA FREEDOM Act will help restore trust in both the US government and the US technology sector.”[68]

The ITIC said “the USA Freedom Act, H.R. 2048, builds on the foundation laid by the House Judiciary Committee last Congress and the result is a bill that strengthens privacy protections while maintaining the interests of national security.”[69]

Civil liberties advocates

The final USA Freedom Act is perceived as containing several concessions to pro-surveillance legislators meant to facilitate its passage.[6][70] The watered down version of the USA Freedom Act that passed the House of Representatives in 2015 has been widely criticized by civil liberties advocates and its original supporters amongst house members for extending the Patriot Act Mass surveillance programs without meaningful restraints, undermining the original purpose of the bill. [71]

“This bill would make only incremental improvements, and at least one provision-the material-support provision-would represent a significant step backwards,” ACLU deputy legal director Jameel Jaffer said in a statement. “The disclosures of the last two years make clear that we need wholesale reform.” Jaffer wants Congress to let Section 215 sunset completely and wait for a better reform package than endorse something half-baked[72], saying that “unless that bill is strengthened, sunset would be the better course.”[73] The ACLU had previously written of the 2013 version that “although the USA Freedom Act does not fix every problem with the government’s surveillance authorities and programs, it is an important first step and it deserves broad support.”[74][75]

Representative Justin Amash, author of the narrowly defeated Amash Amendment, a proposal that would have de-funded the NSA bulk-collection program, backed the 2013 legislation, but not the final 2015 version[76]. “It’s getting out of control,” he commented. “[Courts are issuing] general warrants without specific cause…and you have one agency that’s essentially having superpowers to pass information onto others”.[32]

According to Deputy Attorney General James Cole, even if the Freedom Act becomes law, the NSA could continue its bulk collection of American’s phone records. He explained that “it’s going to depend on how the [FISA] court interprets any number of the provisions” contained within the legislation.[8]Jennifer Granick, Director of Civil Liberties at Stanford Law School, stated:

International human rights groups remain somewhat skeptical of specific provisions of the bill. For example, Human Rights Watch expressed its concern that the “bill would do little to increase protections for the right to privacy for people outside the United States, a key problem that plagues U.S. surveillance activities. Nor would the bill address mass surveillance or bulk collection practices that may be occurring under other laws or regulations, such as Section 702 of the FISA Amendments Act or Executive Order 12333. These practices affect many more people and include the collection of the actual content of internet communications and phone calls, not just metadata”.[77] Zeke Johnson, Director of Amnesty International‘s Security and Human Rights Program, agreed that “any proposal that fails to ban mass surveillance, end blanket secrecy, or stop discrimination against people outside the U.S. will be a false fix”.[11]

Members of the anti-surveillance Civil Liberties Coalition are dismissing the USA Freedom Act in support of the Surveillance State Repeal Act, a far more comprehensive piece of legislation in the House that completely repeals the Patriot Act, as well as 2008’s FISA Amendments Act.[72] A group of 60 organizations called Congress to not stop at ending the NSA’s bulk collection of telephone information under the USA PATRIOT Act, but to also end the FISA Amendments Act and Executive Order 12333 mass surveillance programs and restore accountability for bad actors in the Intelligence Community.[9]

The Center for Democracy and Technology endorses the bill, but it points out that it doesn’t limit data retention for information collected on people who turn out to have no connection to a suspect or target, and emphasizes that this is not an omnibus solution.[72] The group argued the bill had to be supported because “the Senate will weaken the USA FREEDOM Act right before the sunset deadline, forcing the House to accept a weaker bill”.[78]

“This bill purports to ban certain acts under narrow authorities, but it doesn’t ban those behaviors outright. Nor does it increase meaningful oversight of the NSA,” said David Segal, executive director of Demand Progress, who wants Section 215 to expire. The group said “a vote for a bill that does not end mass surveillance is a vote in support of mass surveillance.”[79]

“Companies are provided monetary incentive to spy and share that information with the government and blanket liability once they do under USA Freedom — even if that breaks that law,” said Sascha Meinrath, the director of X-lab, an independent tech policy institute previously associated with New America. “Once companies receive that, they’ll have almost no reason to weigh in on meaningful surveillance reform.”[79] “In a way, it’s kind of like PRISM,” the program revealed by Snowden where major tech companies turned over the content of online communications to the NSA, said longtime independent surveillance researcher Marcy Wheeler. “It pushes things to providers: Everyone gets immunity, but it doesn’t add to the privacy.”[79]

“We think of the USA Freedom Act as yesterday’s news,” said Shahid Buttar of the Bill of Rights Defense Committee, “and we’re interested in forcing the [intelligence] agencies into a future where they comply with constitutional limits.” “If passed, it’ll be the only step,” predicted Patrick Eddington of the Cato Institute, a former House staffer, since the next expiration date for a major piece of surveillance legislation is 31 December 2017.[80]

Following the law’s passage on June 2, 2015, ACLU deputy legal director Jameel Jaffer claimed that “This is the most important surveillance reform bill since 1978, and its passage is an indication that Americans are no longer willing to give the intelligence agencies a blank check.”[81]

Former whistleblowers

Former NSA crypto-mathematician William Binney, who worked three decades at the agency, says the Freedom Act – widely seen as having the best chance of any surveillance-limiting proposal – “won’t do anything” if it passes. “Why do you think NSA [and other intelligence agencies] support it?” he says.[12][12]

Drake, a former NSA senior executive prosecuted unsuccessfully under the Espionage Act before pleading guilty to a misdemeanor in 2011, calls the bill the “Free-dumb Act 2.0,” and says he sees it as a ploy by government officials “to keep the status quo in place.” He also says the fixation on the call record program in public debate is unfortunate, because NSA Internet surveillance is far broader and more invasive. “It’s a shiny, shiny bright spot, [but] there’s a whole lot more being collected,” he says, including a “staggering” amount of American communications. Drake believes support from the Obama administration for the Freedom Act is motivated in part by a desire to hobble lawsuits against the call record program, three of which are pending with appeals courts and may lay the groundwork for a major Supreme Court privacy ruling.[12]

Wiebe, formerly a senior analyst at the NSA, says the anticipated Freedom Act likely will be “more of the same” and is “not going to change anything” in a meaningful way. Like Drake, he has no hope for meaningful reform and doesn’t believe efforts to lobby Congress would work. “We’ve tried,” he says. “It makes no difference.” He believes well-funded government contractors and powerful, “co-opted” lawmakers who lead key committees make up a virtually unstoppable surveillance-industrial complex.[12]

House Passage

The USA Freedom Act passed the U.S. House of Representatives on May 13, 2015.[82] The bill received broad support in the House, with 338 votes for the bill and 88 against it. It was passed without any amendments to the House Judiciary version because the House Rules Committee prohibited consideration of any amendment to the USA Freedom Act, claiming that any changes to the legislation would have weakened its chances of passage.[83] The bill had the support of the White House, Attorney General Eric Holder and Director of National Intelligence James Clapper. While civil liberties groups were divided over the support of the bill, lawmakers opposed to the Bill stated it will handicap the NSA and allow terrorist groups to prosper.[84]

Passage in Senate

The USA Freedom Act was not passed by the U.S. Senate on May 22, 2015. By a vote of 57-42, the Senate did not pass the bill that would have required 60 votes to move forward, which means that the NSA must start winding down its domestic mass surveillance program this week. The Senate also rejected, by 54-45, also short of the necessary 60 votes, a two-month extension for the key provision in the Patriot Act that has been used to justify NSA spying, which is set to expire on June 1, 2015.[85][86][87]

However, on May 31, 2015, the Senate voted 77-17 to limit debate on the act. Senate rules will allow it to be passed after the mass surveillance programs have expired.[88] Richard Burr, chair of the Senate Intelligence Committee, had three amendments he planned to offer to the bill which were likely to further increase opposition to the bill.[89] While several amendments which would strengthen the bill were not allowed to be considered, three amendments to weaken the bill, considered “poison pills”, were allowed to be considered but ultimately rejected.[90]

The bill ultimately passed the Senate 67-32 on June 2, 2015 and was signed into law.[91] “After a needless delay and inexcusable lapse in important national security authorities, my administration will work expeditiously to ensure our national security professionals again have the full set of vital tools they need to continue protecting the country,” Obama said.[92]

NSA Whistle-blower William Binney: The Future of FREEDOM

A 36-year veteran of America’s Intelligence Community, William Binney resigned from his position as Director for Global Communications Intelligence (COMINT) at the National Security Agency (NSA) and blew the whistle, after discovering that his efforts to protect the privacy and security of Americans were being undermined by those above him in the chain of command.

The NSA data-monitoring program which Binney and his team had developed — codenamed ThinThread — was being aimed not at foreign targets as intended, but at Americans (codenamed as Stellar Wind); destroying privacy here and around the world. Binney voices his call to action for the billions of individuals whose rights are currently being violated.

William Binney speaks out in this feature-length interview with Tragedy and Hope’s Richard Grove, focused on the topic of the ever-growing Surveillance State in America.

On January 22, 2015: (Berlin, Germany) – The Government Accountability Project (GAP) is proud to announce that retired NSA Technical Director and GAP client, William “Bill” Binney, will accept the Sam Adams Associates for Integrity in Intelligence Award today in Berlin, Germany. The award is presented annually by the Sam Adams Associates for Integrity in Intelligence (SAAII) to a professional who has taken a strong stand for ethics and integrity. http://whistleblower.org/press/nsa-wh…

NSA Whistle-blower: Everyone in US under virtual surveillance, all info stored, no matter the post

Former NSA Head Exposes Agency’s Real Crimes

Edward Snowden, v 1.0: NSA Whistleblower William Binney Tells All

“Where I see it going is toward a totalitarian state,” says William Binney. “You’ve got the NSA doing all this collecting of material on all of its citizens – that’s what the SS, the Gestapo, the Stasi, the KGB, and the NKVD did.”

Binney is talking about the collection of various forms of personal data on American citizens by the National Security Agency (NSA), where he worked for 30 years before quitting in 2001 from his high-placed post as technical leader for intelligence. A registered Republican for most of his life, Binney volunteered for military service during the Vietnam War, which led to his being hired by the NSA in the early ’70s.

In 2002 – long before the revelations of Edward Snowden rocked the world – Binney and several former colleagues went to Congress and the Department of Defense, asking that the NSA be investigated. Not only was the super-secretive agency wasting taxpayer dollars on ineffective programs, they argued, it was broadly violating constitutional guarantees to privacy and due process.

The government didn’t just turn a blind eye to the agency’s activities; it later accused the whistleblowers of leaking state secrets. A federal investigation of Binney – including an FBI search and seizure of his home and office computers that destroyed his consulting business – exonerated him on all charges.

“We are a clear example that [going through] the proper channels doesn’t work,” says Binney, who approves of Edward Snowden’s strategy of going straight to the media. At the same time, Binney criticizes Snowden’s leaking of documents not directly related to the NSA’s surveillance of American citizens and violation of constitutional rights. Binney believes that the NSA is vital to national security but has been become unmoored due to technological advances that vastly extend its capabilities and leadership that has no use for limits on government power. “They took that program designed [to prevent terrorist attacks] and used it to spy on American citizens and everyone else in the world,” flatly declares Binney (33:30).

Binney sat down with Reason TV’s Nick Gillespie to discuss “Trailblazer”, a data-collection program which was used on American citizens (1:00), why he thinks the NSA had the capability to stop the 9/11 attacks (7:00), his experience being raided by the FBI in 2007 (12:50), and why former President Gerald Ford, usually regarded as a hapless time-server, is one of his personal villians (41:25).

NSA Whistle-Blower Tells All: The Program | Op-Docs | The New York Times

William Edward Binney[2] is a former highly placed intelligence official with the United States National Security Agency (NSA)[3] turned whistleblower who resigned on October 31, 2001, after more than 30 years with the agency. He was a high-profile critic of his former employers during the George W. Bush administration.

Binney continues to speak out during Barack Obama‘s presidency about the NSA’s data collection policies, and continues interviews in the media regarding his experiences and his views on communication intercepts by governmental agencies of American citizens. In a legal case, Binney has testified in an affidavit that the NSA is in deliberate violation of the U.S. Constitution.

Biography

Binney grew up in rural Pennsylvania and graduated with a Bachelor of Science degree in mathematics from the Pennsylvania State University in 1970. He said that he volunteered for the Army during the Vietnam era in order to select work that would interest him rather than be drafted and have no input. He was found to have strong aptitudes for mathematics, analysis, and code-breaking,[4] and served four years from 1965–1969 at the Army Security Agency before going to the NSA in 1970. Binney was a Russia specialist and worked in the operations side of intelligence, starting as an analyst and ending as Technical Director prior to becoming a geopolitical world Technical Director. In the 1990s, he co-founded a unit on automating signals intelligence with NSA research chief Dr. John Taggart.[5] Binney’s NSA career culminated as Technical Leader for intelligence in 2001. Having expertise in intelligence analysis, traffic analysis, systems analysis, knowledge management, and mathematics (including set theory, number theory, and probability),[6] Binney has been described as one of the best analysts in the NSA’s history.[7] After retiring from the NSA he founded “Entity Mapping, LLC”, a private intelligence agency together with fellow NSA whistleblower J. Kirk Wiebe to market their analysis program to government agencies. NSA continued to retaliate against them, ultimately preventing them from getting work, or causing contracts they had secured to be terminated abruptly.[8]

Whistleblowing

Binney sitting in the offices ofDemocracy Now! in New York City, prior to appearing with hosts Amy Goodman, Juan Gonzalez, and guest Jacob Appelbaum. Photo taken byJacob Appelbaum.

In September 2002, he, along with J. Kirk Wiebe and Edward Loomis, asked the U.S. Defense Department to investigate the NSA for allegedly wasting “millions and millions of dollars” on Trailblazer, a system intended to analyze data carried on communications networks such as the Internet. Binney had been one of the inventors of an alternative system, ThinThread, which was shelved when Trailblazer was chosen instead. Binney has also been publicly critical of the NSA for spying on U.S. citizens, saying of its expanded surveillance after the September 11, 2001 attacks that “it’s better than anything that the KGB, the Stasi, or the Gestapo and SS ever had”[9] as well as noting Trailblazer’s ineffectiveness and unjustified high cost compared to the far less intrusive ThinThread.[10] He was furious that the NSA hadn’t uncovered the 9/11 plot and stated that intercepts it had collected but not analyzed likely would have garnered timely attention with his leaner more focused system.[7]

After he left the NSA in 2001, Binney was one of several people investigated as part of an inquiry into the 2005 New York Times exposé[11][12] on the agency’s warrantless eavesdropping program. Binney was cleared of wrongdoing after three interviews with FBI agents beginning in March 2007, but one morning in July 2007, a dozen agents armed with rifles appeared at his house, one of whom entered the bathroom and pointed his gun at Binney, still towelling off from a shower. In that raid, the FBI confiscated a desktop computer, disks, and personal and business records. The NSA revoked his security clearance, forcing him to close a business he ran with former colleagues at a loss of a reported $300,000 in annual income. In 2012, Binney and his co-plaintiffs went to federal court to get the items back. Binney spent more than $7,000 on legal fees.[13]

During interviews on Democracy Now! in April and May 2012[14] with elaboration in July 2012 at 2600’s hacker conference HOPE[4] and at DEF CON a couple weeks later,[15]Binney repeated estimates that the NSA (particularly its Stellar Wind project[16]) had intercepted 20 trillion communications “transactions” of Americans such as phone calls, emails, and other forms of data (but not including financial data). This includes most of the emails of US citizens. Binney disclosed in an affidavit for Jewel v. NSA[17] that the agency was “purposefully violating the Constitution”.[6] Binney also notes that he found out after retiring that the NSA was pursuing collect-it-all vs. targeted surveillance even before the 9/11 attacks.

Binney was invited as a witness by the NSA commission of the German Bundestag. On July 3, 2014 the Spiegel wrote, he said that the NSA wanted to have information about everything. In Binney’s view this is a totalitarian approach, which had previously been seen only in dictatorships.[18] Binney stated the goal was also to control people. Meanwhile, he said it is possible in principle to survey the whole population, abroad and in the US, which in his view contradicts the United States Constitution. In October 2001, shortly after the 9/11 attacks, the NSA began with its mass surveillance, he said. Therefore, he left the secret service shortly afterwards, after more than 30 years of employment. Binney mentioned that there were about 6000 analysts in the surveillance at NSA already during his tenure. According to him, everything changed after 9/11. The NSA used the attacks as a justification to start indiscriminate data collection. “This was a mistake. But they still do it”, he said. The secret service was saving the data as long as possible: “They do not discard anything. If they have anything they keep it.” Since then, the NSA has been saving collected data indefinitely. Binney said he deplored the NSA’s development of the past few years, to collect data not only on groups who are suspicious for criminal or terrorist activities. “We have moved away from the collection of these data to the collection of data of the 7 billion people on our planet.” Binney said he argued even then, to only pull relevant data from the cables. Access to the data was granted to departments of the government or the IRS.[18]

History of the award

Similar in name to the Medal of Freedom,[3] but much closer in meaning and precedence to the Medal for Merit: the Presidential Medal of Freedom is currently the supreme civilian decoration in precedence, whereas the Medal of Freedom was inferior in precedence to the Medal for Merit; the Medal of Freedom was awarded by any of three Cabinet secretaries, whereas the Medal for Merit was awarded by the president, as is the Presidential Medal of Freedom. Another measure of the difference between these two similarly named but very distinct awards is their per-capita frequency of award: from 1946 to 1961 the average annual incidence of award of the Medal of Freedom was approximately 1 per every 86,500 adult U.S. citizens; from 1996 to 2011 the average annual incidence of award of the Presidential Medal of Freedom was approximately 1 per every 20,500,000 adult U.S. citizens (so on an annualized per capita basis, 240 Medals of Freedom have been awarded per one Presidential Medal of Freedom).[2][4]

President John F. Kennedy established the current decoration in 1963 through Executive Order11085, with unique and distinctive insignia, vastly expanded purpose, and far higher prestige.[1] It was the first U.S. civilian neck decoration and, in the grade of Awarded With Distinction, is the only U.S. sash and star decoration (the Chief Commander degree of the Legion of Merit – which may only be awarded to foreign heads of state – is a star decoration, but without a sash). The Executive Order calls for the medal to be awarded annually on or around July 4, and at other convenient times as chosen by the president,[5] but it has not been awarded every year (e.g., 2001, 2010). Recipients are selected by the president, either on his own initiative or based on recommendations. The order establishing the medal also expanded the size and the responsibilities of the Distinguished Civilian Service Awards Board so it could serve as a major source of such recommendations.

Insignia

Graphical representation of the Presidential Medal of Freedom with Distinction

The badge of the Presidential Medal of Freedom is in the form of a golden star with white enamel, with a red enamel pentagon behind it; the central disc bears thirteen gold stars on a blue enamel background (taken from the Great Seal of the United States) within a golden ring. Golden American bald eagles with spread wings stand between the points of the star. It is worn around the neck on a blue ribbon with white edge stripes.

A special grade of the medal, known as the Presidential Medal of Freedom with Distinction,[6] has a larger execution of the same medal design worn as a star on the left chest along with a sash over the right shoulder (similar to how the insignia of a Grand Cross is worn), with its rosette (blue with white edge, bearing the central disc of the medal at its center) resting on the left hip. When the medal With Distinction is awarded, the star may be presented depending from a neck ribbon and can be identified by its larger size than the standard medal (compare size of medals in pictures below; President Reagan’s was awarded With Distinction).

Both medals may also be worn in miniature form on a ribbon on the left chest, with a silver American bald eagle with spread wings on the ribbon, or a golden American bald eagle for a medal awarded With Distinction. In addition, the medal is accompanied by a service ribbon for wear on military service uniform, a miniature medal pendant for wear on mess dress or civilian formal wear, and a lapel badge for wear on civilian clothes (all shown in the accompanying photograph of the full presentation set).

Through a PRISM, Darkly – Everything we know about NSA spying [30c3]

Published on Dec 30, 2013

Through a PRISM, Darkly
Everything we know about NSA spying

From Stellar Wind to PRISM, Boundless Informant to EvilOlive, the NSA spying programs are shrouded in secrecy and rubber-stamped by secret opinions from a court that meets in a faraday cage. The Electronic Frontier Foundation’s Kurt Opsahl explains the known facts about how the programs operate and the laws and regulations the U.S. government asserts allows the NSA to spy on you.
The Electronic Frontier Foundation, a non-profit civil society organization, has been litigating against the NSA spying program for the better part of a decade. EFF has collected and reviewed dozens of documents, from the original NY Times stories in 2005 and the first AT&T whistleblower in 2006, through the latest documents released in the Guardian or obtained through EFF’s Freedom of Information (government transparency) litigation. EFF attorney Kurt Opsahl’s lecture will describe how the NSA spying program works, the underlying technologies, the targeting procedures (how they decide who to focus on), the minimization procedures (how they decide which information to discard), and help you makes sense of the many code names and acronyms in the news. He will also discuss the legal and policy ramifications that have become part of the public debate following the recent disclosures, and what you can do about it. After summarizing the programs, technologies, and legal/policy framework in the lecture, the audience can ask questions.

Glenn Becks “SURVEILLANCE STATE”

Inside the NSA

Ed Snowden, NSA, and Fairy Tales

AT&T Spying On Internet Traffic

For years the National Securities Agency, has been spying on each & every keystroke. The national headquarters of AT&T is in Missouri, where ex-employees describe a secret room. The program is called “Splitter Cut-In & Test Procedure.”

NSA Whistle-Blower Tells All – Op-Docs: The Program

The filmmaker Laura Poitras profiles William Binney, a 32-year veteran of the National Security Agency who helped design a top-secret program he says is broadly collecting Americans’ personal data.

NSA Whistleblower: Everyone in US under virtual surveillance, all info stored, no matter the post

He told you so: Bill Binney talks NSA leaks

William Benny – The Government is Profiling You (The NSA is Spying on You)

‘After 9/11 NSA had secret deal with White House’

The story of Whistleblower Thomas Drake

Whistleblowers, Part Two: Thomas Drake

NSA Whistleblower Thomas Drake speaks at National Press Club – March 15, 2013

Meet Edward Snowden: NSA PRISM Whistleblower

The Truth About Edward Snowden

N.S.A. Spying: Why Does It Matter?

Inside The NSA~Americas Cyber Secrets

NSA Whistleblower Exposes Obama’s Dragnet

AT&T whistleblower against immunity for Bush spy program-1/2

AT&T Whistleblower Urges Against Immunity for Telecoms in Bush Spy Program

The Senate is expected to vote on a controversial measure to amend the Foreign Intelligence Surveillance Act tomorrow. The legislation would rewrite the nation’s surveillance laws and authorize the National Security Agency’s secret program of warrantless wiretapping. We speak with Mark Klein, a technician with AT&T for over twenty-two years. In 2006 Klein leaked internal AT&T documents that revealed the company had set up a secret room in its San Francisco office to give the National Security Agency access to its fiber optic internet cables.

AT&T whistleblower against immunity for Bush spy program-2/2

Enemy Of The State 1998 (1080p) (Full movie)

Background Articles and Videos

Stellar Wind

Stellar Wind was the open secret code name for four surveillance programs by the United States National Security Agency (NSA) during the presidency of George W. Bush and revealed by Thomas Tamm to The New York Times reporters James Risen and Eric Lichtblau.[1] The operation was approved by President George W. Bush shortly after the September 11 attacks in 2001.[2] Stellar Wind was succeeded during the presidency of Barack Obama by four major lines of intelligence collection in the territorial United States, together capable of spanning the full range of modern telecommunications.[3]

The program’s activities involved data mining of a large database of the communications of American citizens, including e-mail communications, phone conversations, financial transactions, and Internet activity.[1] William Binney, a retired Technical Leader with the NSA, discussed some of the architectural and operational elements of the program at the 2012 Chaos Communication Congress.[4]

There were internal disputes within the Justice Department about the legality of the program, because data are collected for large numbers of people, not just the subjects of Foreign Intelligence Surveillance Act (FISA) warrants.[4]

During the Bush Administration, the Stellar Wind cases were referred to by FBI agents as “pizza cases” because many seemingly suspicious cases turned out to be food takeout orders. According to Mueller, approximately 99 percent of the cases led nowhere, but “it’s that other 1% that we’ve got to be concerned about”.[2] One of the known uses of these data were the creation of suspicious activity reports, or “SARS”, about people suspected of terrorist activities. It was one of these reports that revealed former New York governor Eliot Spitzer’s use of prostitutes, even though he was not suspected of terrorist activities.[1]

In March 2012 Wired magazine published “The NSA Is Building the Country’s Biggest Spy Center (Watch What You Say)” talking about a vast new NSA facility in Utah and says “For the first time, a former NSA official has gone on the record to describe the program, codenamed Stellar Wind, in detail,” naming the official William Binney, a former NSA code breaker. Binney went on to say that the NSA had highly secured rooms that tap into major switches, and satellite communications at both AT&T and Verizon.[5] The article suggested that the otherwise dispatched Stellar Wind is actually an active program.

PRISM

PRISM is a clandestine national security electronic surveillance program operated by the United States National Security Agency (NSA) since 2007.[1][2][3][Notes 1]PRISM is a government codename for a data collection effort known officially as US-984XN.[8][9] It is operated under the supervision of the United States Foreign Intelligence Surveillance Court pursuant to the Foreign Intelligence Surveillance Act (FISA).[10] The existence of the program was leaked by NSA contractor Edward Snowden and published by The Guardian and The Washington Post on June 6, 2013.

A document included in the leak indicated that the PRISM SIGAD was “the number one source of raw intelligence used for NSA analytic reports.”[11] The President’s Daily Brief, an all-source intelligence product, cited PRISM data as a source in 1,477 items in 2012.[12] The leaked information came to light one day after the revelation that the United States Foreign Intelligence Surveillance Court had been requiring the telecommunications company Verizon to turn over to the NSA logs tracking all of its customers’ telephone calls on an ongoing daily basis.[13][14]

According to the Director of National Intelligence James Clapper, PRISM cannot be used to intentionally target any Americans or anyone in the United States. Clapper said a special court, Congress, and the executive branch oversee the program and extensive procedures ensure the acquisition, retention, and dissemination of data accidentally collected about Americans is kept to a minimum.[15] Clapper issued a statement and “fact sheet”[16] to correct what he characterized as “significant misimpressions” in articles by The Washington Post and The Guardian newspapers.[17]

History

Slide showing that much of the world’s communications flow through the US

Details of information collected via PRISM

PRISM is a “Special Source Operation” in the tradition of NSA’s intelligence alliances with as many as 100 trusted U.S. companies since the 1970s.[1] A prior program, the Terrorist Surveillance Program, was implemented in the wake of the September 11 attacks under the George W. Bush Administration but was widely criticized and had its legality questioned, because it was conducted without approval of the Foreign Intelligence Surveillance Court (FISC).[18][19][20][21] PRISM was authorized by an order of the FISC.[11] Its creation was enabled by the Protect America Act of 2007 under President Bush and the FISA Amendments Act of 2008, which legally immunized private companies that cooperated voluntarily with US intelligence collection and was renewed by Congress under President Obama in 2012 for five years until December 2017.[2][22] According to The Register, the FISA Amendments Act of 2008 “specifically authorizes intelligence agencies to monitor the phone, email, and other communications of U.S. citizens for up to a week without obtaining a warrant” when one of the parties is outside the U.S.[22]

PRISM was first publicly revealed on June 6, 2013, after classified documents about the program were leaked to The Washington Post and The Guardian by American Edward Snowden.[2][1] The leaked documents included 41 PowerPoint slides, four of which were published in news articles.[1][2] The documents identified several technology companies as participants in the PRISM program, including (date of joining PRISM in parentheses) Microsoft (2007), Yahoo! (2008), Google (2009), Facebook (2009), Paltalk (2009), YouTube (2010), AOL (2011), Skype (2011), and Apple (2012).[23] The speaker’s notes in the briefing document reviewed by The Washington Post indicated that “98 percent of PRISM production is based on Yahoo, Google and Microsoft.”[1]

The slide presentation stated that much of the world’s electronic communications pass through the United States, because electronic communications data tend to follow the least expensive route rather than the most physically direct route, and the bulk of the world’s internet infrastructure is based in the United States.[11] The presentation noted that these facts provide United States intelligence analysts with opportunities for intercepting the communications of foreign targets as their electronic data pass into or through the United States.[2][11]

According to The Washington Post, the intelligence analysts search PRISM data using terms intended to identify suspicious communications of targets whom the analysts suspect with at least 51 percent confidence to not be United States citizens, but in the process, communication data of some United States citizens are also collected unintentionally.[1] Training materials for analysts tell them that while they should periodically report such accidental collection of non-foreign United States data, “it’s nothing to worry about.”[1]

Response from companies

The original Washington Post and Guardian articles reporting on PRISM noted that one of the leaked briefing documents said PRISM involves collection of data “directly from the servers” of several major internet services providers.[2][1]

Initial Public Statements

Corporate executives of several companies identified in the leaked documents told The Guardian that they had no knowledge of the PRISM program in particular and also denied making information available to the government on the scale alleged by news reports.[2][24] Statements of several of the companies named in the leaked documents were reported by TechCrunch and The Washington Post as follows:[25][26]

Slide listing companies and the date that PRISM collection began

Microsoft: “We provide customer data only when we receive a legally binding order or subpoena to do so, and never on a voluntary basis. In addition we only ever comply with orders for requests about specific accounts or identifiers. If the government has a broader voluntary national security program to gather customer data we don’t participate in it.”[25]

Yahoo!: “Yahoo! takes users’ privacy very seriously. We do not provide the government with direct access to our servers, systems, or network.”[25] “Of the hundreds of millions of users we serve, an infinitesimal percentage will ever be the subject of a government data collection directive.”[26]

Facebook: “We do not provide any government organization with direct access to Facebook servers. When Facebook is asked for data or information about specific individuals, we carefully scrutinize any such request for compliance with all applicable laws, and provide information only to the extent required by law.”[25]

Google: “Google cares deeply about the security of our users’ data. We disclose user data to government in accordance with the law, and we review all such requests carefully. From time to time, people allege that we have created a government ‘back door’ into our systems, but Google does not have a backdoor for the government to access private user data.”[25] “[A]ny suggestion that Google is disclosing information about our users’ Internet activity on such a scale is completely false.”[26]

Apple: “We have never heard of PRISM. We do not provide any government agency with direct access to our servers, and any government agency requesting customer data must get a court order.”[27]

Dropbox: “We’ve seen reports that Dropbox might be asked to participate in a government program called PRISM. We are not part of any such program and remain committed to protecting our users’ privacy.”[25]

In response to the technology companies’ denials of the NSA being able to directly access the companies’ servers, The New York Times reported that sources had stated the NSA was gathering the surveillance data from the companies using other technical means in response to court orders for specific sets of data.[13]The Washington Post suggested, “It is possible that the conflict between the PRISM slides and the company spokesmen is the result of imprecision on the part of the NSA author. In another classified report obtained by The Post, the arrangement is described as allowing ‘collection managers [to send] content tasking instructions directly to equipment installed at company-controlled locations,’ rather than directly to company servers.”[1] “[I]n context, ‘direct’ is more likely to mean that the NSA is receiving data sent to them deliberately by the tech companies, as opposed to intercepting communications as they’re transmitted to some other destination.[26]

“If these companies received an order under the FISA amendments act, they are forbidden by law from disclosing having received the order and disclosing any information about the order at all,” Mark Rumold, staff attorney at the Electronic Frontier Foundation, told ABC News.[28]

Slide showing two different sources of NSA data collection. The first source the fiber optic cables of the internet handled by the Upstream program and the second source the servers of major internet companies handled by PRISM.[29]

On May 28, 2013, Google was ordered by United States District Court Judge Susan Illston to comply with a National Security Letter issued by the FBI to provide user data without a warrant.[30] Kurt Opsahl, a senior staff attorney at the Electronic Frontier Foundation, in an interview with VentureBeat said, “I certainly appreciate that Google put out a transparency report, but it appears that the transparency didn’t include this. I wouldn’t be surprised if they were subject to a gag order.”[31]

The New York Times reported on June 7, 2013, that “Twitter declined to make it easier for the government. But other companies were more compliant, according to people briefed on the negotiations.”[32] The other companies held discussions with national security personnel on how to make data available more efficiently and securely.[32] In some cases, these companies made modifications to their systems in support of the intelligence collection effort.[32] The dialogues have continued in recent months, as General Martin Dempsey, the chairman of the Joint Chiefs of Staff, has met with executives including those at Facebook, Microsoft, Google and Intel.[32] These details on the discussions provide insight into the disparity between initial descriptions of the government program including a training slide which states “Collection directly from the servers”[29] and the companies’ denials.[32]

While providing data in response to a legitimate FISA request approved by FISC is a legal requirement, modifying systems to make it easier for the government to collect the data is not. This is why Twitter could legally decline to provide an enhanced mechanism for data transmission.[32] Other than Twitter, the companies were effectively asked to construct a locked mailbox and provide the key to the government, people briefed on the negotiations said.[32] Facebook, for instance, built such a system for requesting and sharing the information.[32] Google does not provide a lockbox system, but instead transmits required data by hand delivery or secure FTP.[33]

Post-PRISM Transparency Reports

In response to the publicity surrounding media reports of data-sharing, several companies requested permission to reveal more public information about the nature and scope of information provided in response to National Security requests.

On June 14, 2013, Facebook reported that the U.S. Government had authorized the communication of “about these numbers in aggregate, and as a range.” In a press release posted to their web site, Facebook reported, “For the six months ending December 31, 2012, the total number of user-data requests Facebook received from any and all government entities in the U.S. (including local, state, and federal, and including criminal and national security-related requests) – was between 9,000 and 10,000.” Facebook further reported that the requests impacted “between 18,000 and 19,000” user accounts, a “tiny fraction of one percent” of more than 1.1 billion active user accounts.[34]

Microsoft reported that for the same period, it received “between 6,000 and 7,000 criminal and national security warrants, subpoenas and orders affecting between 31,000 and 32,000 consumer accounts from U.S. governmental entities (including local, state and federal)” which impacted “a tiny fraction of Microsoft’s global customer base”.[35]

Google issued a statement criticizing the requirement that data be reported in aggregated form, stating that lumping national security requests with criminal request data would be “a step backwards” from its previous, more detailed practices on its site transparency report. The company said that it would continue to seek government permission to publish the number and extent of FISA requests.[36]

Response from United States government

Executive branch

Shortly after publication of the reports by The Guardian and The Washington Post, the United States Director of National Intelligence, James Clapper, on June 7 released a statement confirming that for nearly six years the government of the United States had been using large internet services companies such as Google and Facebook to collect information on foreigners outside the United States as a defense against national security threats.[13] The statement read in part, “The Guardian and The Washington Post articles refer to collection of communications pursuant to Section 702 of the Foreign Intelligence Surveillance Act. They contain numerous inaccuracies.”[37] He went on to say, “Section 702 is a provision of FISA that is designed to facilitate the acquisition of foreign intelligence information concerning non-U.S. persons located outside the United States. It cannot be used to intentionally target any U.S. citizen, any other U.S. person, or anyone located within the United States.”[37] Clapper concluded his statement by stating “The unauthorized disclosure of information about this important and entirely legal program is reprehensible and risks important protections for the security of Americans.”[37] On March 12, 2013, Clapper had told the United States Senate Select Committee on Intelligence that the NSA does “not wittingly” collect any type of data on millions or hundreds of millions of Americans.[38] In an NBC News interview, Clapper said he answered Senator Wyden’s question in the “least untruthful manner by saying no”.[39]

Clapper also stated that “the NSA collects the phone data in broad swaths, because collecting it (in) a narrow fashion would make it harder to identify terrorism-related communications. The information collected lets the government, over time, make connections about terrorist activities. The program doesn’t let the U.S. listen to people’s calls, but only includes information like call length and telephone numbers dialed.”[15]

On June 8, 2013, Clapper said “the surveillance activities published in The Guardian and The Washington Post are lawful and conducted under authorities widely known and discussed, and fully debated and authorized by Congress.”[40][10] The fact sheet described PRISM as “an internal government computer system used to facilitate the government’s statutorily authorized collection of foreign intelligence information from electronic communication service providers under court supervision, as authorized by Section 702 of the Foreign Intelligence Surveillance Act (FISA) (50 U.S.C. § 1881a).”[10]

The National Intelligence fact sheet further stated that “the United States Government does not unilaterally obtain information from the servers of U.S. electronic communication service providers. All such information is obtained with FISA Court approval and with the knowledge of the provider based upon a written directive from the Attorney General and the Director of National Intelligence.” It said that the Attorney General provides FISA Court rulings and semi-annual reports about PRISM activities to Congress, “provid[ing] an unprecedented degree of accountability and transparency.”[10]

The President of the United States, Barack Obama, said on June 7 “What you’ve got is two programs that were originally authorized by Congress, have been repeatedly authorized by Congress. Bipartisan majorities have approved them. Congress is continually briefed on how these are conducted. There are a whole range of safeguards involved. And federal judges are overseeing the entire program throughout.”[41] He also said, “You can’t have 100 percent security and then also have 100 percent privacy and zero inconvenience. You know, we’re going to have to make some choices as a society.”[41]

In separate statements, senior (not mentioned by name in source) Obama administration officials said that Congress had been briefed 13 times on the programs since 2009.[42]

Legislative branch

In contrast to their swift and forceful reactions the previous day to allegations that the government had been conducting surveillance of United States citizens’ telephone records, Congressional leaders initially had little to say about the PRISM program the day after leaked information about the program was published. Several lawmakers declined to discuss PRISM, citing its top-secret classification,[43] and others said that they had not been aware of the program.[44] After statements had been released by the President and the Director of National Intelligence, some lawmakers began to comment:

Senator John McCain (R-AZ)

June 9 “We passed the Patriot Act. We passed specific provisions of the act that allowed for this program to take place, to be enacted in operation,”[45]

June 9 “These programs are within the law”, “part of our obligation is keeping Americans safe”, “Human intelligence isn’t going to do it”.[46]

June 9 “Here’s the rub: the instances where this has produced good — has disrupted plots, prevented terrorist attacks, is all classified, that’s what’s so hard about this.”[47]

June 11 “It went fine…we asked him[ Keith Alexander ] to declassify things because it would be helpful (for people and lawmakers to better understand the intelligence programs).” “I’ve just got to see if the information gets declassified. I’m sure people will find it very interesting.”[48]

Senator Susan Collins (R-ME), member of Senate Intelligence Committee and past member of Homeland Security Committee

June 11 “I had, along with Joe Lieberman, a monthly threat briefing, but I did not have access to this highly compartmentalized information” and “How can you ask when you don’t know the program exists?”[49]

Representative John Boehner (R-OH), Speaker of the House of Representatives

June 9, “This is well beyond what the Patriot Act allows.”[51] “President Obama’s claim that ‘this is the most transparent administration in history’ has once again proven false. In fact, it appears that no administration has ever peered more closely or intimately into the lives of innocent Americans.”[51]

Representative Mike Rogers (R-MI), a Chairman of the Permanent Select Committee on Intelligence.

June 9 “One of the things that we’re charged with is keeping America safe and keeping our civil liberties and privacy intact. I think we have done both in this particular case,”[46]

June 9 “Within the last few years this program was used to stop a program, excuse me, to stop a terrorist attack in the United States we know that. It’s, it’s, it’s important, it fills in a little seam that we have and it’s used to make sure that there is not an international nexus to any terrorism event that they may believe is ongoing in the United States. So in that regard it is a very valuable thing,”[52]

Senator Mark Udall (D-CO)

June 9 “I don’t think the American public knows the extent or knew the extent to which they were being surveilled and their data was being collected.” “I think we ought to reopen the Patriot Act and put some limits on the amount of data that the National Security (Agency) is collecting,” “It ought to remain sacred, and there’s got to be a balance here. That is what I’m aiming for. Let’s have the debate, let’s be transparent, let’s open this up”.[46]

Representative Todd Rokita (R-IN)

June 10 “We have no idea when they [ FISA ] meet, we have no idea what their judgments are”,[53]

Senator Rand Paul (R-KY)

June 6 “When the Senate rushed through a last-minute extension of the FISA Amendments Act late last year, I insisted on a vote on my amendment (SA 3436) to require stronger protections on business records and prohibiting the kind of data-mining this case has revealed. Just last month, I introduced S.1037, the Fourth Amendment Preservation and Protection Act,”[54]

June 9 “I’m going to be seeing if I can challenge this at the Supreme Court level. I’m going to be asking the Internet providers and all of the phone companies: ask your customers to join me in a class-action lawsuit.”[45]

Representative Luis Gutierrez (D-IL)

June 9 “We will be receiving secret briefings and we will be asking, I know I’m going to be asking to get more information. I want to make sure that what they’re doing is harvesting information that is necessary to keep us safe and not simply going into everybody’s private telephone conversations and Facebook and communications. I mean one of the, you know the terrorists win when you debilitate freedom of expression and privacy.”[52]

Judicial branch

The Foreign Intelligence Surveillance Court (FISC) has not acknowledged, denied or confirmed any involvement in the PRISM program at this time. It has not issued any press statement or release relating to the current situation and uncertainty.

Applicable law and practice

On June 8, 2013, the Director of National Intelligence issued a fact sheet stating that PRISM “is not an undisclosed collection or data mining program”, but rather computer software used to facilitate the collection of foreign intelligence information “under court supervision, as authorized by Section 702 of the Foreign Intelligence Surveillance Act (FISA) (50 U.S.C. § 1881a).”[10] Section 702 provides that “the Attorney General [A.G.] and the Director of National Intelligence [DNI] may authorize jointly, for a period of up to 1 year from the effective date of the authorization, the targeting of persons reasonably believed to be located outside the United States to acquire foreign intelligence information.”[55] In order to authorize the targeting, the A.G. and DNI need to get an order from the Foreign Intelligence Surveillance Court (FISC) pursuant to Section 702 or certify that “intelligence important to the national security of the United States may be lost or not timely acquired and time does not permit the issuance of an order.”[55] When asking for an order, the A.G. and DNI must certify to FISC that “a significant purpose of the acquisition is to obtain foreign intelligence information.”[55] They do not need to specify which facilities or property that the targeting will be directed at.[55]

After getting a FISC order or determining that there are emergency circumstances, the A.G. and DNI can direct an electronic communication service provider to give them access to information or facilities to carry out the targeting and keep the targeting secret.[55] The provider then has the option to: (1) comply with the directive; (2) reject it; or (3) challenge it to FISC.

If the provider complies with the directive, it is released from liability to its users for providing the information and reimbursed for the cost of providing it.[55]

If the provider rejects the directive, the A.G. may request an order from FISC to enforce it.[55] A provider that fails to comply with FISC’s order can be punished with contempt of court.[55]

Finally, a provider can petition FISC to reject the directive.[55] In case FISC denies the petition and orders the provider to comply with the directive, the provider risks contempt of court if it refuses to comply with FISC’s order.[55] The provider can appeal FISC’s denial to the Foreign Intelligence Surveillance Court of Review and then appeal the Court of Review’s decision to the Supreme Court by a writ of certiorari for review under seal.[55]

The Senate Select Committee on Intelligence and the FISA Courts had been put in place to oversee intelligence operations in the period after the death of J. Edgar Hoover. Beverly Gage of Slate said, “When they were created, these new mechanisms were supposed to stop the kinds of abuses that men like Hoover had engineered. Instead, it now looks as if they have come to function as rubber stamps for the expansive ambitions of the intelligence community. J. Edgar Hoover no longer rules Washington, but it turns out we didn’t need him anyway.”[56]

Involvement of other countries

Australia

The Australian government has said it will investigate the impact of the PRISM program and the use of the Pine Gap surveillance facility on the privacy of Australian citizens.[57]

Canada

Canada’s national cryptologic agency, the Communications Security Establishment, said that commenting on PRISM “would undermine CSE’s ability to carry out its mandate”. Privacy Commissioner Jennifer Stoddart lamented Canada’s standards when it comes to protecting personal online privacy stating “We have fallen too far behind,” Stoddart wrote in her report. “While other nations’ data protection authorities have the legal power to make binding orders, levy hefty fines and take meaningful action in the event of serious data breaches, we are restricted to a ‘soft’ approach: persuasion, encouragement and, at the most, the potential to publish the names of transgressors in the public interest.” And, “when push comes to shove,” Stoddart wrote, “short of a costly and time-consuming court battle, we have no power to enforce our recommendations.”[58]

Germany

Germany did not receive any raw PRISM data, according to a Reuters report.[59]

Israel

Israeli newspaper Calcalist discussed[60] the Business Insider article[61] about the possible involvement of technologies from two secretive Israeli companies in the PRISM program – Verint Systems and Narus.

New Zealand

In New Zealand, University of Otago information science Associate Professor Hank Wolfe said that “under what was unofficially known as the Five Eyes Alliance, New Zealand and other governments, including the United States, Australia, Canada, and Britain, dealt with internal spying by saying they didn’t do it. But they have all the partners doing it for them and then they share all the information.”[62]

United Kingdom

In the United Kingdom, Government Communications Headquarters (GCHQ) has had access to the PRISM program on or before June 2010 and wrote 197 reports with it in 2012 alone. PRISM may have allowed GCHQ to circumvent the formal legal process required to seek personal material.[63][64]

Domestic response

The neutrality of this section is disputed. Please do not remove this message until the dispute is resolved. (June 2013)

The New York Times editorial board charged that the Obama administration “has now lost all credibility on this issue,”[65] and lamented that “for years, members of Congress ignored evidence that domestic intelligence-gathering had grown beyond their control, and, even now, few seem disturbed to learn that every detail about the public’s calling and texting habits now reside in a N.S.A. database.”[66]

Republican and former member of Congress Ron Paul said, “We should be thankful for individuals like Edward Snowden and Glenn Greenwald who see injustice being carried out by their own government and speak out, despite the risk…. They have done a great service to the American people by exposing the truth about what our government is doing in secret.”[67] Paul denounced the government’s secret surveillance program: “The government does not need to know more about what we are doing…. We need to know more about what the government is doing.”[67] He called Congress “derelict in giving that much power to the government,” and said that had he been elected president, he would have ordered searches only when there was probable cause of a crime having been committed, which he said was not how the PRISM program was being operated.[68]

In response to Obama administration arguments that it could stop terrorism in the cases of Najibullah Zazi and David Headley, Ed Pilkington and Nicholas Watt of The Guardian said in regards to the role of PRISM and Boundless Informant interviews with parties involved in the Zazi scheme and court documents lodged in the United States and the United Kingdom indicated that “conventional” surveillance methods such as “old-fashioned tip-offs” of the British intelligence services initiated the investigation into the Zazi case.[69] An anonymous former CIA agent said that in regards to the Headley case, “That’s nonsense. It played no role at all in the Headley case. That’s not the way it happened at all.”[69] Pilkington and Watt concluded that the data-mining programs “played a relatively minor role in the interception of the two plots.”[69] Michael Daly of The Daily Beast stated that even though Tamerlan Tsarnaev had visited Inspire and even though Russian intelligence officials alerted U.S. intelligence officials about Tsarnaev, PRISM did not prevent him from carrying out the Boston bombings, and that the initial evidence implicating him came from his brother Dzhokhar Tsarnaev and not from federal intelligence. In addition Daly pointed to the fact that Faisal Shahzad visited Inspire but that federal authorities did not stop his attempted terrorist plot. Daly concluded “The problem is not just what the National Security Agency is gathering at the risk of our privacy but what it is apparently unable to monitor at the risk of our safety.”[70] In addition, political commentator Bill O’Reilly criticized the government, saying that PRISM did not stop the Boston bombings.[71]

In a blog post, David Simon, the creator of The Wire, compared the NSA’s programs, including PRISM, to a 1980s effort by the City of Baltimore to add dialed number recorders to all pay phones to know which individuals were being called by the callers;[72] the city believed that drug traffickers were using pay phones and pagers, and a municipal judge allowed the city to place the recorders. The placement of the dialers formed the basis of the show’s first season. Simon argued that the media attention regarding the NSA programs is a “faux scandal.”[72][73] George Takei, an actor who had experienced Japanese American internment, said that due to his memories of the internment, he felt concern towards the NSA surveillance programs that had been revealed.[74]

The Electronic Frontier Foundation (EFF), an international non-profit digital-rights group based in the U.S., is hosting a tool, by which an American resident can write to their government representatives regarding their opposition to mass spying.[75]

On June 11, 2013, the American Civil Liberties Union filed a lawsuit against the NSA citing that PRISM “violates Americans’ constitutional rights of free speech, association, and privacy”.[76]

International response

Reactions of Internet users in China were mixed between viewing a loss of freedom worldwide and seeing state surveillance coming out of secrecy. The story broke just before US President Barack Obama and Chinese President Xi Jinping met in California.[77][78] When asked about NSA hacking China, the spokeswoman of Ministry of Foreign Affairs of the People’s Republic of China said “China strongly advocates cybersecurity”.[79] The party-owned newspaper Liberation Daily described this surveillance like Nineteen Eighty-Four-style.[80] Hong Kong legislators Gary Fan and Claudia Mo wrote a letter to Obama, stating “the revelations of blanket surveillance of global communications by the world’s leading democracy have damaged the image of the U.S. among freedom-loving peoples around the world.”[81]

Sophie in ‘t Veld, a Dutch Member of the European Parliament, called PRISM “a violation of EU laws”.[82]

Protests at Checkpoint Charlie in Berlin

The German Federal Commissioner for Data Protection and Freedom of Information, Peter Schaar, condemned the program as “monstrous”.[83] He further added that White House claims do “not reassure me at all” and that “given the large number of German users of Google, Facebook, Apple or Microsoft services, I expect the German government […] is committed to clarification and limitation of surveillance.” Steffen Seibert, press secretary of the Chancellor’s office, announced that Angela Merkel will put these issues on the agenda of the talks with Barack Obama during his pending visit in Berlin.[84]

The Italian president of the Guarantor for the protection of personal data, Antonello Soro, said that the surveillance dragnet “would not be legal in Italy” and would be “contrary to the principles of our legislation and would represent a very serious violation”.[85]

William Hague, the foreign secretary of the United Kingdom, dismissed accusations that British security agencies had been circumventing British law by using information gathered on British citizens by Prism[86] saying, “Any data obtained by us from the United States involving UK nationals is subject to proper UK statutory controls and safeguards.”[86] David Cameron said Britain’s spy agencies that received data collected from PRISM acted within the law: “I’m satisfied that we have intelligence agencies that do a fantastically important job for this country to keep us safe, and they operate within the law.”[86][87] Malcolm Rifkind, the chairman of parliament’s Intelligence and Security Committee, said that if the British intelligence agencies were seeking to know the content of emails about people living in the UK, then they actually have to get lawful authority.[87] The UK’s Information Commissioner’s Office was more cautious, saying it would investigate PRISM alongside other European data agencies: “There are real issues about the extent to which U.S. law agencies can access personal data of UK and other European citizens. Aspects of U.S. law under which companies can be compelled to provide information to U.S. agencies potentially conflict with European data protection law, including the UK’s own Data Protection Act. The ICO has raised this with its European counterparts, and the issue is being considered by the European Commission, who are in discussions with the U.S. Government.”[82]

Ai Weiwei, a Chinese dissident, said “Even though we know governments do all kinds of things I was shocked by the information about the US surveillance operation, Prism. To me, it’s abusively using government powers to interfere in individuals’ privacy. This is an important moment for international society to reconsider and protect individual rights.”[88]

Kim Dotcom, a German-Finnish Internet entrepreneur who owned Megaupload, which was closed by the U.S. federal government, said “We should heed warnings from Snowden because the prospect of an Orwellian society outweighs whatever security benefits we derive from Prism or Five Eyes.”[89] The Hong Kong law firm representing Dotcom expressed a fear that the communication between Dotcom and the firm had been compromised by U.S. intelligence programs.[90]

Russia has offered to consider an asylum request from Edward Snowden.[91]

Taliban spokesperson Zabiullah Mujahid said “We knew about their past efforts to trace our system. We have used our technical resources to foil their efforts and have been able to stop them from succeeding so far.”[92][93]

Related government Internet surveillance programs

A parallel program, code-named BLARNEY, gathers up metadata as it streams past choke points along the backbone of the Internet. BLARNEY’s summary, set down in the slides alongside a cartoon insignia of a shamrock and a leprechaun hat, describes it as “an ongoing collection program that leverages IC [intelligence community] and commercial partnerships to gain access and exploit foreign intelligence obtained from global networks.”[94]

A related program, a big data visualization system based on cloud computing and free and open-source software (FOSS) technology known as “Boundless Informant”, was disclosed in documents leaked to The Guardian and reported on June 8, 2013. A leaked, top secret map allegedly produced by Boundless Informant revealed the extent of NSA surveillance in the U.S.[95]

ThinThread

ThinThread is the name of a project that the United States National Security Agency (NSA) pursued during the 1990s, according to a May 17, 2006 article in The Baltimore Sun.[1] The program involved wiretapping and sophisticated analysis of the resulting data, but according to the article, the program was discontinued three weeks before the September 11, 2001 attacks due to the changes in priorities and the consolidation of U.S. intelligence authority.[2] The “change in priority” consisted of the decision made by the director of NSA General Michael V. Hayden to go with a concept called Trailblazer, despite the fact that ThinThread was a working prototype that protected the privacy of U.S. citizens.

ThinThread was dismissed and replaced by the Trailblazer Project, which lacked the privacy protections.[3] A consortium led by Science Applications International Corporation was awarded a $280 million contract to develop Trailblazer in 2002.[4]

Trailblazer

Trailblazer was a United States National Security Agency (NSA) program intended to develop a capability to analyze data carried on communications networks like the Internet. It was intended to track entities using communication methods such as cell phones and e-mail.[1][2] It ran over budget, failed to accomplish critical goals, and was cancelled.

NSA whistleblowers J. Kirk Wiebe, William Binney, Ed Loomis, and House Permanent Select Committee on Intelligence staffer Diane Roark complained to the Department of Defense’s Inspector General (IG) about waste, fraud, and abuse in the program, and the fact that a successful operating prototype existed, but was ignored when the Trailblazer program was launched. The complaint was accepted by the IG and an investigation began that lasted until mid-2005 when the final results were issued. The results were largely hidden, as the report given to the public was heavily (90%) redacted, while the original report was heavily classified, thus restricting the ability of most people to see it.

The people who filed the IG complaint were later raided by armed Federal Bureau of Investigation (FBI) agents. While the Government threatened to prosecute all who signed the IG report, it ultimately chose to pursue an NSA Senior Executive — Thomas Andrews Drake — who helped with the report internally to NSA and who had spoken with a reporter about the project. Drake was later charged under the Espionage Act of 1917. His defenders claimed this was retaliation.[3][4] The charges against him were later dropped, and he agreed to plead guilty to having committed a misdemeanor under the Computer Fraud and Abuse Act, something that Jesselyn Radack of the Government Accountability Project (which helped represent him) called an “act of civil disobedience”.[5]

Background

Trailblazer was chosen over a similar program named ThinThread, a less costly project which had been designed with built-in privacy protections for United States citizens.[4][3] Trailblazer was later linked to the NSA electronic surveillance program and the NSA warrantless surveillance controversy.[3]

In 2002 a consortium led by Science Applications International Corporation was chosen by the NSA to produce a technology demonstration platform in a contract worth $280 million. Project participants included Boeing, Computer Sciences Corporation, and Booz Allen Hamilton. The project was overseen by NSA Deputy Director William B. Black, Jr., an NSA worker who had gone to SAIC, and then been re-hired back to NSA by NSA director Michael Hayden in 2000.[6][7][8] SAIC had also hired a former NSA director to its management; Bobby Inman.[9] SAIC also participated in the concept definition phase of Trailblazer.[10][11]

Redacted version of the DoD Inspector General audit, obtained through the Freedom of Information Act by the Project on Government Oversight and others. [12][5]

The NSA Inspector General issued a report on Trailblazer that “discussed improperly based contract cost increases, non-conformance in the management of the Statement of Work, and excessive labor rates for contractor personnel.” [13]

In 2004 the DoD IG report criticized the program (see the Whistleblowing section below). It said that the “NSA ‘disregarded solutions to urgent national security needs'” and “that TRAILBLAZER was poorly executed and overly expensive …” Several contractors for the project were worried about cooperating with DoD’s audit for fear of “management reprisal.”[5] The Director of NSA “nonconcurred” with several statements in the IG audit, and the report contains a discussion of those disagreements.[14]

In 2005, NSA director Michael Hayden told a Senate hearing that the Trailblazer program was several hundred million dollars over budget and years behind schedule.[15] In 2006 the program was shut down,[3] after having cost billions of US Dollars.[16] Several anonymous NSA sources told Hosenball of Newsweek later on that the project was a “wasteful failure”.[17]

The new project replacing Trailblazer is called Turbulence.[3]

Whistleblowing

According to a 2011 New Yorker article, in the early days of the project several NSA employees met with Diane S Roark, an NSA budget expert on the House Intelligence Committee. They aired their grievances about Trailblazer. In response, NSA director Michael Hayden sent out a memo saying that “individuals, in a session with our congressional overseers, took a position in direct opposition to one that we had corporately decided to follow … Actions contrary to our decisions will have a serious adverse effect on our efforts to transform N.S.A., and I cannot tolerate them.”[3]

In September 2002, several people filed a complaint with the Department of Defense IG’s office regarding problems with Trailblazer: they included Roark (aforementioned), ex-NSA senior analysts Bill Binney, Kirk Wiebe, and Senior Computer Systems Analyst Ed Loomis, who had quit the agency over concerns about its mismanagement of acquisition and allegedly illegal domestic spying.[3][18][19] A major source for the report was NSA senior officer Thomas Andrews Drake. Drake had been complaining to his superiors for some time about problems at the agency, and about the superiority of ThinThread over Trailblazer, for example, at protecting privacy.[19] Drake gave info to DoD during its investigation of the matter.[19] Roark also went to her boss at the House committee, Porter Goss, about problems, but was rebuffed.[20] She also attempted to contact William Renquist, the Supreme Court Chief Justice at the time.[19]

Drake’s own boss, Maureen Baginski, the third-highest officer at NSA, quit partly over concerns about the legality of its behavior.[3]

In 2003, the NSA IG (not the DoD IG)[19] had declared Trailblazer an expensive failure.[21] It had cost more than $1 billion.[8][22][23]

In 2005, the DoD IG produced a report on the result of its investigation of the complaint of Roark and the others in 2002. This report was not released to the public, but it has been described as very negative.[18] Mayer writes that it hastened the closure of Trailblazer, which was at the time in trouble from congress for being over budget.[3]

In November 2005, Drake contacted Siobhan Gorman, a reporter of The Baltimore Sun.[24][17][25] Gorman wrote several articles about problems at the NSA, including articles on Trailblazer. This series got her an award from the Society of Professional Journalists.[17]

In 2005, President George W. Bush ordered the FBI to find whoever had disclosed information about the NSA electronic surveillance program and its disclosure in the New York Times. Eventually, this investigation led to the people who had filed the 2002 DoD IG request, even though they had nothing to do with the New York Times disclosure. In 2007, the houses of Roark, Binney, and Wiebe were raided by armed FBI agents. According to Mayer, Binney claims the FBI pointed guns at his head and that of his wife. Wiebe said it reminded him of the Soviet Union.[3][18] None of these people were ever charged with any crime. Four months later, Drake was raided in November 2007 and his computers and documents were confiscated.

In 2010 Drake was indicted by the U.S. Department of Justice on charges of obstructing justice, providing false information, and violating the Espionage Act of 1917,[17][26][27] part of President Barack Obama’s crackdown on whistleblowers and “leakers”.[24][17][28][18] The government tried to get Roark to testify to a conspiracy, and made similar requests to Drake, offering him a plea bargain. They both refused.[3]

In June 2011, the ten original charges against Drake were dropped, instead he pleaded guilty to a misdemeanor.[5]

Boundless Informant

Boundless Informant is a big data analysis and data visualization system used by the United States National Security Agency (NSA) to give NSA managers summaries of NSA’s world wide data collection activities.[1] It is described in an unclassified, For Official Use Only Frequently Asked Questions (FAQ) memo published by The Guardian.[2] According to a Top Secret heat map display also published by The Guardian and allegedly produced by the Boundless Informant program, almost 3 billion data elements from inside the United States were captured by NSA over a 30-day period ending in March 2013.

Data analyzed by Boundless Informant includes electronic surveillance program records (DNI) and telephone call metadata records (DNR) stored in an NSA data archive called GM-PLACE. It does not include FISA data, according to the FAQ memo. PRISM, a government codename for a collection effort known officially as US-984XN, which was revealed at the same time as Boundless Informant, is one source of DNR data. According to the map, Boundless Informant summarizes data records from 504 separate DNR and DNI collection sources (SIGADs). In the map, countries that are under surveillance are assigned a color from green, representing least coverage to red, most intensive.[3][4]

History

Slide showing that much of the world’s communications flow through the US.

Intelligence gathered by the United States government inside the United States or specifically targeting US citizens is legally required to be gathered in compliance with the Foreign Intelligence Surveillance Act of 1978 (FISA) and under the authority of the Foreign Intelligence Surveillance Court (FISA court).[5][6][7]

NSA global data mining projects have existed for decades, but recent programs of intelligence gathering and analysis that include data gathered from inside the United States such as PRISM were enabled by changes to US surveillance law introduced under President Bush and renewed under President Obama in December 2012.[8]

Boundless Informant was first publicly revealed on June 8, 2013, after classified documents about the program were leaked to The Guardian.[1][9] The newspaper identified its informant, at his request, as Edward Snowden, who worked at the NSA for the defense contractor Booz Allen Hamilton.[10]

Technology

According to published slides, Boundless Informant leverages Free and Open Source Software—and is therefore “available to all NSA developers”—and corporate services hosted in the cloud. The tool uses HDFS, MapReduce, and Cloudbase for data processing.[11]

Legality and FISA Amendments Act of 2008

The FISA Amendments Act (FAA) Section 702 is referenced in PRISM documents detailing the electronic interception, capture and analysis of metadata. Many reports and letters of concern written by members of Congress suggest that this section of FAA in particular is legally and constitutionally problematic, such as by targeting U.S. persons, insofar as “Collections occur in U.S.” as published documents indicate.[12][13][14][15]

The ACLU has asserted the following regarding the FAA: “Regardless of abuses, the problem with the FAA is more fundamental: the statute itself is unconstitutional.”[16]

Senator Rand Paul is introducing new legislation called the Fourth Amendment Restoration Act of 2013 to stop the NSA or other agencies of the United States government from violating the Fourth Amendment to the U.S. Constitution using technology and big data information systems like PRISM and Boundless Informant.[17][18]

ECHELON

ECHELON is a name used in global media and in popular culture to describe a signals intelligence (SIGINT) collection and analysis network operated on behalf of the five signatory states to the UKUSA Security Agreement[1] (Australia, Canada, New Zealand, the United Kingdom, and the United States, referred to by a number of abbreviations, including AUSCANNZUKUS[1] and Five Eyes).[2][3] It has also been described as the only software system which controls the download and dissemination of the intercept of commercial satellite trunk communications.[4]

ECHELON, according to information in the European Parliament document, “On the existence of a global system for the interception of private and commercial communications (ECHELON interception system)” was created to monitor the military and diplomatic communications of the Soviet Union and its Eastern Bloc allies during the Cold War in the early 1960s.[5]

The system has been reported in a number of public sources.[6] Its capabilities and political implications were investigated by a committee of the European Parliament during 2000 and 2001 with a report published in 2001,[5] and by author James Bamford in his books on the National Security Agency of the United States.[4] The European Parliament stated in its report that the term ECHELON is used in a number of contexts, but that the evidence presented indicates that it was the name for a signals intelligence collection system. The report concludes that, on the basis of information presented, ECHELON was capable of interception and content inspection of telephone calls, fax, e-mail and other data traffic globally through the interception of communication bearers including satellite transmission, public switched telephone networks (which once carried most Internet traffic) and microwave links.[5]

Bamford describes the system as the software controlling the collection and distribution of civilian telecommunications traffic conveyed using communication satellites, with the collection being undertaken by ground stations located in the footprint of the downlink leg.

Organization

The UKUSA intelligence community was assessed by the European Parliament (EP) in 2000 to include the signals intelligence agencies of each of the member states:

the Government Communications Headquarters of the United Kingdom,

the National Security Agency of the United States,

the Communications Security Establishment of Canada,

the Defence Signals Directorate of Australia, and

the Government Communications Security Bureau of New Zealand.

the National SIGINT Organisation (NSO) of The Netherlands

The EP report concluded that it seemed likely that ECHELON is a method of sorting captured signal traffic, rather than a comprehensive analysis tool.[5]

Capabilities

The ability to intercept communications depends on the medium used, be it radio, satellite, microwave, cellular or fiber-optic.[5] During World War II and through the 1950s, high frequency (“short wave”) radio was widely used for military and diplomatic communication,[7] and could be intercepted at great distances.[5] The rise of geostationary communications satellites in the 1960s presented new possibilities for intercepting international communications. The report to the European Parliament of 2001 states: “If UKUSA states operate listening stations in the relevant regions of the earth, in principle they can intercept all telephone, fax and data traffic transmitted via such satellites.”[5]

The role of satellites in point-to-point voice and data communications has largely been supplanted by fiber optics; in 2006, 99% of the world’s long-distance voice and data traffic was carried over optical-fiber.[8] The proportion of international communications accounted for by satellite links is said to have decreased substantially over the past few years[when?] in Central Europe to an amount between 0.4% and 5%.[5] Even in less-developed parts of the world, communications satellites are used largely for point-to-multipoint applications, such as video.[9] Thus, the majority of communications can no longer be intercepted by earth stations; they can only be collected by tapping cables and intercepting line-of-sight microwave signals, which is possible only to a limited extent.[5]

One method of interception is to place equipment at locations where fiber optic communications are switched. For the Internet, much of the switching occurs at relatively few sites. There have been reports of one such intercept site, Room 641A, in the United States. In the past[when?] much Internet traffic was routed through the U.S. and the UK, but this has changed; for example, in 2000, 95% of intra-German Internet communications was routed via the DE-CIX Internet exchange point in Frankfurt.[5] A comprehensive worldwide surveillance network is possible only if clandestine intercept sites are installed in the territory of friendly nations, and/or if local authorities cooperate. The report to the European Parliament points out that interception of private communications by foreign intelligence services is not necessarily limited to the U.S. or British foreign intelligence services.[5]

Most reports on ECHELON focus on satellite interception; testimony before the European Parliament indicated that separate but similar UK-US systems are in place to monitor communication through undersea cables, microwave transmissions and other lines.[10]

Controversy

See also: Industrial espionage

Intelligence monitoring of citizens, and their communications, in the area covered by the AUSCANNZUKUS security agreement has caused concern. British journalist Duncan Campbell and New Zealand journalist Nicky Hager asserted in the 1990s that the United States was exploiting ECHELON traffic for industrial espionage, rather than military and diplomatic purposes.[10] Examples alleged by the journalists include the gear-less wind turbine technology designed by the German firm Enercon[5][11] and the speech technology developed by the Belgian firm Lernout & Hauspie.[12] An article in the US newspaper Baltimore Sun reported in 1995 that European aerospace company Airbus lost a $6 billion contract with Saudi Arabia in 1994 after the US National Security Agency reported that Airbus officials had been bribing Saudi officials to secure the contract.[13][14]

In 2001, the Temporary Committee on the ECHELON Interception System recommended to the European Parliament that citizens of member states routinely use cryptography in their communications to protect their privacy, because economic espionage with ECHELON has been conducted by the US intelligence agencies.[5]

Bamford provides an alternative view, highlighting that legislation prohibits the use of intercepted communications for commercial purposes, although he does not elaborate on how intercepted communications are used as part of an all-source intelligence process.

Hardware

According to its website, the U.S. National Security Agency (NSA) is “a high technology organization … on the frontiers of communications and data processing”. In 1999 the Australian Senate Joint Standing Committee on Treaties was told by Professor Desmond Ball that the Pine Gap facility was used as a ground station for a satellite-based interception network. The satellites were said to be large radio dishes between 20 and 100 meters in diameter in geostationary orbits.[citation needed] The original purpose of the network was to monitor the telemetry from 1970s Soviet weapons, air defence radar, communications satellites and ground based microwave communications.[15]

Name

The European Parliament’s Temporary Committee on the ECHELON Interception System stated: “It seems likely, in view of the evidence and the consistent pattern of statements from a very wide range of individuals and organisations, including American sources, that its name is in fact ECHELON, although this is a relatively minor detail.”[5] The U.S. intelligence community uses many code names (see, for example, CIA cryptonym).

Former NSA employee Margaret Newsham claims that she worked on the configuration and installation of software that makes up the ECHELON system while employed at Lockheed Martin, for whom she worked from 1974 to 1984 in Sunnyvale, California, US, and in Menwith Hill, England, UK.[16] At that time, according to Newsham, the code name ECHELON was NSA’s term for the computer network itself. Lockheed called it P415. The software programs were called SILKWORTH and SIRE. A satellite named VORTEX intercepted communications. An image available on the internet of a fragment apparently torn from a job description shows Echelon listed along with several other code names.[17]

Ground stations

The 2001 European Parliamentary (EP) report[5] lists several ground stations as possibly belonging to, or participating in, the ECHELON network. These include:

Likely satellite intercept stations

The following stations are listed in the EP report (p. 54 ff) as likely to have, or to have had, a role in intercepting transmissions from telecommunications satellites:

Hong Kong (since closed)

Australian Defence Satellite Communications Station (Geraldton, Western Australia)

Room 641A

Room 641A is a telecommunication interception facility operated by AT&T for the U.S. National Security Agency that commenced operations in 2003 and was exposed in 2006.[1][2]

Description

Room 641A is located in the SBC Communications building at 611 Folsom Street, San Francisco, three floors of which were occupied by AT&T before SBC purchased AT&T.[1] The room was referred to in internal AT&T documents as the SG3 [Study Group 3] Secure Room. It is fed by fiber optic lines from beam splitters installed in fiber optic trunks carrying Internet backbone traffic[3] and, as analyzed by J. Scott Marcus, a former CTO for GTE and a former adviser to the FCC, who has access to all Internet traffic that passes through the building, and therefore “the capability to enable surveillance and analysis of internet content on a massive scale, including both overseas and purely domestic traffic.”[4] Former director of the NSA’s World Geopolitical and Military Analysis Reporting Group, William Binney, has estimated that 10 to 20 such facilities have been installed throughout the United States.[2]

The room measures about 24 by 48 feet (7.3 by 15 m) and contains several racks of equipment, including a Narus STA 6400, a device designed to intercept and analyze Internet communications at very high speeds.[1]

The very existence of the room was revealed by a former AT&T technician, Mark Klein, and was the subject of a 2006 class action lawsuit by the Electronic Frontier Foundation against AT&T.[5] Klein claims he was told that similar black rooms are operated at other facilities around the country.

Room 641A and the controversies surrounding it were subjects of an episode of Frontline, the current affairs documentary program on PBS. It was originally broadcast on May 15, 2007. It was also featured on PBS’s NOW on March 14, 2008. The room was also covered in the PBS Nova episode “The Spy Factory”.

Lawsuit

Basic diagram of how the alleged wiretapping was accomplished. From EFF court filings[4]

The Electronic Frontier Foundation (EFF) filed a class-action lawsuit against AT&T on January 31, 2006, accusing the telecommunication company of violating the law and the privacy of its customers by collaborating with the National Security Agency (NSA) in a massive, illegal program to wiretap and data-mine Americans’ communications. On July 20, 2006, a federal judge denied the government’s and AT&T’s motions to dismiss the case, chiefly on the ground of the States Secrets Privilege, allowing the lawsuit to go forward. On August 15, 2007, the case was heard by the Ninth Circuit Court of Appeals and was dismissed on December 29, 2011 based on a retroactive grant of immunity by Congress for telecommunications companies that cooperated with the government. The U.S. Supreme Court declined to hear the case.[6] A different case by the EFF was filed on September 18, 2008, titled Jewel v. NSA.

PRISM: A clandestine national security electronic surveillance program operated by the United States National Security Agency (NSA) which can target customers of participating corporations outside or inside the United States

Main Core: A personal and financial database storing information of millions of U.S. citizens believed to be threats to national security.[7] The data mostly comes from the NSA, FBI, CIA, as well as other government sources.[7]

Story 1: NSA Metadata To Be Held By Telephone Companies — Great Distraction — Still Collecting and Intercepting All Americans Telephone Calls and All Information Transmitted Over The Internet and Telephone Exchanges — Stop Deceiving The American People Mr. President — Videos

Obama: NSA Proposal Satisfies Public Concerns

Obama announces overhaul of NSA metadata collection

NSA – Changes To Metadata Program – Special Report All Star

President Obama Names Michael Rogers As New Head Of The NSA

Background Articles and Videos

Through a PRISM, Darkly – Everything we know about NSA spying [30c3]

Published on Dec 30, 2013

Through a PRISM, Darkly
Everything we know about NSA spying

From Stellar Wind to PRISM, Boundless Informant to EvilOlive, the NSA spying programs are shrouded in secrecy and rubber-stamped by secret opinions from a court that meets in a faraday cage. The Electronic Frontier Foundation’s Kurt Opsahl explains the known facts about how the programs operate and the laws and regulations the U.S. government asserts allows the NSA to spy on you.
The Electronic Frontier Foundation, a non-profit civil society organization, has been litigating against the NSA spying program for the better part of a decade. EFF has collected and reviewed dozens of documents, from the original NY Times stories in 2005 and the first AT&T whistleblower in 2006, through the latest documents released in the Guardian or obtained through EFF’s Freedom of Information (government transparency) litigation. EFF attorney Kurt Opsahl’s lecture will describe how the NSA spying program works, the underlying technologies, the targeting procedures (how they decide who to focus on), the minimization procedures (how they decide which information to discard), and help you makes sense of the many code names and acronyms in the news. He will also discuss the legal and policy ramifications that have become part of the public debate following the recent disclosures, and what you can do about it. After summarizing the programs, technologies, and legal/policy framework in the lecture, the audience can ask questions.

Glenn Becks “SURVEILLANCE STATE”

Inside the NSA

Ed Snowden, NSA, and Fairy Tales

AT&T Spying On Internet Traffic

For years the National Securities Agency, has been spying on each & every keystroke. The national headquarters of AT&T is in Missouri, where ex-employees describe a secret room. The program is called “Splitter Cut-In & Test Procedure.”

NSA Whistle-Blower Tells All – Op-Docs: The Program

The filmmaker Laura Poitras profiles William Binney, a 32-year veteran of the National Security Agency who helped design a top-secret program he says is broadly collecting Americans’ personal data.

NSA Whistleblower: Everyone in US under virtual surveillance, all info stored, no matter the post

He told you so: Bill Binney talks NSA leaks

William Benny – The Government is Profiling You (The NSA is Spying on You)

‘After 9/11 NSA had secret deal with White House’

The story of Whistleblower Thomas Drake

Whistleblowers, Part Two: Thomas Drake

NSA Whistleblower Thomas Drake speaks at National Press Club – March 15, 2013

Meet Edward Snowden: NSA PRISM Whistleblower

The Truth About Edward Snowden

N.S.A. Spying: Why Does It Matter?

Inside The NSA~Americas Cyber Secrets

NSA Whistleblower Exposes Obama’s Dragnet

AT&T whistleblower against immunity for Bush spy program-1/2

AT&T Whistleblower Urges Against Immunity for Telecoms in Bush Spy Program

The Senate is expected to vote on a controversial measure to amend the Foreign Intelligence Surveillance Act tomorrow. The legislation would rewrite the nation’s surveillance laws and authorize the National Security Agency’s secret program of warrantless wiretapping. We speak with Mark Klein, a technician with AT&T for over twenty-two years. In 2006 Klein leaked internal AT&T documents that revealed the company had set up a secret room in its San Francisco office to give the National Security Agency access to its fiber optic internet cables.

AT&T whistleblower against immunity for Bush spy program-2/2

Enemy Of The State 1998 (1080p) (Full movie)

Background Articles and Videos

Stellar Wind

Stellar Wind was the open secret code name for four surveillance programs by the United States National Security Agency (NSA) during the presidency of George W. Bush and revealed by Thomas Tamm to The New York Times reporters James Risen and Eric Lichtblau.[1] The operation was approved by President George W. Bush shortly after the September 11 attacks in 2001.[2] Stellar Wind was succeeded during the presidency of Barack Obama by four major lines of intelligence collection in the territorial United States, together capable of spanning the full range of modern telecommunications.[3]

The program’s activities involved data mining of a large database of the communications of American citizens, including e-mail communications, phone conversations, financial transactions, and Internet activity.[1] William Binney, a retired Technical Leader with the NSA, discussed some of the architectural and operational elements of the program at the 2012 Chaos Communication Congress.[4]

There were internal disputes within the Justice Department about the legality of the program, because data are collected for large numbers of people, not just the subjects of Foreign Intelligence Surveillance Act (FISA) warrants.[4]

During the Bush Administration, the Stellar Wind cases were referred to by FBI agents as “pizza cases” because many seemingly suspicious cases turned out to be food takeout orders. According to Mueller, approximately 99 percent of the cases led nowhere, but “it’s that other 1% that we’ve got to be concerned about”.[2] One of the known uses of these data were the creation of suspicious activity reports, or “SARS”, about people suspected of terrorist activities. It was one of these reports that revealed former New York governor Eliot Spitzer’s use of prostitutes, even though he was not suspected of terrorist activities.[1]

In March 2012 Wired magazine published “The NSA Is Building the Country’s Biggest Spy Center (Watch What You Say)” talking about a vast new NSA facility in Utah and says “For the first time, a former NSA official has gone on the record to describe the program, codenamed Stellar Wind, in detail,” naming the official William Binney, a former NSA code breaker. Binney went on to say that the NSA had highly secured rooms that tap into major switches, and satellite communications at both AT&T and Verizon.[5] The article suggested that the otherwise dispatched Stellar Wind is actually an active program.

PRISM

PRISM is a clandestine national security electronic surveillance program operated by the United States National Security Agency (NSA) since 2007.[1][2][3][Notes 1]PRISM is a government codename for a data collection effort known officially as US-984XN.[8][9] It is operated under the supervision of the United States Foreign Intelligence Surveillance Court pursuant to the Foreign Intelligence Surveillance Act (FISA).[10] The existence of the program was leaked by NSA contractor Edward Snowden and published by The Guardian and The Washington Post on June 6, 2013.

A document included in the leak indicated that the PRISM SIGAD was “the number one source of raw intelligence used for NSA analytic reports.”[11] The President’s Daily Brief, an all-source intelligence product, cited PRISM data as a source in 1,477 items in 2012.[12] The leaked information came to light one day after the revelation that the United States Foreign Intelligence Surveillance Court had been requiring the telecommunications company Verizon to turn over to the NSA logs tracking all of its customers’ telephone calls on an ongoing daily basis.[13][14]

According to the Director of National Intelligence James Clapper, PRISM cannot be used to intentionally target any Americans or anyone in the United States. Clapper said a special court, Congress, and the executive branch oversee the program and extensive procedures ensure the acquisition, retention, and dissemination of data accidentally collected about Americans is kept to a minimum.[15] Clapper issued a statement and “fact sheet”[16] to correct what he characterized as “significant misimpressions” in articles by The Washington Post and The Guardian newspapers.[17]

History

Slide showing that much of the world’s communications flow through the US

Details of information collected via PRISM

PRISM is a “Special Source Operation” in the tradition of NSA’s intelligence alliances with as many as 100 trusted U.S. companies since the 1970s.[1] A prior program, the Terrorist Surveillance Program, was implemented in the wake of the September 11 attacks under the George W. Bush Administration but was widely criticized and had its legality questioned, because it was conducted without approval of the Foreign Intelligence Surveillance Court (FISC).[18][19][20][21] PRISM was authorized by an order of the FISC.[11] Its creation was enabled by the Protect America Act of 2007 under President Bush and the FISA Amendments Act of 2008, which legally immunized private companies that cooperated voluntarily with US intelligence collection and was renewed by Congress under President Obama in 2012 for five years until December 2017.[2][22] According to The Register, the FISA Amendments Act of 2008 “specifically authorizes intelligence agencies to monitor the phone, email, and other communications of U.S. citizens for up to a week without obtaining a warrant” when one of the parties is outside the U.S.[22]

PRISM was first publicly revealed on June 6, 2013, after classified documents about the program were leaked to The Washington Post and The Guardian by American Edward Snowden.[2][1] The leaked documents included 41 PowerPoint slides, four of which were published in news articles.[1][2] The documents identified several technology companies as participants in the PRISM program, including (date of joining PRISM in parentheses) Microsoft (2007), Yahoo! (2008), Google (2009), Facebook (2009), Paltalk (2009), YouTube (2010), AOL (2011), Skype (2011), and Apple (2012).[23] The speaker’s notes in the briefing document reviewed by The Washington Post indicated that “98 percent of PRISM production is based on Yahoo, Google and Microsoft.”[1]

The slide presentation stated that much of the world’s electronic communications pass through the United States, because electronic communications data tend to follow the least expensive route rather than the most physically direct route, and the bulk of the world’s internet infrastructure is based in the United States.[11] The presentation noted that these facts provide United States intelligence analysts with opportunities for intercepting the communications of foreign targets as their electronic data pass into or through the United States.[2][11]

According to The Washington Post, the intelligence analysts search PRISM data using terms intended to identify suspicious communications of targets whom the analysts suspect with at least 51 percent confidence to not be United States citizens, but in the process, communication data of some United States citizens are also collected unintentionally.[1] Training materials for analysts tell them that while they should periodically report such accidental collection of non-foreign United States data, “it’s nothing to worry about.”[1]

Response from companies

The original Washington Post and Guardian articles reporting on PRISM noted that one of the leaked briefing documents said PRISM involves collection of data “directly from the servers” of several major internet services providers.[2][1]

Initial Public Statements

Corporate executives of several companies identified in the leaked documents told The Guardian that they had no knowledge of the PRISM program in particular and also denied making information available to the government on the scale alleged by news reports.[2][24] Statements of several of the companies named in the leaked documents were reported by TechCrunch and The Washington Post as follows:[25][26]

Slide listing companies and the date that PRISM collection began

Microsoft: “We provide customer data only when we receive a legally binding order or subpoena to do so, and never on a voluntary basis. In addition we only ever comply with orders for requests about specific accounts or identifiers. If the government has a broader voluntary national security program to gather customer data we don’t participate in it.”[25]

Yahoo!: “Yahoo! takes users’ privacy very seriously. We do not provide the government with direct access to our servers, systems, or network.”[25] “Of the hundreds of millions of users we serve, an infinitesimal percentage will ever be the subject of a government data collection directive.”[26]

Facebook: “We do not provide any government organization with direct access to Facebook servers. When Facebook is asked for data or information about specific individuals, we carefully scrutinize any such request for compliance with all applicable laws, and provide information only to the extent required by law.”[25]

Google: “Google cares deeply about the security of our users’ data. We disclose user data to government in accordance with the law, and we review all such requests carefully. From time to time, people allege that we have created a government ‘back door’ into our systems, but Google does not have a backdoor for the government to access private user data.”[25] “[A]ny suggestion that Google is disclosing information about our users’ Internet activity on such a scale is completely false.”[26]

Apple: “We have never heard of PRISM. We do not provide any government agency with direct access to our servers, and any government agency requesting customer data must get a court order.”[27]

Dropbox: “We’ve seen reports that Dropbox might be asked to participate in a government program called PRISM. We are not part of any such program and remain committed to protecting our users’ privacy.”[25]

In response to the technology companies’ denials of the NSA being able to directly access the companies’ servers, The New York Times reported that sources had stated the NSA was gathering the surveillance data from the companies using other technical means in response to court orders for specific sets of data.[13]The Washington Post suggested, “It is possible that the conflict between the PRISM slides and the company spokesmen is the result of imprecision on the part of the NSA author. In another classified report obtained by The Post, the arrangement is described as allowing ‘collection managers [to send] content tasking instructions directly to equipment installed at company-controlled locations,’ rather than directly to company servers.”[1] “[I]n context, ‘direct’ is more likely to mean that the NSA is receiving data sent to them deliberately by the tech companies, as opposed to intercepting communications as they’re transmitted to some other destination.[26]

“If these companies received an order under the FISA amendments act, they are forbidden by law from disclosing having received the order and disclosing any information about the order at all,” Mark Rumold, staff attorney at the Electronic Frontier Foundation, told ABC News.[28]

Slide showing two different sources of NSA data collection. The first source the fiber optic cables of the internet handled by the Upstream program and the second source the servers of major internet companies handled by PRISM.[29]

On May 28, 2013, Google was ordered by United States District Court Judge Susan Illston to comply with a National Security Letter issued by the FBI to provide user data without a warrant.[30] Kurt Opsahl, a senior staff attorney at the Electronic Frontier Foundation, in an interview with VentureBeat said, “I certainly appreciate that Google put out a transparency report, but it appears that the transparency didn’t include this. I wouldn’t be surprised if they were subject to a gag order.”[31]

The New York Times reported on June 7, 2013, that “Twitter declined to make it easier for the government. But other companies were more compliant, according to people briefed on the negotiations.”[32] The other companies held discussions with national security personnel on how to make data available more efficiently and securely.[32] In some cases, these companies made modifications to their systems in support of the intelligence collection effort.[32] The dialogues have continued in recent months, as General Martin Dempsey, the chairman of the Joint Chiefs of Staff, has met with executives including those at Facebook, Microsoft, Google and Intel.[32] These details on the discussions provide insight into the disparity between initial descriptions of the government program including a training slide which states “Collection directly from the servers”[29] and the companies’ denials.[32]

While providing data in response to a legitimate FISA request approved by FISC is a legal requirement, modifying systems to make it easier for the government to collect the data is not. This is why Twitter could legally decline to provide an enhanced mechanism for data transmission.[32] Other than Twitter, the companies were effectively asked to construct a locked mailbox and provide the key to the government, people briefed on the negotiations said.[32] Facebook, for instance, built such a system for requesting and sharing the information.[32] Google does not provide a lockbox system, but instead transmits required data by hand delivery or secure FTP.[33]

Post-PRISM Transparency Reports

In response to the publicity surrounding media reports of data-sharing, several companies requested permission to reveal more public information about the nature and scope of information provided in response to National Security requests.

On June 14, 2013, Facebook reported that the U.S. Government had authorized the communication of “about these numbers in aggregate, and as a range.” In a press release posted to their web site, Facebook reported, “For the six months ending December 31, 2012, the total number of user-data requests Facebook received from any and all government entities in the U.S. (including local, state, and federal, and including criminal and national security-related requests) – was between 9,000 and 10,000.” Facebook further reported that the requests impacted “between 18,000 and 19,000” user accounts, a “tiny fraction of one percent” of more than 1.1 billion active user accounts.[34]

Microsoft reported that for the same period, it received “between 6,000 and 7,000 criminal and national security warrants, subpoenas and orders affecting between 31,000 and 32,000 consumer accounts from U.S. governmental entities (including local, state and federal)” which impacted “a tiny fraction of Microsoft’s global customer base”.[35]

Google issued a statement criticizing the requirement that data be reported in aggregated form, stating that lumping national security requests with criminal request data would be “a step backwards” from its previous, more detailed practices on its site transparency report. The company said that it would continue to seek government permission to publish the number and extent of FISA requests.[36]

Response from United States government

Executive branch

Shortly after publication of the reports by The Guardian and The Washington Post, the United States Director of National Intelligence, James Clapper, on June 7 released a statement confirming that for nearly six years the government of the United States had been using large internet services companies such as Google and Facebook to collect information on foreigners outside the United States as a defense against national security threats.[13] The statement read in part, “The Guardian and The Washington Post articles refer to collection of communications pursuant to Section 702 of the Foreign Intelligence Surveillance Act. They contain numerous inaccuracies.”[37] He went on to say, “Section 702 is a provision of FISA that is designed to facilitate the acquisition of foreign intelligence information concerning non-U.S. persons located outside the United States. It cannot be used to intentionally target any U.S. citizen, any other U.S. person, or anyone located within the United States.”[37] Clapper concluded his statement by stating “The unauthorized disclosure of information about this important and entirely legal program is reprehensible and risks important protections for the security of Americans.”[37] On March 12, 2013, Clapper had told the United States Senate Select Committee on Intelligence that the NSA does “not wittingly” collect any type of data on millions or hundreds of millions of Americans.[38] In an NBC News interview, Clapper said he answered Senator Wyden’s question in the “least untruthful manner by saying no”.[39]

Clapper also stated that “the NSA collects the phone data in broad swaths, because collecting it (in) a narrow fashion would make it harder to identify terrorism-related communications. The information collected lets the government, over time, make connections about terrorist activities. The program doesn’t let the U.S. listen to people’s calls, but only includes information like call length and telephone numbers dialed.”[15]

On June 8, 2013, Clapper said “the surveillance activities published in The Guardian and The Washington Post are lawful and conducted under authorities widely known and discussed, and fully debated and authorized by Congress.”[40][10] The fact sheet described PRISM as “an internal government computer system used to facilitate the government’s statutorily authorized collection of foreign intelligence information from electronic communication service providers under court supervision, as authorized by Section 702 of the Foreign Intelligence Surveillance Act (FISA) (50 U.S.C. § 1881a).”[10]

The National Intelligence fact sheet further stated that “the United States Government does not unilaterally obtain information from the servers of U.S. electronic communication service providers. All such information is obtained with FISA Court approval and with the knowledge of the provider based upon a written directive from the Attorney General and the Director of National Intelligence.” It said that the Attorney General provides FISA Court rulings and semi-annual reports about PRISM activities to Congress, “provid[ing] an unprecedented degree of accountability and transparency.”[10]

The President of the United States, Barack Obama, said on June 7 “What you’ve got is two programs that were originally authorized by Congress, have been repeatedly authorized by Congress. Bipartisan majorities have approved them. Congress is continually briefed on how these are conducted. There are a whole range of safeguards involved. And federal judges are overseeing the entire program throughout.”[41] He also said, “You can’t have 100 percent security and then also have 100 percent privacy and zero inconvenience. You know, we’re going to have to make some choices as a society.”[41]

In separate statements, senior (not mentioned by name in source) Obama administration officials said that Congress had been briefed 13 times on the programs since 2009.[42]

Legislative branch

In contrast to their swift and forceful reactions the previous day to allegations that the government had been conducting surveillance of United States citizens’ telephone records, Congressional leaders initially had little to say about the PRISM program the day after leaked information about the program was published. Several lawmakers declined to discuss PRISM, citing its top-secret classification,[43] and others said that they had not been aware of the program.[44] After statements had been released by the President and the Director of National Intelligence, some lawmakers began to comment:

Senator John McCain (R-AZ)

June 9 “We passed the Patriot Act. We passed specific provisions of the act that allowed for this program to take place, to be enacted in operation,”[45]

June 9 “These programs are within the law”, “part of our obligation is keeping Americans safe”, “Human intelligence isn’t going to do it”.[46]

June 9 “Here’s the rub: the instances where this has produced good — has disrupted plots, prevented terrorist attacks, is all classified, that’s what’s so hard about this.”[47]

June 11 “It went fine…we asked him[ Keith Alexander ] to declassify things because it would be helpful (for people and lawmakers to better understand the intelligence programs).” “I’ve just got to see if the information gets declassified. I’m sure people will find it very interesting.”[48]

Senator Susan Collins (R-ME), member of Senate Intelligence Committee and past member of Homeland Security Committee

June 11 “I had, along with Joe Lieberman, a monthly threat briefing, but I did not have access to this highly compartmentalized information” and “How can you ask when you don’t know the program exists?”[49]

Representative John Boehner (R-OH), Speaker of the House of Representatives

June 9, “This is well beyond what the Patriot Act allows.”[51] “President Obama’s claim that ‘this is the most transparent administration in history’ has once again proven false. In fact, it appears that no administration has ever peered more closely or intimately into the lives of innocent Americans.”[51]

Representative Mike Rogers (R-MI), a Chairman of the Permanent Select Committee on Intelligence.

June 9 “One of the things that we’re charged with is keeping America safe and keeping our civil liberties and privacy intact. I think we have done both in this particular case,”[46]

June 9 “Within the last few years this program was used to stop a program, excuse me, to stop a terrorist attack in the United States we know that. It’s, it’s, it’s important, it fills in a little seam that we have and it’s used to make sure that there is not an international nexus to any terrorism event that they may believe is ongoing in the United States. So in that regard it is a very valuable thing,”[52]

Senator Mark Udall (D-CO)

June 9 “I don’t think the American public knows the extent or knew the extent to which they were being surveilled and their data was being collected.” “I think we ought to reopen the Patriot Act and put some limits on the amount of data that the National Security (Agency) is collecting,” “It ought to remain sacred, and there’s got to be a balance here. That is what I’m aiming for. Let’s have the debate, let’s be transparent, let’s open this up”.[46]

Representative Todd Rokita (R-IN)

June 10 “We have no idea when they [ FISA ] meet, we have no idea what their judgments are”,[53]

Senator Rand Paul (R-KY)

June 6 “When the Senate rushed through a last-minute extension of the FISA Amendments Act late last year, I insisted on a vote on my amendment (SA 3436) to require stronger protections on business records and prohibiting the kind of data-mining this case has revealed. Just last month, I introduced S.1037, the Fourth Amendment Preservation and Protection Act,”[54]

June 9 “I’m going to be seeing if I can challenge this at the Supreme Court level. I’m going to be asking the Internet providers and all of the phone companies: ask your customers to join me in a class-action lawsuit.”[45]

Representative Luis Gutierrez (D-IL)

June 9 “We will be receiving secret briefings and we will be asking, I know I’m going to be asking to get more information. I want to make sure that what they’re doing is harvesting information that is necessary to keep us safe and not simply going into everybody’s private telephone conversations and Facebook and communications. I mean one of the, you know the terrorists win when you debilitate freedom of expression and privacy.”[52]

Judicial branch

The Foreign Intelligence Surveillance Court (FISC) has not acknowledged, denied or confirmed any involvement in the PRISM program at this time. It has not issued any press statement or release relating to the current situation and uncertainty.

Applicable law and practice

On June 8, 2013, the Director of National Intelligence issued a fact sheet stating that PRISM “is not an undisclosed collection or data mining program”, but rather computer software used to facilitate the collection of foreign intelligence information “under court supervision, as authorized by Section 702 of the Foreign Intelligence Surveillance Act (FISA) (50 U.S.C. § 1881a).”[10] Section 702 provides that “the Attorney General [A.G.] and the Director of National Intelligence [DNI] may authorize jointly, for a period of up to 1 year from the effective date of the authorization, the targeting of persons reasonably believed to be located outside the United States to acquire foreign intelligence information.”[55] In order to authorize the targeting, the A.G. and DNI need to get an order from the Foreign Intelligence Surveillance Court (FISC) pursuant to Section 702 or certify that “intelligence important to the national security of the United States may be lost or not timely acquired and time does not permit the issuance of an order.”[55] When asking for an order, the A.G. and DNI must certify to FISC that “a significant purpose of the acquisition is to obtain foreign intelligence information.”[55] They do not need to specify which facilities or property that the targeting will be directed at.[55]

After getting a FISC order or determining that there are emergency circumstances, the A.G. and DNI can direct an electronic communication service provider to give them access to information or facilities to carry out the targeting and keep the targeting secret.[55] The provider then has the option to: (1) comply with the directive; (2) reject it; or (3) challenge it to FISC.

If the provider complies with the directive, it is released from liability to its users for providing the information and reimbursed for the cost of providing it.[55]

If the provider rejects the directive, the A.G. may request an order from FISC to enforce it.[55] A provider that fails to comply with FISC’s order can be punished with contempt of court.[55]

Finally, a provider can petition FISC to reject the directive.[55] In case FISC denies the petition and orders the provider to comply with the directive, the provider risks contempt of court if it refuses to comply with FISC’s order.[55] The provider can appeal FISC’s denial to the Foreign Intelligence Surveillance Court of Review and then appeal the Court of Review’s decision to the Supreme Court by a writ of certiorari for review under seal.[55]

The Senate Select Committee on Intelligence and the FISA Courts had been put in place to oversee intelligence operations in the period after the death of J. Edgar Hoover. Beverly Gage of Slate said, “When they were created, these new mechanisms were supposed to stop the kinds of abuses that men like Hoover had engineered. Instead, it now looks as if they have come to function as rubber stamps for the expansive ambitions of the intelligence community. J. Edgar Hoover no longer rules Washington, but it turns out we didn’t need him anyway.”[56]

Involvement of other countries

Australia

The Australian government has said it will investigate the impact of the PRISM program and the use of the Pine Gap surveillance facility on the privacy of Australian citizens.[57]

Canada

Canada’s national cryptologic agency, the Communications Security Establishment, said that commenting on PRISM “would undermine CSE’s ability to carry out its mandate”. Privacy Commissioner Jennifer Stoddart lamented Canada’s standards when it comes to protecting personal online privacy stating “We have fallen too far behind,” Stoddart wrote in her report. “While other nations’ data protection authorities have the legal power to make binding orders, levy hefty fines and take meaningful action in the event of serious data breaches, we are restricted to a ‘soft’ approach: persuasion, encouragement and, at the most, the potential to publish the names of transgressors in the public interest.” And, “when push comes to shove,” Stoddart wrote, “short of a costly and time-consuming court battle, we have no power to enforce our recommendations.”[58]

Germany

Germany did not receive any raw PRISM data, according to a Reuters report.[59]

Israel

Israeli newspaper Calcalist discussed[60] the Business Insider article[61] about the possible involvement of technologies from two secretive Israeli companies in the PRISM program – Verint Systems and Narus.

New Zealand

In New Zealand, University of Otago information science Associate Professor Hank Wolfe said that “under what was unofficially known as the Five Eyes Alliance, New Zealand and other governments, including the United States, Australia, Canada, and Britain, dealt with internal spying by saying they didn’t do it. But they have all the partners doing it for them and then they share all the information.”[62]

United Kingdom

In the United Kingdom, Government Communications Headquarters (GCHQ) has had access to the PRISM program on or before June 2010 and wrote 197 reports with it in 2012 alone. PRISM may have allowed GCHQ to circumvent the formal legal process required to seek personal material.[63][64]

Domestic response

The neutrality of this section is disputed. Please do not remove this message until the dispute is resolved. (June 2013)

The New York Times editorial board charged that the Obama administration “has now lost all credibility on this issue,”[65] and lamented that “for years, members of Congress ignored evidence that domestic intelligence-gathering had grown beyond their control, and, even now, few seem disturbed to learn that every detail about the public’s calling and texting habits now reside in a N.S.A. database.”[66]

Republican and former member of Congress Ron Paul said, “We should be thankful for individuals like Edward Snowden and Glenn Greenwald who see injustice being carried out by their own government and speak out, despite the risk…. They have done a great service to the American people by exposing the truth about what our government is doing in secret.”[67] Paul denounced the government’s secret surveillance program: “The government does not need to know more about what we are doing…. We need to know more about what the government is doing.”[67] He called Congress “derelict in giving that much power to the government,” and said that had he been elected president, he would have ordered searches only when there was probable cause of a crime having been committed, which he said was not how the PRISM program was being operated.[68]

In response to Obama administration arguments that it could stop terrorism in the cases of Najibullah Zazi and David Headley, Ed Pilkington and Nicholas Watt of The Guardian said in regards to the role of PRISM and Boundless Informant interviews with parties involved in the Zazi scheme and court documents lodged in the United States and the United Kingdom indicated that “conventional” surveillance methods such as “old-fashioned tip-offs” of the British intelligence services initiated the investigation into the Zazi case.[69] An anonymous former CIA agent said that in regards to the Headley case, “That’s nonsense. It played no role at all in the Headley case. That’s not the way it happened at all.”[69] Pilkington and Watt concluded that the data-mining programs “played a relatively minor role in the interception of the two plots.”[69] Michael Daly of The Daily Beast stated that even though Tamerlan Tsarnaev had visited Inspire and even though Russian intelligence officials alerted U.S. intelligence officials about Tsarnaev, PRISM did not prevent him from carrying out the Boston bombings, and that the initial evidence implicating him came from his brother Dzhokhar Tsarnaev and not from federal intelligence. In addition Daly pointed to the fact that Faisal Shahzad visited Inspire but that federal authorities did not stop his attempted terrorist plot. Daly concluded “The problem is not just what the National Security Agency is gathering at the risk of our privacy but what it is apparently unable to monitor at the risk of our safety.”[70] In addition, political commentator Bill O’Reilly criticized the government, saying that PRISM did not stop the Boston bombings.[71]

In a blog post, David Simon, the creator of The Wire, compared the NSA’s programs, including PRISM, to a 1980s effort by the City of Baltimore to add dialed number recorders to all pay phones to know which individuals were being called by the callers;[72] the city believed that drug traffickers were using pay phones and pagers, and a municipal judge allowed the city to place the recorders. The placement of the dialers formed the basis of the show’s first season. Simon argued that the media attention regarding the NSA programs is a “faux scandal.”[72][73] George Takei, an actor who had experienced Japanese American internment, said that due to his memories of the internment, he felt concern towards the NSA surveillance programs that had been revealed.[74]

The Electronic Frontier Foundation (EFF), an international non-profit digital-rights group based in the U.S., is hosting a tool, by which an American resident can write to their government representatives regarding their opposition to mass spying.[75]

On June 11, 2013, the American Civil Liberties Union filed a lawsuit against the NSA citing that PRISM “violates Americans’ constitutional rights of free speech, association, and privacy”.[76]

International response

Reactions of Internet users in China were mixed between viewing a loss of freedom worldwide and seeing state surveillance coming out of secrecy. The story broke just before US President Barack Obama and Chinese President Xi Jinping met in California.[77][78] When asked about NSA hacking China, the spokeswoman of Ministry of Foreign Affairs of the People’s Republic of China said “China strongly advocates cybersecurity”.[79] The party-owned newspaper Liberation Daily described this surveillance like Nineteen Eighty-Four-style.[80] Hong Kong legislators Gary Fan and Claudia Mo wrote a letter to Obama, stating “the revelations of blanket surveillance of global communications by the world’s leading democracy have damaged the image of the U.S. among freedom-loving peoples around the world.”[81]

Sophie in ‘t Veld, a Dutch Member of the European Parliament, called PRISM “a violation of EU laws”.[82]

Protests at Checkpoint Charlie in Berlin

The German Federal Commissioner for Data Protection and Freedom of Information, Peter Schaar, condemned the program as “monstrous”.[83] He further added that White House claims do “not reassure me at all” and that “given the large number of German users of Google, Facebook, Apple or Microsoft services, I expect the German government […] is committed to clarification and limitation of surveillance.” Steffen Seibert, press secretary of the Chancellor’s office, announced that Angela Merkel will put these issues on the agenda of the talks with Barack Obama during his pending visit in Berlin.[84]

The Italian president of the Guarantor for the protection of personal data, Antonello Soro, said that the surveillance dragnet “would not be legal in Italy” and would be “contrary to the principles of our legislation and would represent a very serious violation”.[85]

William Hague, the foreign secretary of the United Kingdom, dismissed accusations that British security agencies had been circumventing British law by using information gathered on British citizens by Prism[86] saying, “Any data obtained by us from the United States involving UK nationals is subject to proper UK statutory controls and safeguards.”[86] David Cameron said Britain’s spy agencies that received data collected from PRISM acted within the law: “I’m satisfied that we have intelligence agencies that do a fantastically important job for this country to keep us safe, and they operate within the law.”[86][87] Malcolm Rifkind, the chairman of parliament’s Intelligence and Security Committee, said that if the British intelligence agencies were seeking to know the content of emails about people living in the UK, then they actually have to get lawful authority.[87] The UK’s Information Commissioner’s Office was more cautious, saying it would investigate PRISM alongside other European data agencies: “There are real issues about the extent to which U.S. law agencies can access personal data of UK and other European citizens. Aspects of U.S. law under which companies can be compelled to provide information to U.S. agencies potentially conflict with European data protection law, including the UK’s own Data Protection Act. The ICO has raised this with its European counterparts, and the issue is being considered by the European Commission, who are in discussions with the U.S. Government.”[82]

Ai Weiwei, a Chinese dissident, said “Even though we know governments do all kinds of things I was shocked by the information about the US surveillance operation, Prism. To me, it’s abusively using government powers to interfere in individuals’ privacy. This is an important moment for international society to reconsider and protect individual rights.”[88]

Kim Dotcom, a German-Finnish Internet entrepreneur who owned Megaupload, which was closed by the U.S. federal government, said “We should heed warnings from Snowden because the prospect of an Orwellian society outweighs whatever security benefits we derive from Prism or Five Eyes.”[89] The Hong Kong law firm representing Dotcom expressed a fear that the communication between Dotcom and the firm had been compromised by U.S. intelligence programs.[90]

Russia has offered to consider an asylum request from Edward Snowden.[91]

Taliban spokesperson Zabiullah Mujahid said “We knew about their past efforts to trace our system. We have used our technical resources to foil their efforts and have been able to stop them from succeeding so far.”[92][93]

Related government Internet surveillance programs

A parallel program, code-named BLARNEY, gathers up metadata as it streams past choke points along the backbone of the Internet. BLARNEY’s summary, set down in the slides alongside a cartoon insignia of a shamrock and a leprechaun hat, describes it as “an ongoing collection program that leverages IC [intelligence community] and commercial partnerships to gain access and exploit foreign intelligence obtained from global networks.”[94]

A related program, a big data visualization system based on cloud computing and free and open-source software (FOSS) technology known as “Boundless Informant”, was disclosed in documents leaked to The Guardian and reported on June 8, 2013. A leaked, top secret map allegedly produced by Boundless Informant revealed the extent of NSA surveillance in the U.S.[95]

ThinThread

ThinThread is the name of a project that the United States National Security Agency (NSA) pursued during the 1990s, according to a May 17, 2006 article in The Baltimore Sun.[1] The program involved wiretapping and sophisticated analysis of the resulting data, but according to the article, the program was discontinued three weeks before the September 11, 2001 attacks due to the changes in priorities and the consolidation of U.S. intelligence authority.[2] The “change in priority” consisted of the decision made by the director of NSA General Michael V. Hayden to go with a concept called Trailblazer, despite the fact that ThinThread was a working prototype that protected the privacy of U.S. citizens.

ThinThread was dismissed and replaced by the Trailblazer Project, which lacked the privacy protections.[3] A consortium led by Science Applications International Corporation was awarded a $280 million contract to develop Trailblazer in 2002.[4]

Trailblazer

Trailblazer was a United States National Security Agency (NSA) program intended to develop a capability to analyze data carried on communications networks like the Internet. It was intended to track entities using communication methods such as cell phones and e-mail.[1][2] It ran over budget, failed to accomplish critical goals, and was cancelled.

NSA whistleblowers J. Kirk Wiebe, William Binney, Ed Loomis, and House Permanent Select Committee on Intelligence staffer Diane Roark complained to the Department of Defense’s Inspector General (IG) about waste, fraud, and abuse in the program, and the fact that a successful operating prototype existed, but was ignored when the Trailblazer program was launched. The complaint was accepted by the IG and an investigation began that lasted until mid-2005 when the final results were issued. The results were largely hidden, as the report given to the public was heavily (90%) redacted, while the original report was heavily classified, thus restricting the ability of most people to see it.

The people who filed the IG complaint were later raided by armed Federal Bureau of Investigation (FBI) agents. While the Government threatened to prosecute all who signed the IG report, it ultimately chose to pursue an NSA Senior Executive — Thomas Andrews Drake — who helped with the report internally to NSA and who had spoken with a reporter about the project. Drake was later charged under the Espionage Act of 1917. His defenders claimed this was retaliation.[3][4] The charges against him were later dropped, and he agreed to plead guilty to having committed a misdemeanor under the Computer Fraud and Abuse Act, something that Jesselyn Radack of the Government Accountability Project (which helped represent him) called an “act of civil disobedience”.[5]

Background

Trailblazer was chosen over a similar program named ThinThread, a less costly project which had been designed with built-in privacy protections for United States citizens.[4][3] Trailblazer was later linked to the NSA electronic surveillance program and the NSA warrantless surveillance controversy.[3]

In 2002 a consortium led by Science Applications International Corporation was chosen by the NSA to produce a technology demonstration platform in a contract worth $280 million. Project participants included Boeing, Computer Sciences Corporation, and Booz Allen Hamilton. The project was overseen by NSA Deputy Director William B. Black, Jr., an NSA worker who had gone to SAIC, and then been re-hired back to NSA by NSA director Michael Hayden in 2000.[6][7][8] SAIC had also hired a former NSA director to its management; Bobby Inman.[9] SAIC also participated in the concept definition phase of Trailblazer.[10][11]

Redacted version of the DoD Inspector General audit, obtained through the Freedom of Information Act by the Project on Government Oversight and others. [12][5]

The NSA Inspector General issued a report on Trailblazer that “discussed improperly based contract cost increases, non-conformance in the management of the Statement of Work, and excessive labor rates for contractor personnel.” [13]

In 2004 the DoD IG report criticized the program (see the Whistleblowing section below). It said that the “NSA ‘disregarded solutions to urgent national security needs'” and “that TRAILBLAZER was poorly executed and overly expensive …” Several contractors for the project were worried about cooperating with DoD’s audit for fear of “management reprisal.”[5] The Director of NSA “nonconcurred” with several statements in the IG audit, and the report contains a discussion of those disagreements.[14]

In 2005, NSA director Michael Hayden told a Senate hearing that the Trailblazer program was several hundred million dollars over budget and years behind schedule.[15] In 2006 the program was shut down,[3] after having cost billions of US Dollars.[16] Several anonymous NSA sources told Hosenball of Newsweek later on that the project was a “wasteful failure”.[17]

The new project replacing Trailblazer is called Turbulence.[3]

Whistleblowing

According to a 2011 New Yorker article, in the early days of the project several NSA employees met with Diane S Roark, an NSA budget expert on the House Intelligence Committee. They aired their grievances about Trailblazer. In response, NSA director Michael Hayden sent out a memo saying that “individuals, in a session with our congressional overseers, took a position in direct opposition to one that we had corporately decided to follow … Actions contrary to our decisions will have a serious adverse effect on our efforts to transform N.S.A., and I cannot tolerate them.”[3]

In September 2002, several people filed a complaint with the Department of Defense IG’s office regarding problems with Trailblazer: they included Roark (aforementioned), ex-NSA senior analysts Bill Binney, Kirk Wiebe, and Senior Computer Systems Analyst Ed Loomis, who had quit the agency over concerns about its mismanagement of acquisition and allegedly illegal domestic spying.[3][18][19] A major source for the report was NSA senior officer Thomas Andrews Drake. Drake had been complaining to his superiors for some time about problems at the agency, and about the superiority of ThinThread over Trailblazer, for example, at protecting privacy.[19] Drake gave info to DoD during its investigation of the matter.[19] Roark also went to her boss at the House committee, Porter Goss, about problems, but was rebuffed.[20] She also attempted to contact William Renquist, the Supreme Court Chief Justice at the time.[19]

Drake’s own boss, Maureen Baginski, the third-highest officer at NSA, quit partly over concerns about the legality of its behavior.[3]

In 2003, the NSA IG (not the DoD IG)[19] had declared Trailblazer an expensive failure.[21] It had cost more than $1 billion.[8][22][23]

In 2005, the DoD IG produced a report on the result of its investigation of the complaint of Roark and the others in 2002. This report was not released to the public, but it has been described as very negative.[18] Mayer writes that it hastened the closure of Trailblazer, which was at the time in trouble from congress for being over budget.[3]

In November 2005, Drake contacted Siobhan Gorman, a reporter of The Baltimore Sun.[24][17][25] Gorman wrote several articles about problems at the NSA, including articles on Trailblazer. This series got her an award from the Society of Professional Journalists.[17]

In 2005, President George W. Bush ordered the FBI to find whoever had disclosed information about the NSA electronic surveillance program and its disclosure in the New York Times. Eventually, this investigation led to the people who had filed the 2002 DoD IG request, even though they had nothing to do with the New York Times disclosure. In 2007, the houses of Roark, Binney, and Wiebe were raided by armed FBI agents. According to Mayer, Binney claims the FBI pointed guns at his head and that of his wife. Wiebe said it reminded him of the Soviet Union.[3][18] None of these people were ever charged with any crime. Four months later, Drake was raided in November 2007 and his computers and documents were confiscated.

In 2010 Drake was indicted by the U.S. Department of Justice on charges of obstructing justice, providing false information, and violating the Espionage Act of 1917,[17][26][27] part of President Barack Obama’s crackdown on whistleblowers and “leakers”.[24][17][28][18] The government tried to get Roark to testify to a conspiracy, and made similar requests to Drake, offering him a plea bargain. They both refused.[3]

In June 2011, the ten original charges against Drake were dropped, instead he pleaded guilty to a misdemeanor.[5]

Boundless Informant

Boundless Informant is a big data analysis and data visualization system used by the United States National Security Agency (NSA) to give NSA managers summaries of NSA’s world wide data collection activities.[1] It is described in an unclassified, For Official Use Only Frequently Asked Questions (FAQ) memo published by The Guardian.[2] According to a Top Secret heat map display also published by The Guardian and allegedly produced by the Boundless Informant program, almost 3 billion data elements from inside the United States were captured by NSA over a 30-day period ending in March 2013.

Data analyzed by Boundless Informant includes electronic surveillance program records (DNI) and telephone call metadata records (DNR) stored in an NSA data archive called GM-PLACE. It does not include FISA data, according to the FAQ memo. PRISM, a government codename for a collection effort known officially as US-984XN, which was revealed at the same time as Boundless Informant, is one source of DNR data. According to the map, Boundless Informant summarizes data records from 504 separate DNR and DNI collection sources (SIGADs). In the map, countries that are under surveillance are assigned a color from green, representing least coverage to red, most intensive.[3][4]

History

Slide showing that much of the world’s communications flow through the US.

Intelligence gathered by the United States government inside the United States or specifically targeting US citizens is legally required to be gathered in compliance with the Foreign Intelligence Surveillance Act of 1978 (FISA) and under the authority of the Foreign Intelligence Surveillance Court (FISA court).[5][6][7]

NSA global data mining projects have existed for decades, but recent programs of intelligence gathering and analysis that include data gathered from inside the United States such as PRISM were enabled by changes to US surveillance law introduced under President Bush and renewed under President Obama in December 2012.[8]

Boundless Informant was first publicly revealed on June 8, 2013, after classified documents about the program were leaked to The Guardian.[1][9] The newspaper identified its informant, at his request, as Edward Snowden, who worked at the NSA for the defense contractor Booz Allen Hamilton.[10]

Technology

According to published slides, Boundless Informant leverages Free and Open Source Software—and is therefore “available to all NSA developers”—and corporate services hosted in the cloud. The tool uses HDFS, MapReduce, and Cloudbase for data processing.[11]

Legality and FISA Amendments Act of 2008

The FISA Amendments Act (FAA) Section 702 is referenced in PRISM documents detailing the electronic interception, capture and analysis of metadata. Many reports and letters of concern written by members of Congress suggest that this section of FAA in particular is legally and constitutionally problematic, such as by targeting U.S. persons, insofar as “Collections occur in U.S.” as published documents indicate.[12][13][14][15]

The ACLU has asserted the following regarding the FAA: “Regardless of abuses, the problem with the FAA is more fundamental: the statute itself is unconstitutional.”[16]

Senator Rand Paul is introducing new legislation called the Fourth Amendment Restoration Act of 2013 to stop the NSA or other agencies of the United States government from violating the Fourth Amendment to the U.S. Constitution using technology and big data information systems like PRISM and Boundless Informant.[17][18]

ECHELON

ECHELON is a name used in global media and in popular culture to describe a signals intelligence (SIGINT) collection and analysis network operated on behalf of the five signatory states to the UKUSA Security Agreement[1] (Australia, Canada, New Zealand, the United Kingdom, and the United States, referred to by a number of abbreviations, including AUSCANNZUKUS[1] and Five Eyes).[2][3] It has also been described as the only software system which controls the download and dissemination of the intercept of commercial satellite trunk communications.[4]

ECHELON, according to information in the European Parliament document, “On the existence of a global system for the interception of private and commercial communications (ECHELON interception system)” was created to monitor the military and diplomatic communications of the Soviet Union and its Eastern Bloc allies during the Cold War in the early 1960s.[5]

The system has been reported in a number of public sources.[6] Its capabilities and political implications were investigated by a committee of the European Parliament during 2000 and 2001 with a report published in 2001,[5] and by author James Bamford in his books on the National Security Agency of the United States.[4] The European Parliament stated in its report that the term ECHELON is used in a number of contexts, but that the evidence presented indicates that it was the name for a signals intelligence collection system. The report concludes that, on the basis of information presented, ECHELON was capable of interception and content inspection of telephone calls, fax, e-mail and other data traffic globally through the interception of communication bearers including satellite transmission, public switched telephone networks (which once carried most Internet traffic) and microwave links.[5]

Bamford describes the system as the software controlling the collection and distribution of civilian telecommunications traffic conveyed using communication satellites, with the collection being undertaken by ground stations located in the footprint of the downlink leg.

Organization

The UKUSA intelligence community was assessed by the European Parliament (EP) in 2000 to include the signals intelligence agencies of each of the member states:

the Government Communications Headquarters of the United Kingdom,

the National Security Agency of the United States,

the Communications Security Establishment of Canada,

the Defence Signals Directorate of Australia, and

the Government Communications Security Bureau of New Zealand.

the National SIGINT Organisation (NSO) of The Netherlands

The EP report concluded that it seemed likely that ECHELON is a method of sorting captured signal traffic, rather than a comprehensive analysis tool.[5]

Capabilities

The ability to intercept communications depends on the medium used, be it radio, satellite, microwave, cellular or fiber-optic.[5] During World War II and through the 1950s, high frequency (“short wave”) radio was widely used for military and diplomatic communication,[7] and could be intercepted at great distances.[5] The rise of geostationary communications satellites in the 1960s presented new possibilities for intercepting international communications. The report to the European Parliament of 2001 states: “If UKUSA states operate listening stations in the relevant regions of the earth, in principle they can intercept all telephone, fax and data traffic transmitted via such satellites.”[5]

The role of satellites in point-to-point voice and data communications has largely been supplanted by fiber optics; in 2006, 99% of the world’s long-distance voice and data traffic was carried over optical-fiber.[8] The proportion of international communications accounted for by satellite links is said to have decreased substantially over the past few years[when?] in Central Europe to an amount between 0.4% and 5%.[5] Even in less-developed parts of the world, communications satellites are used largely for point-to-multipoint applications, such as video.[9] Thus, the majority of communications can no longer be intercepted by earth stations; they can only be collected by tapping cables and intercepting line-of-sight microwave signals, which is possible only to a limited extent.[5]

One method of interception is to place equipment at locations where fiber optic communications are switched. For the Internet, much of the switching occurs at relatively few sites. There have been reports of one such intercept site, Room 641A, in the United States. In the past[when?] much Internet traffic was routed through the U.S. and the UK, but this has changed; for example, in 2000, 95% of intra-German Internet communications was routed via the DE-CIX Internet exchange point in Frankfurt.[5] A comprehensive worldwide surveillance network is possible only if clandestine intercept sites are installed in the territory of friendly nations, and/or if local authorities cooperate. The report to the European Parliament points out that interception of private communications by foreign intelligence services is not necessarily limited to the U.S. or British foreign intelligence services.[5]

Most reports on ECHELON focus on satellite interception; testimony before the European Parliament indicated that separate but similar UK-US systems are in place to monitor communication through undersea cables, microwave transmissions and other lines.[10]

Controversy

See also: Industrial espionage

Intelligence monitoring of citizens, and their communications, in the area covered by the AUSCANNZUKUS security agreement has caused concern. British journalist Duncan Campbell and New Zealand journalist Nicky Hager asserted in the 1990s that the United States was exploiting ECHELON traffic for industrial espionage, rather than military and diplomatic purposes.[10] Examples alleged by the journalists include the gear-less wind turbine technology designed by the German firm Enercon[5][11] and the speech technology developed by the Belgian firm Lernout & Hauspie.[12] An article in the US newspaper Baltimore Sun reported in 1995 that European aerospace company Airbus lost a $6 billion contract with Saudi Arabia in 1994 after the US National Security Agency reported that Airbus officials had been bribing Saudi officials to secure the contract.[13][14]

In 2001, the Temporary Committee on the ECHELON Interception System recommended to the European Parliament that citizens of member states routinely use cryptography in their communications to protect their privacy, because economic espionage with ECHELON has been conducted by the US intelligence agencies.[5]

Bamford provides an alternative view, highlighting that legislation prohibits the use of intercepted communications for commercial purposes, although he does not elaborate on how intercepted communications are used as part of an all-source intelligence process.

Hardware

According to its website, the U.S. National Security Agency (NSA) is “a high technology organization … on the frontiers of communications and data processing”. In 1999 the Australian Senate Joint Standing Committee on Treaties was told by Professor Desmond Ball that the Pine Gap facility was used as a ground station for a satellite-based interception network. The satellites were said to be large radio dishes between 20 and 100 meters in diameter in geostationary orbits.[citation needed] The original purpose of the network was to monitor the telemetry from 1970s Soviet weapons, air defence radar, communications satellites and ground based microwave communications.[15]

Name

The European Parliament’s Temporary Committee on the ECHELON Interception System stated: “It seems likely, in view of the evidence and the consistent pattern of statements from a very wide range of individuals and organisations, including American sources, that its name is in fact ECHELON, although this is a relatively minor detail.”[5] The U.S. intelligence community uses many code names (see, for example, CIA cryptonym).

Former NSA employee Margaret Newsham claims that she worked on the configuration and installation of software that makes up the ECHELON system while employed at Lockheed Martin, for whom she worked from 1974 to 1984 in Sunnyvale, California, US, and in Menwith Hill, England, UK.[16] At that time, according to Newsham, the code name ECHELON was NSA’s term for the computer network itself. Lockheed called it P415. The software programs were called SILKWORTH and SIRE. A satellite named VORTEX intercepted communications. An image available on the internet of a fragment apparently torn from a job description shows Echelon listed along with several other code names.[17]

Ground stations

The 2001 European Parliamentary (EP) report[5] lists several ground stations as possibly belonging to, or participating in, the ECHELON network. These include:

Likely satellite intercept stations

The following stations are listed in the EP report (p. 54 ff) as likely to have, or to have had, a role in intercepting transmissions from telecommunications satellites:

Hong Kong (since closed)

Australian Defence Satellite Communications Station (Geraldton, Western Australia)

Room 641A

Room 641A is a telecommunication interception facility operated by AT&T for the U.S. National Security Agency that commenced operations in 2003 and was exposed in 2006.[1][2]

Description

Room 641A is located in the SBC Communications building at 611 Folsom Street, San Francisco, three floors of which were occupied by AT&T before SBC purchased AT&T.[1] The room was referred to in internal AT&T documents as the SG3 [Study Group 3] Secure Room. It is fed by fiber optic lines from beam splitters installed in fiber optic trunks carrying Internet backbone traffic[3] and, as analyzed by J. Scott Marcus, a former CTO for GTE and a former adviser to the FCC, who has access to all Internet traffic that passes through the building, and therefore “the capability to enable surveillance and analysis of internet content on a massive scale, including both overseas and purely domestic traffic.”[4] Former director of the NSA’s World Geopolitical and Military Analysis Reporting Group, William Binney, has estimated that 10 to 20 such facilities have been installed throughout the United States.[2]

The room measures about 24 by 48 feet (7.3 by 15 m) and contains several racks of equipment, including a Narus STA 6400, a device designed to intercept and analyze Internet communications at very high speeds.[1]

The very existence of the room was revealed by a former AT&T technician, Mark Klein, and was the subject of a 2006 class action lawsuit by the Electronic Frontier Foundation against AT&T.[5] Klein claims he was told that similar black rooms are operated at other facilities around the country.

Room 641A and the controversies surrounding it were subjects of an episode of Frontline, the current affairs documentary program on PBS. It was originally broadcast on May 15, 2007. It was also featured on PBS’s NOW on March 14, 2008. The room was also covered in the PBS Nova episode “The Spy Factory”.

Lawsuit

Basic diagram of how the alleged wiretapping was accomplished. From EFF court filings[4]

The Electronic Frontier Foundation (EFF) filed a class-action lawsuit against AT&T on January 31, 2006, accusing the telecommunication company of violating the law and the privacy of its customers by collaborating with the National Security Agency (NSA) in a massive, illegal program to wiretap and data-mine Americans’ communications. On July 20, 2006, a federal judge denied the government’s and AT&T’s motions to dismiss the case, chiefly on the ground of the States Secrets Privilege, allowing the lawsuit to go forward. On August 15, 2007, the case was heard by the Ninth Circuit Court of Appeals and was dismissed on December 29, 2011 based on a retroactive grant of immunity by Congress for telecommunications companies that cooperated with the government. The U.S. Supreme Court declined to hear the case.[6] A different case by the EFF was filed on September 18, 2008, titled Jewel v. NSA.

PRISM: A clandestine national security electronic surveillance program operated by the United States National Security Agency (NSA) which can target customers of participating corporations outside or inside the United States

Main Core: A personal and financial database storing information of millions of U.S. citizens believed to be threats to national security.[7] The data mostly comes from the NSA, FBI, CIA, as well as other government sources.[7]

FULL: Edward Snowden and ACLU at SXSW

Snowden Appears Via Video Conference At SXSW Panel

Speaking remotely from Russia on Monday, former National Security Agency contractor Edward Snowden told attendees at the SXSW Interactive conference in Austin, Texas that encryption is still a powerful deterrent against government surveillance.

“Obama Is BIG BROTHER And He’s A LIAR!”

Edward Snowden looms over Pulitzer Prizes

Next month, the trustees who oversee America’s most distinguished journalistic award could face their toughest decision in at least four decades.

The issue before the Pulitzer Prize Board: Does it honor reporting by The Washington Post and The Guardian based on stolen government documents that are arguably detrimental to the national security of the United States, and which were provided by a man who many see as a traitor? Or, does it pass over what is widely viewed as the single most significant story of the year — if not the decade — for the sake of playing it safe?

The politically charged debate surrounding the National Security Agency’s widespread domestic surveillance program, and the man who revealed it, Edward Snowden, is certain to prompt intense discussion for the 19-member Board as it gathers to decide this year’s winners, according to past Board members, veteran journalists and media watchdogs. The debate echoes the historic decision in 1972, when the Board honored The New York Times for its reporting on Daniel Ellsberg’s Pentagon Papers, they said.

“This is an institutional question for them,” said Robert Kaiser, the veteran Washington Post journalist and a previous Pulitzer Prize finalist. “This is a very good argument to have, and there are members of that Board who are going to raise these questions and want to talk about them.”

The risks are manifold, and there is no easy answer: Honoring the NSA reporting — particularly in the coveted category of Public Service — would inevitably be perceived as a political act, with the Pulitzer committee invoking its prestige on behalf of one side in a bitter national argument. In effect, it would be a rebuttal to prominent establishment voices in both parties who say that Snowden’s revelations, and the decision by journalists to publish them, were the exact opposite of a public service. President Barack Obama has said that Snowden’s leaks “could impact our operations in ways that we may not fully understand for years to come.” Former Vice President Dick Cheney has called him “a traitor.” Snowden, who is living in Russia, is facing three felony charges in a criminal complaint filed by the Justice Department.

Yet to pass on the NSA story would be to risk giving the appearance of timidity, siding with the government over the journalists who are trying to hold it accountable and ignoring the most significant disclosure of state secrets in recent memory. It would also look like a willful decision to deny the obvious: No other event has had as dramatic an impact on national and international debates over state surveillance and individual privacy. Last December, in a move that Snowden later described as vindication, a federal district judge ruled that the NSA surveillance Snowden exposed most likely violates the Constitution. Another judge later found the surveillance lawful.

“The stories that came out of this completely changed the agenda on the discussion on privacy and the NSA,” said David Remnick, editor of The New Yorker. “There’s an enormous public good in that, and it’s yet to be proven at all that somehow did great damage to national security.”

Two teams are being considered for their work on the NSA leaks, POLITICO has confirmed. One is made up of The Guardian’s Glenn Greenwald, Laura Poitras and Ewen MacAskill, who published the first landmark report on the NSA’s collection of Verizon phone records, and have since played an integral role in building upon those revelations. The other is Poitras and Barton Gellman, who reported on the wide-ranging surveillance program known as “PRISM” for The Washington Post.

Here, too, the Board faces a challenge: In the eyes of privacy advocates, Greenwald’s work has been much more consequential in the larger arc of the Snowden story, and it was Greenwald who flew to Hong Kong to meet with Snowden and earn his trust. But Greenwald, a staunch anti-surveillance advocate with a brash, outsider’s persona, is not the type of journalist the Pulitzer Board has typically admired. Gellman, by contrast, with his serious and soft-spoken demeanor and decades in the business, comes straight out of Pulitzer central casting. But on what grounds could the Pulitzers recognize Gellman and not Greenwald?

All of these questions will be on the table when the Pulitzer committee meets on April 10 and 11. The winners will be announced on Monday, April 14, at a 3 p.m. news conference at Columbia’s Journalism School.

Sig Gissler, the administrator of the Pulitzer Prizes, and several board members declined to comment on the group’s approach to the NSA reporting, citing the confidentiality of the selection process. “Jurors sign oaths of confidentiality. We certainly do not comment on what is or is not entered or nominated,” Gissler said.

Both Greenwald and Gellman also declined to comment, as did the top editors at The Guardian and The Washington Post. Submissions in each category have already been considered by separate juries, which nominate three finalists to the Board. The Board then considers those nominations for the prizes; with a three-fourths vote, they can move a submission to a different category or recommend another work for consideration. The Guardian’s reporting was conducted through its U.S. outlet in New York, making it eligible for submission.

Several journalists believe that Snowden’s actions should have no bearing on the Pulitzer board’s considerations. It is the reporting that is being honored, not the source, they said.

“The question always is, ‘What was the best journalism produced in the past year?’ And it’s hard to think of a story that has had the impact of the NSA revelations,” said Rem Rieder, the media editor and columnist at USA Today . “These articles made public really important information that the public needs to know, and started a very important national debate over something that should not be decided unilaterally by the executive branch without public input or knowledge.”

Others have a harder time drawing such a definitive line. Michael Kinsley, the veteran political columnist and commentator, has wondered if there isn’t a dubious double standard in the way journalists are honored as heroes while their sources are portrayed as criminals. “If Snowden is guilty of a crime, why isn’t Bart Gellman guilty also?” he asked in an essay for The New Republic last year. Kinsley declined to comment for this piece.

Many of Snowden’s critics are often quick to paint Greenwald, Snowden’s staunchest public advocate, as an accomplice. James Clapper, President Obama’s director of national intelligence, even referred to “Snowden and his accomplices” while testifying before the Senate Intelligence Committee in January.

Whatever the Board’s intention, the decision to give an award to any NSA-related journalism would almost certainly be interpreted as a vindication of Snowden’s efforts, many said. That perceived declaration would surely invite blowback from those who see Snowden in a negative light. In January, after The New York Times editorial board called for clemency for Snowden, Rep. Peter King (R-NY) accused the paper’s editors of being “apologists for terrorists.”

The complications don’t end there. If the Board does decide to honor the reporting on the NSA, it will then have to wrestle with the fact that reporters from two publications were involved in the revelations. Though the Board has given dual awards in the past — the last occasion was in 2006 when The New Orleans Times-Picayune and the Biloxi-Gulfport Sun Herald split the Public Service award for their coverage of Hurricane Katrina — Greenwald’s role as an advocate could further impact the decision. The Brazil-based lawyer, who now works for Pierre Omidyar’s First Look Media, has kept a high profile throughout the past nine months, publicly advocating on Snowden’s behalf — and against the U.S. government — in television appearances, news interviews, and on social media.

“This institution [the Pulitzers] has a tendency to take itself awfully seriously,” said Kaiser, who described Greenwald’s work as “causist” reporting. “Whether committed causes should get a Pulitzer Prize for any kind of reporting is an open question. They’ll have to decide how judgmental they need to be.”

Gellman’s more traditional handling of the NSA story may have more appeal to the board. Instead of jumping into the fray on a near-daily basis, fighting on Twitter and giving contentious cable news interviews, Gellman has produced a few comprehensive reports that sought to put new revelations in a greater context. His ties to The Washington Post have also given the NSA story the imprimatur of “old media” integrity, which the Board is said to value.

Finally, there is the issue of effort. Though Greenwald and Gellman have dismissed the suggestion that Snowden’s trove of NSA files simply fell into their laps, the Pulitzer Board could feel conflicted about giving an award to the recipients of stolen documents when other applicants may have dedicated a significant amount of time and resources to old-fashioned shoe-leather reporting on, say, a local government issue. In several instances throughout its history, the Board has honored reporting based to a significant degree on the amount of effort and diligence shown by the reporters.

“The one wild card is the degree of difficulty question,” Rieder said. “Not to minimize the role of the reporters — it’s not just stenography. You have to sift through the information, present it clearly, explain why it matters, put it in context, etc. The real challenge would be if you had entries where reporters had to go to extraordinary lengths to pry out information of vital interest to the public, as opposed to having it turned over to them. If you had examples of great magnitude, that would make it complicated. That said, this was clearly the story of last year.”

“There’s a real question about whether this is reporting,” Kaiser said. “It might be a public service award, but it’s not a great reporting coup when a source comes to you and hands you this stuff.”

Both Greenwald and Gellman have adamantly dismissed the suggestion that they were merely stenographers for Snowden. Greenwald in particular traveled to Hong Kong and spent hours working with Snowden and earning his trust. Greenwald also continues to pore over the files in his possession, and says he has published just a small fraction of what Snowden gave him.

While the Board refuses to discuss next month’s awards, there are precedents that shed light on how that committee may decide to handle the NSA-related submissions.

In 1972, after what The Associated Press then described as “unprecedented debate,” the Pulitzer committee gave The New York Times the Public Service award for Neil Sheehan’s reporting on the Pentagon Papers, which he had received from former military analyst Daniel Ellsberg. At the time the award was given, Ellsberg was awaiting trial on charges of theft, which were later dropped.

Michael Gartner, the former NBC News president and Iowa newspaperman who spent 10 years on the Pulitzer Board, said he saw no substantive difference between the journalism that resulted from Ellsberg and Snowden’s stolen documents.

“I’m sure that there will be great debates over Snowden’s stuff, but really wasn’t that precedent set with the Pentagon Papers? The nature of the theft might be different, but isn’t the journalism the same — great stories produced from documents that were leaked by an employee of a private contractor?” Gartner wrote in an email. “I can make a distinction between Ellsberg and Snowden, if I have to, based on the nature of what they stole, but how can the board make a distinction between what was published then and what was published now? Reporting is reporting. If I were arguing for the Snowden stuff — and I would — that is the argument I would make.”

In 2006, the Pulitzer committee honored James Risen and Eric Lichtblau of The New York Times for their reporting on the George W. Bush administration’s secret wiretapping program. That decision, too, was a subject of intense internal debate. President Bush had personally asked the Times not to publish the article, and the committee’s decision to honor Risen and Lichtblau’s report was seen as a public rebuke of Bush administration policies.

Last month, in a move that set the stage for April’s Pulitzer debate, Long Island University gave both the Greenwald and Gellman teams the George Polk Award for National Security Reporting.

John Darnton, the curator of the Polk Awards, said he received emails from critics who, seemingly unaware of the precedent set by the Pentagon Papers, blasted the group’s decision to honor reporting based on stolen government documents. One of those emails came from Accuracy In Media, the conservative watchdog.

In a lengthy email to POLITICO, Cliff Kincaid, director of the AIM Center for Investigative Journalism, criticized Snowden and Greenwald for threatening national security.

“Political figures in both political parties agree that Snowden is a traitor. So what does that make his enablers in the media? They are certainly not journalists who deserve journalism prizes,” Kincaid wrote. “Journalism awards should not be given to recipients of stolen national security documents whose work has made America more vulnerable to terrorist attacks and its military personnel more likely to die at the hands of terrorists or enemy regimes.”

To date, no substantial evidence has emerged publicly that any of Greenwald or Gellman’s reporting has compromised America’s national security or military personnel, although intelligence officials have said they’ve detected changes in how groups like Al Qaeda communicate as a result of the broad controversy.

In the end, Darnton said the 10-member Polk panel hardly thought twice about the decision to bestow awards on Greenwald and Gellman.

“In the case of the NSA coverage, we began with a predisposition to seriously consider it because the repercussions were immense,” he explained. “There was a bit of discussion, but not much. The story itself is just so significant — there was no great dissent.”

Python Web Scraping Tutorial 1 (Intro To Web Scraping)

Web Scraping Techniques

Web scraping: Reliably and efficiently pull data from pages that don’t expect it

Lecture -38 Search Engine And Web Crawler – Part-I

Lecture -39 Search Engine And Web Crawlers: Part-II

Web Scraping Review 1

Web Scraping Review 2

Snowden Used Low-Cost Tool to Best N.S.A.

By DAVID E. SANGER and ERIC SCHMITT

Intelligence officials investigating how Edward J. Snowden gained access to a huge trove of the country’s most highly classified documents say they have determined that he used inexpensive and widely available software to “scrape” the National Security Agency’s networks, and kept at it even after he was briefly challenged by agency officials.

Using “web crawler” software designed to search, index and back up a website, Mr. Snowden “scraped data out of our systems” while he went about his day job, according to a senior intelligence official. “We do not believe this was an individual sitting at a machine and downloading this much material in sequence,” the official said. The process, he added, was “quite automated.”

The findings are striking because the N.S.A.’s mission includes protecting the nation’s most sensitive military and intelligence computer systems from cyberattacks, especially the sophisticated attacks that emanate from Russia and China. Mr. Snowden’s “insider attack,” by contrast, was hardly sophisticated and should have been easily detected, investigators found.

Moreover, Mr. Snowden succeeded nearly three years after the WikiLeaks disclosures, in which military and State Department files, of far less sensitivity, were taken using similar techniques.

Mr. Snowden had broad access to the N.S.A.’s complete files because he was working as a technology contractor for the agency in Hawaii, helping to manage the agency’s computer systems in an outpost that focuses on China and North Korea. A web crawler, also called a spider, automatically moves from website to website, following links embedded in each document, and can be programmed to copy everything in its path.

Mr. Snowden appears to have set the parameters for the searches, including which subjects to look for and how deeply to follow links to documents and other data on the N.S.A.’s internal networks. Intelligence officials told a House hearing last week that he accessed roughly 1.7 million files.

Among the materials prominent in the Snowden files are the agency’s shared “wikis,” databases to which intelligence analysts, operatives and others contributed their knowledge. Some of that material indicates that Mr. Snowden “accessed” the documents. But experts say they may well have been downloaded not by him but by the program acting on his behalf.

Agency officials insist that if Mr. Snowden had been working from N.S.A. headquarters at Fort Meade, Md., which was equipped with monitors designed to detect when a huge volume of data was being accessed and downloaded, he almost certainly would have been caught. But because he worked at an agency outpost that had not yet been upgraded with modern security measures, his copying of what the agency’s newly appointed No. 2 officer, Rick Ledgett, recently called “the keys to the kingdom” raised few alarms.

“Some place had to be last” in getting the security upgrade, said one official familiar with Mr. Snowden’s activities. But he added that Mr. Snowden’s actions had been “challenged a few times.”

In at least one instance when he was questioned, Mr. Snowden provided what were later described to investigators as legitimate-sounding explanations for his activities: As a systems administrator he was responsible for conducting routine network maintenance. That could include backing up the computer systems and moving information to local servers, investigators were told.

But from his first days working as a contractor inside the N.S.A.’s aging underground Oahu facility for Dell, the computer maker, and then at a modern office building on the island for Booz Allen Hamilton, the technology consulting firm that sells and operates computer security services used by the government, Mr. Snowden learned something critical about the N.S.A.’s culture: While the organization built enormously high electronic barriers to keep out foreign invaders, it had rudimentary protections against insiders.

“Once you are inside the assumption is that you are supposed to be there, like in most organizations,” said Richard Bejtlich, the chief security strategist for FireEye, a Silicon Valley computer security firm, and a senior fellow at the Brookings Institution. “But that doesn’t explain why they weren’t more vigilant about excessive activity in the system.”

Investigators have yet to answer the question of whether Mr. Snowden happened into an ill-defended outpost of the N.S.A. or sought a job there because he knew it had yet to install the security upgrades that might have stopped him.

“He was either very lucky or very strategic,” one intelligence official said. A new book, “The Snowden Files,” by Luke Harding, a correspondent for The Guardian in London, reports that Mr. Snowden sought his job at Booz Allen because “to get access to a final tranche of documents” he needed “greater security privileges than he enjoyed in his position at Dell.”

Through his lawyer at the American Civil Liberties Union, Mr. Snowden did not specifically address the government’s theory of how he obtained the files, saying in a statement: “It’s ironic that officials are giving classified information to journalists in an effort to discredit me for giving classified information to journalists. The difference is that I did so to inform the public about the government’s actions, and they’re doing so to misinform the public about mine.”

Launch media viewer

The headquarters of Booz Allen Hamilton, one of Edward J. Snowden’s former employers, in McLean, Va. He had broad access to National Security Agency files as a contractor in Hawaii. Michael Reynolds/European Pressphoto Agency

The N.S.A. declined to comment on its investigation or the security changes it has made since the Snowden disclosures. Other intelligence officials familiar with the findings of the investigations underway — there are at least four — were granted anonymity to discuss the investigations.

In interviews, officials declined to say which web crawler Mr. Snowden had used, or whether he had written some of the software himself. Officials said it functioned like Googlebot, a widely used web crawler that Google developed to find and index new pages on the web. What officials cannot explain is why the presence of such software in a highly classified system was not an obvious tip-off to unauthorized activity.

When inserted with Mr. Snowden’s passwords, the web crawler became especially powerful. Investigators determined he probably had also made use of the passwords of some colleagues or supervisors.

But he was also aided by a culture within the N.S.A., officials say, that “compartmented” relatively little information. As a result, a 29-year-old computer engineer, working from a World War II-era tunnel in Oahu and then from downtown Honolulu, had access to unencrypted files that dealt with information as varied as the bulk collection of domestic phone numbers and the intercepted communications of Chancellor Angela Merkel of Germany and dozens of other leaders.

Officials say web crawlers are almost never used on the N.S.A.’s internal systems, making it all the more inexplicable that the one used by Mr. Snowden did not set off alarms as it copied intelligence and military documents stored in the N.S.A.’s systems and linked through the agency’s internal equivalent of Wikipedia.

The answer, officials and outside experts say, is that no one was looking inside the system in Hawaii for hard-to-explain activity. “The N.S.A. had the solution to this problem in hand, but they simply didn’t push it out fast enough,” said James Lewis, a computer expert at the Center for Strategic and International Studies who has talked extensively with intelligence officials about how the Snowden experience could have been avoided.

Nonetheless, the government had warning that it was vulnerable to such attacks. Similar techniques were used by Chelsea Manning, then known as Pfc. Bradley Manning, who was convicted of turning documents and videos over to WikiLeaks in 2010.

Evidence presented during Private Manning’s court-martial for his role as the source for large archives of military and diplomatic files given to WikiLeaks revealed that he had used a program called “wget” to download the batches of files. That program automates the retrieval of large numbers of files, but it is considered less powerful than the tool Mr. Snowden used.

The program’s use prompted changes in how secret information is handled at the State Department, the Pentagon and the intelligence agencies, but recent assessments suggest that those changes may not have gone far enough. For example, arguments have broken out about whether the N.S.A.’s data should all be encrypted “at rest” — when it is stored in servers — to make it harder to search and steal. But that would also make it harder to retrieve for legitimate purposes.

Investigators have found no evidence that Mr. Snowden’s searches were directed by a foreign power, despite suggestions to that effect by the chairman of the House Intelligence Committee, Representative Mike Rogers, Republican of Michigan, in recent television appearances and at a hearing last week.

But that leaves open the question of how Mr. Snowden chose the search terms to obtain his trove of documents, and why, according to James R. Clapper Jr., the director of national intelligence, they yielded a disproportionately large number of documents detailing American military movements, preparations and abilities around the world.

In his statement, Mr. Snowden denied any deliberate effort to gain access to any military information. “They rely on a baseless premise, which is that I was after military information,” Mr. Snowden said.

The head of the Defense Intelligence Agency, Lt. Gen. Michael T. Flynn, told lawmakers last week that Mr. Snowden’s disclosures could tip off adversaries to American military tactics and operations, and force the Pentagon to spend vast sums to safeguard against that. But he admitted a great deal of uncertainty about what Mr. Snowden possessed.

“Everything that he touched, we assume that he took,” said General Flynn, including details of how the military tracks terrorists, of enemies’ vulnerabilities and of American defenses against improvised explosive devices. He added, “We assume the worst case.”

Web Crawler

A Web crawler may also be called a Web spider,[1] an ant, an automatic indexer,[2] or (in the FOAF software context) a Web scutter.[3]

Web search engines and some other sites use Web crawling or spidering software to update their web content or indexes of others sites’ web content. Web crawlers can copy all the pages they visit for later processing by a search engine that indexes the downloaded pages so that users can search them much more quickly.

Overview

A Web crawler starts with a list of URLs to visit, called the seeds. As the crawler visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit, called the crawl frontier. URLs from the frontier are recursively visited according to a set of policies.

The large volume implies that the crawler can only download a limited number of the Web pages within a given time, so it needs to prioritize its downloads. The high rate of change implies that the pages might have already been updated or even deleted.

The number of possible URLs crawled being generated by server-side software has also made it difficult for web crawlers to avoid retrieving duplicate content. Endless combinations of HTTP GET (URL-based) parameters exist, of which only a small selection will actually return unique content. For example, a simple online photo gallery may offer three options to users, as specified through HTTP GET parameters in the URL. If there exist four ways to sort images, three choices of thumbnail size, two file formats, and an option to disable user-provided content, then the same set of content can be accessed with 48 different URLs, all of which may be linked on the site. This mathematical combination creates a problem for crawlers, as they must sort through endless combinations of relatively minor scripted changes in order to retrieve unique content.

As Edwards et al. noted, “Given that the bandwidth for conducting crawls is neither infinite nor free, it is becoming essential to crawl the Web in not only a scalable, but efficient way, if some reasonable measure of quality or freshness is to be maintained.”[4] A crawler must carefully choose at each step which pages to visit next.

Crawling policy

The behavior of a Web crawler is the outcome of a combination of policies:[5]

a selection policy that states which pages to download,

a re-visit policy that states when to check for changes to the pages,

a politeness policy that states how to avoid overloading Web sites, and

Selection policy

Given the current size of the Web, even large search engines cover only a portion of the publicly available part. A 2005 study showed that large-scale search engines index no more than 40-70% of the indexable Web;[6] a previous study by Steve Lawrence and Lee Giles showed that no search engine indexed more than 16% of the Web in 1999.[7] As a crawler always downloads just a fraction of the Web pages, it is highly desirable that the downloaded fraction contains the most relevant pages and not just a random sample of the Web.

This requires a metric of importance for prioritizing Web pages. The importance of a page is a function of its intrinsic quality, its popularity in terms of links or visits, and even of its URL (the latter is the case of vertical search engines restricted to a single top-level domain, or search engines restricted to a fixed Web site). Designing a good selection policy has an added difficulty: it must work with partial information, as the complete set of Web pages is not known during crawling.

Cho et al. made the first study on policies for crawling scheduling. Their data set was a 180,000-pages crawl from the stanford.edu domain, in which a crawling simulation was done with different strategies.[8] The ordering metrics tested were breadth-first, backlink count and partial Pagerank calculations. One of the conclusions was that if the crawler wants to download pages with high Pagerank early during the crawling process, then the partial Pagerank strategy is the better, followed by breadth-first and backlink-count. However, these results are for just a single domain. Cho also wrote his Ph.D. dissertation at Stanford on web crawling.[9]

Najork and Wiener performed an actual crawl on 328 million pages, using breadth-first ordering.[10] They found that a breadth-first crawl captures pages with high Pagerank early in the crawl (but they did not compare this strategy against other strategies). The explanation given by the authors for this result is that “the most important pages have many links to them from numerous hosts, and those links will be found early, regardless of on which host or page the crawl originates.”

Abiteboul designed a crawling strategy based on an algorithm called OPIC (On-line Page Importance Computation).[11] In OPIC, each page is given an initial sum of “cash” that is distributed equally among the pages it points to. It is similar to a Pagerank computation, but it is faster and is only done in one step. An OPIC-driven crawler downloads first the pages in the crawling frontier with higher amounts of “cash”. Experiments were carried in a 100,000-pages synthetic graph with a power-law distribution of in-links. However, there was no comparison with other strategies nor experiments in the real Web.

Boldi et al. used simulation on subsets of the Web of 40 million pages from the .it domain and 100 million pages from the WebBase crawl, testing breadth-first against depth-first, random ordering and an omniscient strategy. The comparison was based on how well PageRank computed on a partial crawl approximates the true PageRank value. Surprisingly, some visits that accumulate PageRank very quickly (most notably, breadth-first and the omniscient visit) provide very poor progressive approximations.[12][13]

Baeza-Yates et al. used simulation on two subsets of the Web of 3 million pages from the .gr and .cl domain, testing several crawling strategies.[14] They showed that both the OPIC strategy and a strategy that uses the length of the per-site queues are better than breadth-first crawling, and that it is also very effective to use a previous crawl, when it is available, to guide the current one.

Daneshpajouh et al. designed a community based algorithm for discovering good seeds.[15] Their method crawls web pages with high PageRank from different communities in less iteration in comparison with crawl starting from random seeds. One can extract good seed from a previously-crawled-Web graph using this new method. Using these seeds a new crawl can be very effective.

Restricting followed links

A crawler may only want to seek out HTML pages and avoid all other MIME types. In order to request only HTML resources, a crawler may make an HTTP HEAD request to determine a Web resource’s MIME type before requesting the entire resource with a GET request. To avoid making numerous HEAD requests, a crawler may examine the URL and only request a resource if the URL ends with certain characters such as .html, .htm, .asp, .aspx, .php, .jsp, .jspx or a slash. This strategy may cause numerous HTML Web resources to be unintentionally skipped.

Some crawlers may also avoid requesting any resources that have a “?” in them (are dynamically produced) in order to avoid spider traps that may cause the crawler to download an infinite number of URLs from a Web site. This strategy is unreliable if the site uses a rewrite engine to simplify its URLs.

URL normalization

Crawlers usually perform some type of URL normalization in order to avoid crawling the same resource more than once. The term URL normalization, also called URL canonicalization, refers to the process of modifying and standardizing a URL in a consistent manner. There are several types of normalization that may be performed including conversion of URLs to lowercase, removal of “.” and “..” segments, and adding trailing slashes to the non-empty path component.[16]

Path-ascending crawling

Some crawlers intend to download as many resources as possible from a particular web site. So path-ascending crawler was introduced that would ascend to every path in each URL that it intends to crawl.[17] For example, when given a seed URL of http://llama.org/hamster/monkey/page.html, it will attempt to crawl /hamster/monkey/, /hamster/, and /. Cothey found that a path-ascending crawler was very effective in finding isolated resources, or resources for which no inbound link would have been found in regular crawling.

Many path-ascending crawlers are also gallery — from a specific page or host.

Focused crawling

The importance of a page for a crawler can also be expressed as a function of the similarity of a page to a given query. Web crawlers that attempt to download pages that are similar to each other are called focused crawler or topical crawlers. The concepts of topical and focused crawling were first introduced by Menczer[18][19] and by Chakrabarti et al.[20]

The main problem in focused crawling is that in the context of a Web crawler, we would like to be able to predict the similarity of the text of a given page to the query before actually downloading the page. A possible predictor is the anchor text of links; this was the approach taken by Pinkerton[21] in the first web crawler of the early days of the Web. Diligenti et al.[22] propose using the complete content of the pages already visited to infer the similarity between the driving query and the pages that have not been visited yet. The performance of a focused crawling depends mostly on the richness of links in the specific topic being searched, and a focused crawling usually relies on a general Web search engine for providing starting points.

Academic-focused crawler

An example of the focused crawlers are academic crawlers, which crawls free-access academic related documents, such as the citeseerxbot, which is the crawler of CiteSeerX search engine. Other academic search engines are Google Scholar and Microsoft Academic Search etc. Because most academic papers are published in PDF formats, such kind of crawler is particularly interested in crawling PDF, PostScript files, Microsoft Word including their zipped formats. Because of this, general open source crawlers, such as Heritrix, must be customized to filter out other MIME types, or a middleware is used to extract these documents out and import them to the focused crawl database and repository.[23] Identifying whether these documents are academic or not is challenging and can add a significant overhead to the crawling process, so this is performed as a post crawling process using machine learning or regular expression algorithms. These academic documents are usually obtained from home pages of faculties and students or from publication page of research institutes. Because academic documents takes only a small faction in the entire web pages, a good seed selection are important in boosting the efficiencies of these web crawlers.[24] Other academic crawlers may download plain text and HTML files, that contains metadata of academic papers, such as titles, papers, and abstracts. This increases the overall number of papers, but a significant fraction may not provide free PDF downloads.

Re-visit policy

The Web has a very dynamic nature, and crawling a fraction of the Web can take weeks or months. By the time a Web crawler has finished its crawl, many events could have happened, including creations, updates and deletions.

From the search engine’s point of view, there is a cost associated with not detecting an event, and thus having an outdated copy of a resource. The most-used cost functions are freshness and age.[25]

Freshness: This is a binary measure that indicates whether the local copy is accurate or not. The freshness of a page p in the repository at time t is defined as:

Age: This is a measure that indicates how outdated the local copy is. The age of a page p in the repository, at time t is defined as:

Coffmanet al. worked with a definition of the objective of a Web crawler that is equivalent to freshness, but use a different wording: they propose that a crawler must minimize the fraction of time pages remain outdated. They also noted that the problem of Web crawling can be modeled as a multiple-queue, single-server polling system, on which the Web crawler is the server and the Web sites are the queues. Page modifications are the arrival of the customers, and switch-over times are the interval between page accesses to a single Web site. Under this model, mean waiting time for a customer in the polling system is equivalent to the average age for the Web crawler.[26]

The objective of the crawler is to keep the average freshness of pages in its collection as high as possible, or to keep the average age of pages as low as possible. These objectives are not equivalent: in the first case, the crawler is just concerned with how many pages are out-dated, while in the second case, the crawler is concerned with how old the local copies of pages are.

Two simple re-visiting policies were studied by Cho and Garcia-Molina:[27]

Uniform policy: This involves re-visiting all pages in the collection with the same frequency, regardless of their rates of change.

Proportional policy: This involves re-visiting more often the pages that change more frequently. The visiting frequency is directly proportional to the (estimated) change frequency.

(In both cases, the repeated crawling order of pages can be done either in a random or a fixed order.)

Cho and Garcia-Molina proved the surprising result that, in terms of average freshness, the uniform policy outperforms the proportional policy in both a simulated Web and a real Web crawl. Intuitively, the reasoning is that, as web crawlers have a limit to how many pages they can crawl in a given time frame, (1) they will allocate too many new crawls to rapidly changing pages at the expense of less frequently updating pages, and (2) the freshness of rapidly changing pages lasts for shorter period than that of less frequently changing pages. In other words, a proportional policy allocates more resources to crawling frequently updating pages, but experiences less overall freshness time from them.

To improve freshness, the crawler should penalize the elements that change too often.[28] The optimal re-visiting policy is neither the uniform policy nor the proportional policy. The optimal method for keeping average freshness high includes ignoring the pages that change too often, and the optimal for keeping average age low is to use access frequencies that monotonically (and sub-linearly) increase with the rate of change of each page. In both cases, the optimal is closer to the uniform policy than to the proportional policy: as Coffmanet al. note, “in order to minimize the expected obsolescence time, the accesses to any particular page should be kept as evenly spaced as possible”.[26] Explicit formulas for the re-visit policy are not attainable in general, but they are obtained numerically, as they depend on the distribution of page changes. Cho and Garcia-Molina show that the exponential distribution is a good fit for describing page changes,[28] while Ipeirotiset al. show how to use statistical tools to discover parameters that affect this distribution.[29] Note that the re-visiting policies considered here regard all pages as homogeneous in terms of quality (“all pages on the Web are worth the same”), something that is not a realistic scenario, so further information about the Web page quality should be included to achieve a better crawling policy.

Politeness policy

Crawlers can retrieve data much quicker and in greater depth than human searchers, so they can have a crippling impact on the performance of a site. Needless to say, if a single crawler is performing multiple requests per second and/or downloading large files, a server would have a hard time keeping up with requests from multiple crawlers.

As noted by Koster, the use of Web crawlers is useful for a number of tasks, but comes with a price for the general community.[30] The costs of using Web crawlers include:

network resources, as crawlers require considerable bandwidth and operate with a high degree of parallelism during a long period of time;

server overload, especially if the frequency of accesses to a given server is too high;

poorly written crawlers, which can crash servers or routers, or which download pages they cannot handle; and

personal crawlers that, if deployed by too many users, can disrupt networks and Web servers.

A partial solution to these problems is the robots exclusion protocol, also known as the robots.txt protocol that is a standard for administrators to indicate which parts of their Web servers should not be accessed by crawlers.[31] This standard does not include a suggestion for the interval of visits to the same server, even though this interval is the most effective way of avoiding server overload. Recently commercial search engines like Google, Ask Jeeves, MSN and Yahoo! Search are able to use an extra “Crawl-delay:” parameter in the robots.txt file to indicate the number of seconds to delay between requests.

The first proposed interval between successive pageloads was 60 seconds.[32] However, if pages were downloaded at this rate from a website with more than 100,000 pages over a perfect connection with zero latency and infinite bandwidth, it would take more than 2 months to download only that entire Web site; also, only a fraction of the resources from that Web server would be used. This does not seem acceptable.

Cho uses 10 seconds as an interval for accesses,[27] and the WIRE crawler uses 15 seconds as the default.[33] The MercatorWeb crawler follows an adaptive politeness policy: if it took t seconds to download a document from a given server, the crawler waits for 10t seconds before downloading the next page.[34] Dill et al. use 1 second.[35]

For those using Web crawlers for research purposes, a more detailed cost-benefit analysis is needed and ethical considerations should be taken into account when deciding where to crawl and how fast to crawl.[36]

Anecdotal evidence from access logs shows that access intervals from known crawlers vary between 20 seconds and 3–4 minutes. It is worth noticing that even when being very polite, and taking all the safeguards to avoid overloading Web servers, some complaints from Web server administrators are received. Brin and Page note that: “… running a crawler which connects to more than half a million servers (…) generates a fair amount of e-mail and phone calls. Because of the vast number of people coming on line, there are always those who do not know what a crawler is, because this is the first one they have seen.”[37]

Parallelisation policy

A parallel crawler is a crawler that runs multiple processes in parallel. The goal is to maximize the download rate while minimizing the overhead from parallelization and to avoid repeated downloads of the same page. To avoid downloading the same page more than once, the crawling system requires a policy for assigning the new URLs discovered during the crawling process, as the same URL can be found by two different crawling processes.

Architectures

High-level architecture of a standard Web crawler

A crawler must not only have a good crawling strategy, as noted in the previous sections, but it should also have a highly optimized architecture.

While it is fairly easy to build a slow crawler that downloads a few pages per second for a short period of time, building a high-performance system that can download hundreds of millions of pages over several weeks presents a number of challenges in system design, I/O and network efficiency, and robustness and manageability.

Web crawlers are a central part of search engines, and details on their algorithms and architecture are kept as business secrets. When crawler designs are published, there is often an important lack of detail that prevents others from reproducing the work. There are also emerging concerns about “search engine spamming“, which prevent major search engines from publishing their ranking algorithms.

Crawler identification

Web crawlers typically identify themselves to a Web server by using the User-agent field of an HTTP request. Web site administrators typically examine their Web servers‘ log and use the user agent field to determine which crawlers have visited the web server and how often. The user agent field may include a URL where the Web site administrator may find out more information about the crawler. Examining Web server log is tedious task therefore some administrators use tools such as CrawlTrack[39] or SEO Crawlytics[40] to identify, track and verify Web crawlers. Spambots and other malicious Web crawlers are unlikely to place identifying information in the user agent field, or they may mask their identity as a browser or other well-known crawler.

It is important for Web crawlers to identify themselves so that Web site administrators can contact the owner if needed. In some cases, crawlers may be accidentally trapped in a crawler trap or they may be overloading a Web server with requests, and the owner needs to stop the crawler. Identification is also useful for administrators that are interested in knowing when they may expect their Web pages to be indexed by a particular search engine.

Crawling the deep web

A vast amount of web pages lie in the deep or invisible web.[41] These pages are typically only accessible by submitting queries to a database, and regular crawlers are unable to find these pages if there are no links that point to them. Google’s Sitemaps protocol and mod oai[42] are intended to allow discovery of these deep-Web resources.

Deep web crawling also multiplies the number of web links to be crawled. Some crawlers only take some of the URLs in <a href="URL"> form. In some cases, such as the Googlebot, Web crawling is done on all text contained inside the hypertext content, tags, or text.

Strategic approaches may be taken to target deep Web content. With a technique called screen scraping, specialized software may be customized to automatically and repeatedly query a given Web form with the intention of aggregating the resulting data. Such software can be used to span multiple Web forms across multiple Websites. Data extracted from the results of one Web form submission can be taken and applied as input to another Web form thus establishing continuity across the Deep Web in a way not possible with traditional web crawlers.

Pages built on AJAX are among those causing problems to web crawlers. Google has proposed a format of AJAX calls that their bot can recognize and index[43]

Web crawler bias

A recent study based on a large scale analysis of robots.txt files showed that certain web crawlers were preferred over others, with Googlebot being the most preferred web crawler.[citation needed]

The following is a list of published crawler architectures for general-purpose crawlers (excluding focused web crawlers), with a brief description that includes the names given to the different components and outstanding features:

Googlebot[37] is described in some detail, but the reference is only about an early version of its architecture, which was based in C++ and Python. The crawler was integrated with the indexing process, because text parsing was done for full-text indexing and also for URL extraction. There is a URL server that sends lists of URLs to be fetched by several crawling processes. During parsing, the URLs found were passed to a URL server that checked if the URL have been previously seen. If not, the URL was added to the queue of the URL server.

PolyBot[38] is a distributed crawler written in C++ and Python, which is composed of a “crawl manager”, one or more “downloaders” and one or more “DNS resolvers”. Collected URLs are added to a queue on disk, and processed later to search for seen URLs in batch mode. The politeness policy considers both third and second level domains (e.g.: http://www.example.com and www2.example.com are third level domains) because third level domains are usually hosted by the same Web server.

RBSE[45] was the first published web crawler. It was based on two programs: the first program, “spider” maintains a queue in a relational database, and the second program “mite“, is a modified wwwASCII browser that downloads the pages from the Web.

WebCrawler[21] was used to build the first publicly available full-text index of a subset of the Web. It was based on lib-WWW to download pages, and another program to parse and order URLs for breadth-first exploration of the Web graph. It also included a real-time crawler that followed links based on the similarity of the anchor text with the provided query.

World Wide Web Worm[46] was a crawler used to build a simple index of document titles and URLs. The index could be searched by using the grepUnix command.

WebFountain[4] is a distributed, modular crawler similar to Mercator but written in C++. It features a “controller” machine that coordinates a series of “ant” machines. After repeatedly downloading pages, a change rate is inferred for each page and a non-linear programming method must be used to solve the equation system for maximizing freshness. The authors recommend to use this crawling order in the early stages of the crawl, and then switch to a uniform crawling order, in which all pages are being visited with the same frequency.

WebRACE[47] is a crawling and caching module implemented in Java, and used as a part of a more generic system called eRACE. The system receives requests from users for downloading web pages, so the crawler acts in part as a smart proxy server. The system also handles requests for “subscriptions” to Web pages that must be monitored: when the pages change, they must be downloaded by the crawler and the subscriber must be notified. The most outstanding feature of WebRACE is that, while most crawlers start with a set of “seed” URLs, WebRACE is continuously receiving new starting URLs to crawl from.

In addition to the specific crawler architectures listed above, there are general crawler architectures published by Cho[48] and Chakrabarti.[49]

Web scraping is closely related to web indexing, which indexes information on the web using a bot or web crawler and is a universal technique adopted by most search engines. In contrast, web scraping focuses more on the transformation of unstructured data on the web, typically in HTML format, into structured data that can be stored and analyzed in a central local database or spreadsheet. Web scraping is also related to web automation, which simulates human browsing using computer software. Uses of web scraping include online price comparison, contact scraping, weather data monitoring, website change detection, research, web mashup and web data integration.

Techniques

Web scraping is the process of automatically collecting information from the World Wide Web. It is a field with active developments sharing a common goal with the semantic web vision, an ambitious initiative that still requires breakthroughs in text processing, semantic understanding, artificial intelligence and human-computer interactions. Web scraping, instead, favors practical solutions based on existing technologies that are often entirely ad hoc. Therefore, there are different levels of automation that existing web-scraping technologies can provide:

Human copy-and-paste: Sometimes even the best web-scraping technology cannot replace a human’s manual examination and copy-and-paste, and sometimes this may be the only workable solution when the websites for scraping explicitly set up barriers to prevent machine automation.

Text grepping and regular expression matching: A simple yet powerful approach to extract information from web pages can be based on the UNIX grep command or regular expression-matching facilities of programming languages (for instance Perl or Python).

HTML parsers. Many websites have large collections of pages generated dynamically from an underlying structured source like a database. Data of the same category are typically encoded into similar pages by a common script or template. In data mining, a program that detects such templates in a particular information source, extracts its content and translates it into a relational form called a wrapper. Wrapper generation algorithms assume that input pages of a wrapper induction system conform to a common template and that they can be easily identified in terms of a URL common scheme.[1]. Moreover, some semi-structured data query languages, such as XQuery and the HTQL, can be used to parse HTML pages and to retrieve and transform page content.

DOM parsing: By embedding a full-fledged web browser, such as the Internet Explorer or the Mozilla browser control, programs can retrieve the dynamic content generated by client-side scripts. These browser controls also parse web pages into a DOM tree, based on which programs can retrieve parts of the pages.

Web-scraping software: There are many software tools available that can be used to customize web-scraping solutions. This software may attempt to automatically recognize the data structure of a page or provide a recording interface that removes the necessity to manually write web-scraping code, or some scripting functions that can be used to extract and transform content, and database interfaces that can store the scraped data in local databases.

Vertical aggregation platforms: There are several companies that have developed vertical specific harvesting platforms. These platforms create and monitor a multitude of “bots” for specific verticals with no man-in-the-loop,[clarification needed] and no work related to a specific target site. The preparation involves establishing the knowledge base for the entire vertical and then the platform creates the bots automatically. The platform’s robustness is measured by the quality of the information it retrieves (usually number of fields) and its scalability (how quick it can scale up to hundreds or thousands of sites). This scalability is mostly used to target the Long Tail of sites that common aggregators find complicated or too labor-intensive to harvest content from.

Semantic annotation recognizing: The pages being scraped may embrace metadata or semantic markups and annotations, which can be used to locate specific data snippets. If the annotations are embedded in the pages, as Microformat does, this technique can be viewed as a special case of DOM parsing. In another case, the annotations, organized into a semantic layer,[2] are stored and managed separately from the web pages, so the scrapers can retrieve data schema and instructions from this layer before scraping the pages.

Computer vision web-page analyzers. There are efforts using machine learning and computer vision that attempt to identify and extract information from web pages by interpreting pages visually as a human being might.[3]

Legal issues

Web scraping may be against the terms of use of some websites. The enforceability of these terms is unclear.[4] While outright duplication of original expression will in many cases be illegal, in the United States the courts ruled in Feist Publications v. Rural Telephone Service that duplication of facts is allowable. U.S. courts have acknowledged that users of “scrapers” or “robots” may be held liable for committing trespass to chattels,[5][6] which involves a computer system itself being considered personal property upon which the user of a scraper is trespassing. The best known of these cases, eBay v. Bidder’s Edge, resulted in an injunction ordering Bidder’s Edge to stop accessing, collecting, and indexing auctions from the eBay web site. This case involved automatic placing of bids, known as auction sniping. However, in order to succeed on a claim of trespass to chattels, the plaintiff must demonstrate that the defendant intentionally and without authorization interfered with the plaintiff’s possessory interest in the computer system and that the defendant’s unauthorized use caused damage to the plaintiff. Not all cases of web spidering brought before the courts have been considered trespass to chattels.[7]

One of the first major tests of screen scraping involved American Airlines, and a firm called FareChase.[8] AA successfully obtained an injunction from a Texas trial court, stopping FareChase from selling software that enables users to compare online fares if it also searches AA’s website. The airline argued that FareChase’s websearch software trespassed on AA’s servers when it collected the publicly available data. FareChase filed an appeal in March 2003. By June, FareChase and AA agreed to settle and the appeal was dropped.[9]

Southwest Airlines has also challenged screen-scraping practices, and has involved both FareChase and another firm, Outtask, in a legal claim. Southwest Airlines charged that the screen-scraping is Illegal since it is an example of “Computer Fraud and Abuse” and has led to “Damage and Loss” and “Unauthorized Access” of Southwest’s site. It also constitutes “Interference with Business Relations”, “Trespass”, and “Harmful Access by Computer”. They also claimed that screen-scraping constitutes what is legally known as “Misappropriation and Unjust Enrichment”, as well as being a breach of the web site’s user agreement. Outtask denied all these claims, claiming that the prevailing law in this case should be US Copyright law, and that under copyright, the pieces of information being scraped would not be subject to copyright protection. Although the cases were never resolved in the Supreme Court of the United States, FareChase was eventually shuttered by parent company Yahoo!, and Outtask was purchased by travel expense company Concur.[10]

Although these are early scraping decisions, and the theories of liability are not uniform, it is difficult to ignore a pattern emerging that the courts are prepared to protect proprietary content on commercial sites from uses which are undesirable to the owners of such sites. However, the degree of protection for such content is not settled, and will depend on the type of access made by the scraper, the amount of information accessed and copied, the degree to which the access adversely affects the site owner’s system and the types and manner of prohibitions on such conduct.[11]

While the law in this area becomes more settled, entities contemplating using scraping programs to access a public web site should also consider whether such action is authorized by reviewing the terms of use and other terms or notices posted on or made available through the site. In the latest ruling in the Cvent, Inc. v. Eventbrite, Inc. In the United States district court for the eastern district of Virginia, the court ruled that the terms of use should be brought to the users’ attention In order for a browse wrap contract or license to be enforced.[12]

In the plaintiff’s web site during the period of this trial the terms of use link is displayed among all the links of the site, at the bottom of the page as most sites on the internet. This ruling contradicts the Irish ruling described below. The court also rejected the plaintiff’s argument that the browse wrap restrictions were enforceable in view of Virginia’s adoption of the Uniform Computer Information Transactions Act (UCITA)—a uniform law that many believed was in favor on common browse wrap contracting practices.[13]

Outside of the United States, in February 2006, the Danish Maritime and Commercial Court (Copenhagen) ruled that systematic crawling, indexing, and deep linking by portal site ofir.dk of real estate site Home.dk does not conflict with Danish law or the database directive of the European Union.[14]

In 2009 Facebook won one of the first copyright suits against a known web scraper. This laid the groundwork for numerous lawsuits that tie any web scraping with a direct copyright violation and very clear monetary damages. The most recent case being AP v Meltwater, where the courts stripped what is referred to as fair use on the internet.[15]

In a February 2010 case complicated by matters of jurisdiction, Ireland’s An Ard-Chúirt delivered a verdict that illustrates the inchoate state of developing case law. In the case of Ryanair Ltd v Billigfluege.de GmbH, Ireland’s High Court ruled Ryanair’s “click-wrap” agreement to be legally binding. In contrast to the findings of the United States District Court Eastern District of Virginia and those of the Danish Maritime and Commercial Court, Mr. Justice Michael Hanna ruled that the hyperlink to Ryanair’s terms and conditions was plainly visible, and that placing the onus on the user to agree to terms and conditions in order to gain access to online services is sufficient to comprise a contractual relationship. Where here may be another legal issue, of data non-valid or incorrect information, as most of data mixed with lot of “Junk” or “Spam”. Where with respect to data ware house there are: may be limitation of query interface, inconsistence output and rapid change in data by admin without any notice. [16] The decision is under appeal in Ireland’s Supreme Court, the Cúirt Uachtarach na hÉireann.[17]

In Australia, the Spam Act 2003 outlaws some forms of web harvesting, although this only applies to email addresses.[18][19]

Technical measures to stop bots

The administrator of a website can use various measures to stop or slow a bot. Some techniques include:

Blocking an IP address. This will also block all browsing from that address.

Bots sometimes declare who they are (using user agent strings) and can be blocked on that basis (using robots.txt); ‘googlebot‘ is an example. Some bots make no distinction between themselves and a human browser.

Bots can be blocked by excess traffic monitoring.

Bots can sometimes be blocked with tools to verify that it is a real person accessing the site, like a CAPTCHA. Bots are sometimes coded to explicitly break specific Captcha patterns.

Glenn Greenwald Piers Morgan On XKEYSCORE. Greenwald

XKeyscore- A New Level of Invasive NSA Data Spying

Leading members of the House Armed Services Committee emerged from a classified briefing on the Edward Snowden leaks Wednesday afternoon “shocked” at the amount of information he reportedly leaked beyond the NSA surveillance programs.

Rep. Mac Thornberry (R-Texas), chairman of the Armed Service panel’s Intelligence, Emerging Threats and Capabilities Subcommittee and also a member of the House Intelligence Committee, said the briefing on the defense consequences of Snowden’s leaks was “very highly classified,” and therefore details couldn’t be discussed.

Thornberry did say that lawmakers “left the briefing disturbed and angered” after hearing that the leaks by the former Booz Allen Hamilton employee “went well beyond programs associated with the NSA and data collection.”

He characterized the leaks as so severe that they “compromise military capability and defense of the country” and “could cost lives” — while they “will certainly cost billions to repair.”

“His actions were espionage, plain and simple,” Thornberry said.

Armed Services Chairman Buck McKeon (R-Calif.) read his statement rather than making comments on the fly “because of the seriousness of this issue and the sensitivity” of the information they’d just heard.

“Ed Snowden isn’t a whistleblower; he’s a traitor,” McKeon said.

No matter what opinion people hold of the data collection programs, he added, people should be “shocked and outraged to find that a substantial amount of the information has nothing to do with the NSA.”

“He’s given our enemies an edge and put American lives at risk,” said the chairman.

McKeon said Americans should “demand” that Snowden be “brought to justice” and not be extended any sort of deal by the government.

NSA and the One Trillion Dollar scam [Empire]

The Truth About Edward Snowden

A White House-appointed task force has proposed a series of curbs on key National Security Agency surveillance operations exposed by Edward Snowden. On Thursday, the panel recommended the NSA halt its bulk collection of billions of U.S. phone call records, citing ‘potential risks to public trust, personal privacy, and civil liberty.’ The panel says telecommunications providers or a private third party should store the records instead. The panel also calls for banning the NSA from ‘undermining encryption’ and criticizes its use of computer programming flaws to mount cyber-attacks. And it backs the creation of an independent review board to monitor government programs for potential violations of civil liberties. We discuss the panel’s findings with two guests: Ben Wizner, Snowden’s legal advisor and director o

Construction of the NCI supercomputer – timelapse video

NSA’s Largest Spy Center Located in Utah (What you need to know)

What You Should Know About The New NSA Utah Data Center

How NSA Spys On You With Utah Data Center Super Computer

The End of Internet Privacy? Glenn Greenwald On Secret NSA Program to Crack Online Encryption

NSA and the One Trillion Dollar scam [Empire]

NSA seeks to build quantum computer that could crack most types of encryption

In room-size metal boxes, secure against electromagnetic leaks, the National Security Agency is racing to build a computer that could break nearly every kind of encryption used to protect banking, medical, business and government records around the world.

According to documents provided by former NSA contractor Edward Snowden, the effort to build “a cryptologically useful quantum computer” — a machine exponentially faster than classical computers — is part of a $79.7 million research program titled, “Penetrating Hard Targets.” Much of the work is hosted under classified contracts at a laboratory in College Park.

The development of a quantum computer has long been a goal of many in the scientific community, with revolutionary implications for fields like medicine as well as for the NSA’s code-breaking mission. With such technology, all forms of public key encryption would be broken, including those used on many secure Web sites as well as the type used to protect state secrets.

Physicists and computer scientists have long speculated whether the NSA’s efforts are more advanced than those of the best civilian labs. Although the full extent of the agency’s research remains unknown, the documents provided by Snowden suggest that the NSA is no closer to success than others in the scientific community.

“It seems improbable that the NSA could be that far ahead of the open world without anybody knowing it,” said Scott Aaronson, an associate professor of electrical engineering and computer science at MIT.

The NSA appears to regard itself as running neck and neck with quantum computing labs sponsored by the European Union and the Swiss government, with steady progress but little prospect of an immediate breakthrough.

“The geographic scope has narrowed from a global effort to a discrete focus on the European Union and Switzerland,” one NSA document states.

Seth Lloyd, professor of quantum mechanical engineering at MIT, said the NSA’s focus is not misplaced. “The E.U. and Switzerland have made significant advances over the last decade and have caught up to the U.S. in quantum computing technology,” he said.

The NSA declined to comment for this story.

The documents, however, indicate that the agency carries out some of its research in large, shielded rooms known as Faraday cages, which are designed to prevent electromagnetic energy from coming in or out. Those, according to one brief description, are required “to keep delicate quantum computing experiments running.”

The basic principle underlying quantum computing is known as “quantum superposition,” the idea that an object simultaneously exists in all states. A classical computer uses binary bits, which are either zeroes or ones. A quantum computer uses quantum bits, or qubits, which are simultaneously zero and one.

This seeming impossibility is part of the mystery that lies at the heart of quantum theory, which even theoretical physicists say no one completely understands.

“If you think you understand quantum mechanics, you don’t understand quantum mechanics,” said the late Nobel laureate Richard Feynman, who is widely regarded as the pioneer in quantum computing.

Here’s how it works, in theory: While a classical computer, however fast, must do one calculation at a time, a quantum computer can sometimes avoid having to make calculations that are unnecessary to solving a problem. That allows it to home in on the correct answer much more quickly and efficiently.

Quantum computing is so difficult to attain because of the fragile nature of such computers. In theory, the building blocks of such a computer might include individual atoms, photons or electrons. To maintain the quantum nature of the computer, these particles would need to be carefully isolated from their external environments.

“Quantum computers are extremely delicate, so if you don’t protect them from their environment, then the computation will be useless,” said Daniel Lidar, a professor of electrical engineering and the director of the Center for Quantum Information Science and Technology at the University of Southern California.

A working quantum computer would open the door to easily breaking the strongest encryption tools in use today, including a standard known as RSA, named for the initials of its creators. RSA scrambles communications, making them unreadable to anyone but the intended recipient, without requiring the use of a shared password. It is commonly used in Web browsers to secure financial transactions and in encrypted e-mails. RSA is used because of the difficulty of factoring the product of two large prime numbers. Breaking the encryption involves finding those two numbers. This cannot be done in a reasonable amount of time on a classical computer.

In 2009, computer scientists using classical methods were able to discover the primes within a 768-bit number, but it took almost two years and hundreds of computers to factor it. The scientists estimated that it would take 1,000 times longer to break a 1,024-bit encryption key, which is commonly used for online transactions.

A large-scale quantum computer, however, could theoretically break a 1,024-bit encryption much faster. Some leading Internet companies are moving to 2,048-bit keys, but even those are thought to be vulnerable to rapid decryption with a quantum computer.

Quantum computers have many applications for today’s scientific community, including the creation of artificial intelligence. But the NSA fears the implications for national security.

“The application of quantum technologies to encryption algorithms threatens to dramatically impact the US government’s ability to both protect its communications and eavesdrop on the communications of foreign governments,” according to an internal document provided by Snowden.

Experts are not sure how feasible a quantum computer is in the near future. A decade ago, some experts said that developing a large quantum computer was likely 10 to 100 years in the future. Five years ago, Lloyd said the goal was at least 10 years away.

Last year, Jeff Forshaw, a professor at the University of Manchester, told Britain’s Guardian newspaper, “It is probably too soon to speculate on when the first full-scale quantum computer will be built but recent progress indicates that there is every reason to be optimistic.”

“I don’t think we’re likely to have the type of quantum computer the NSA wants within at least five years, in the absence of a significant breakthrough maybe much longer,” Lloyd told the Post in a recent interview.

However, some companies claim to already be producing small quantum computers. A Canadian company, D-Wave Systems , says it has been making quantum computers since 2009. In 2012, it sold a $10 million version to Google, NASA and the Universities Space Research Association, according to news reports.

That quantum computer, however, would never be useful for breaking public key encryption like RSA.

“Even if everything they’re claiming is correct, that computer, by its design, cannot run Shor’s algorithm,” said Matthew Green, a research professor at the Johns Hopkins Information Security Institute, referring to the algorithm that could be used to break encryption like RSA.

Experts believe that one of the largest hurdles to breaking encryption with a quantum computer is building a computer with enough qubits, which is difficult given the very fragile state of quantum computers. By the end of September, the NSA expected to be able to have some basic building blocks, which it described in a document as “dynamical decoupling and complete quantum control on two semiconductor qubits.”

“That’s a great step, but it’s a pretty small step on the road to building a large-scale quantum computer,” Lloyd said.

A quantum computer capable of breaking cryptography would need hundreds or thousands more qubits than that.

The budget for the National Intelligence Program, commonly referred to as the “black budget,” details the “Penetrating Hard Targets” project and noted that this step “will enable initial scaling towards large systems in related and follow-on efforts.”

Another project, called the “Owning the Net,” is using quantum research to support the creation of new quantum-based attacks on encryptions like RSA, documents show.

“The irony of quantum computing is that if you can imagine someone building a quantum computer that can break encryption a few decades into the future, then you need to be worried right now,” Lidar said.

NSA Interception: Spy malware installed on laptops bought online

Glenn Greenwald Keynote on 30c3

The Tor Network [30c3] (with Jacob Applebaum)

NSA Spying Project Prism Glenn Greenwald Interview

Glenn Greenwald: The NSA Can “Literally Watch Every Keystroke You Make”

Spiegel has revealed new details about a secretive hacking unit inside the National Security Agency called the Office of Tailored Access Operations, or TAO. The unit was created in 1997 to hack into global communications traffic. Hackers inside the TAO have developed a way to break into computers running Microsoft Windows by gaining passive access to machines when users report program crashes to Microsoft. In addition, with help from the CIA and FBI, the NSA has the ability to intercept computers and other electronic accessories purchased online in order to secretly insert spyware and components that can provide backdoor access for the intelligence agencies. American Civil Liberties Union Deputy Legal Director Jameel Jaffer and journalist Glenn Greenwald join us to discuss the latest revelations, along with the future of Edward Snowden, who has recently offered to assist U.S. targets Germany and Brazil with their respective probes into NSA spying.

Glenn Greenwald and Ruth Marcus Get in Explosive Exchange over Snowden and ‘Horrible’ D.C. Media

How The NSA Hacks Your iPhone (Presenting DROPOUT JEEP)

Following up on the latest stunning revelations released yesterday by German Spiegel which exposed the spy agency’s 50 page catalog of “backdoor penetration techniques“, today during a speech given by Jacob Applebaum (@ioerror) at the 30th Chaos Communication Congress, a new bombshell emerged: specifically the complete and detailed description of how the NSA bugs, remotely, your iPhone. The way the NSA accomplishes this is using software known as Dropout Jeep, which it describes as follows: “DROPOUT JEEP is a software implant for the Apple iPhone that utilizes modular mission applications to provide specific SIGINT functionality. This functionality includes the ability to remotely push/pull files from the device. SMS retrieval, contact list retrieval, voicemail, geolocation, hot mic, camera capture, cell tower location, etc. Command, control and data exfiltration can occur over SMS messaging or a GPRS data connection. All communications with the implant will be covert and encrypted.”

The flowchart of how the NSA makes your iPhone its iPhone is presented below:

NSA ROC operator

Load specified module

Send data request

iPhone accepts request

Retrieves required SIGINT data

Encrypt and send exfil data

Rinse repeat

And visually:

What is perhaps just as disturbing is the following rhetorical sequence from Applebaum:

“Do you think Apple helped them build that? I don’t know. I hope Apple will clarify that. Here’s the problem: I don’t really believe that Apple didn’t help them, I can’t really prove it but [the NSA] literally claim that anytime they target an iOS device that it will succeed for implantation. Either they have a huge collection of exploits that work against Apple products, meaning that they are hoarding information about critical systems that American companies produce and sabotaging them, or Apple sabotaged it themselves. Not sure which one it is. I’d like to believe that since Apple didn’t join the PRISM program until after Steve Jobs died, that maybe it’s just that they write shitty software. We know that’s true.”

Or, Apple’s software is hardly “shitty” even if it seems like that to the vast majority of experts (kinda like the Fed’s various programs), and in fact it achieves precisely what it is meant to achieve.

How ironic would it be if Blackberry, left for dead by virtually everyone, began marketing its products as the only smartphone that does not allow the NSA access to one’s data (and did so accordingly). Since pretty much everything else it has tried has failed, we don’t see the downside to this hail mary attempt to strike back at Big Brother and maybe make some money, by doing the right thing for once.

We urge readers to watch the full one hour speech by Jacob Applebaum to realize just how massive Big Brother truly is, but those who want to just listen to the section on Apple can do so beginning 44 minutes 30 seconds in the presentation below.

Top secret documents submitted to the court that oversees surveillance by US intelligence agencies show the judges have signed off on broad orders which allow the NSA to make use of information “inadvertently” collected from domestic US communications without a warrant.

The Guardian is publishing in full two documents submitted to the secret Foreign Intelligence Surveillance Court (known as the Fisa court), signed by Attorney General Eric Holder and stamped 29 July 2009. They detail the procedures the NSA is required to follow to target “non-US persons” under its foreign intelligence powers and what the agency does to minimize data collected on US citizens and residents in the course of that surveillance.

The documents show that even under authorities governing the collection of foreign intelligence from foreign targets, US communications can still be collected, retained and used.

The procedures cover only part of the NSA’s surveillance of domestic US communications. The bulk collection of domestic call records, as first revealed by the Guardian earlier this month, takes place under rolling court orders issued on the basis of a legal interpretation of a different authority, section 215 of the Patriot Act.

The Fisa court’s oversight rolehas been referenced many times by Barack Obama and senior intelligence officials as they have sought to reassure the public about surveillance, but the procedures approved by the court have never before been publicly disclosed.

The top secret documents published today detail the circumstances in which data collected on US persons under the foreign intelligence authority must be destroyed, extensive steps analysts must take to try to check targets are outside the US, and reveals how US call records are used to help remove US citizens and residents from data collection.

However, alongside those provisions, the Fisa court-approved policies allow the NSA to:

• Keep data that could potentially contain details of US persons for up to five years;

• Retain and make use of “inadvertently acquired” domestic communications if they contain usable intelligence, information on criminal activity, threat of harm to people or property, are encrypted, or are believed to contain any information relevant to cybersecurity;

• Access the content of communications gathered from “U.S. based machine[s]” or phone numbers in order to establish if targets are located in the US, for the purposes of ceasing further surveillance.

The broad scope of the court orders, and the nature of the procedures set out in the documents, appear to clash with assurances from President Obama and senior intelligence officials that the NSA could not access Americans’ call or email information without warrants.

The documents also show that discretion as to who is actually targeted under the NSA’s foreign surveillance powers lies directly with its own analysts, without recourse to courts or superiors – though a percentage of targeting decisions are reviewed by internal audit teams on a regular basis.

Since the Guardian first revealed the extent of the NSA’s collection of US communications, there have been repeated calls for the legal basis of the programs to be released. On Thursday, two US congressmen introduced a bill compelling the Obama administration to declassify the secret legal justifications for NSA surveillance.

The disclosure bill, sponsored by Adam Schiff, a California Democrat, and Todd Rokita, an Indiana Republican, is a complement to one proposed in the Senate last week. It would “increase the transparency of the Fisa Court and the state of the law in this area,” Schiff told the Guardian. “It would give the public a better understanding of the safeguards, as well as the scope of these programs.”

Section 702 of the Fisa Amendments Act (FAA), which was renewed for five years last December, is the authority under which the NSA is allowed to collect large-scale data, including foreign communications and also communications between the US and other countries, provided the target is overseas.

FAA warrants are issued by the Fisa court for up to 12 months at a time, and authorise the collection of bulk information – some of which can include communications of US citizens, or people inside the US. To intentionally target either of those groups requires an individual warrant.

One-paragraph order

One such warrant seen by the Guardian shows that they do not contain detailed legal rulings or explanation. Instead, the one-paragraph order, signed by a Fisa court judge in 2010, declares that the procedures submitted by the attorney general on behalf of the NSA are consistent with US law and the fourth amendment.

Those procedures state that the “NSA determines whether a person is a non-United States person reasonably believed to be outside the United States in light of the totality of the circumstances based on the information available with respect to that person, including information concerning the communications facility or facilities used by that person”.

It includes information that the NSA analyst uses to make this determination – including IP addresses, statements made by the potential target, and other information in the NSA databases, which can include public information and data collected by other agencies.

Where the NSA has no specific information on a person’s location, analysts are free to presume they are overseas, the document continues.

“In the absence of specific information regarding whether a target is a United States person,” it states “a person reasonably believed to be located outside the United States or whose location is not known will be presumed to be a non-United States person unless such person can be positively identified as a United States person.”

If it later appears that a target is in fact located in the US, analysts are permitted to look at the content of messages, or listen to phone calls, to establish if this is indeed the case.

Referring to steps taken to prevent intentional collection of telephone content of those inside the US, the document states: “NSA analysts may analyze content for indications that a foreign target has entered or intends to enter the United States. Such content analysis will be conducted according to analytic and intelligence requirements and priorities.”

Details set out in the “minimization procedures”, regularly referred to in House and Senate hearings, as well as public statements in recent weeks, also raise questions as to the extent of monitoring of US citizens and residents.

NSA minimization procedures signed by Holder in 2009 set out that once a target is confirmed to be within the US, interception must stop immediately. However, these circumstances do not apply to large-scale data where the NSA claims it is unable to filter US communications from non-US ones.

The NSA is empowered to retain data for up to five years and the policy states “communications which may be retained include electronic communications acquired because of limitations on the NSA’s ability to filter communications”.

Even if upon examination a communication is found to be domestic – entirely within the US – the NSA can appeal to its director to keep what it has found if it contains “significant foreign intelligence information”, “evidence of a crime”, “technical data base information” (such as encrypted communications), or “information pertaining to a threat of serious harm to life or property”.

Domestic communications containing none of the above must be destroyed. Communications in which one party was outside the US, but the other is a US-person, are permitted for retention under FAA rules.

The minimization procedure adds that these can be disseminated to other agencies or friendly governments if the US person is anonymised, or including the US person’s identity under certain criteria.

A separate section of the same document notes that as soon as any intercepted communications are determined to have been between someone under US criminal indictment and their attorney, surveillance must stop. However, the material collected can be retained, if it is useful, though in a segregated database:

“The relevant portion of the communication containing that conversation will be segregated and the National Security Division of the Department of Justice will be notified so that appropriate procedures may be established to protect such communications from review or use in any criminal prosecution, while preserving foreign intelligence information contained therein,” the document states.

In practice, much of the decision-making appears to lie with NSA analysts, rather than the Fisa court or senior officials.

A transcript of a 2008 briefing on FAA from the NSA’s general counsel sets out how much discretion NSA analysts possess when it comes to the specifics of targeting, and making decisions on who they believe is a non-US person. Referring to a situation where there has been a suggestion a target is within the US.

“Once again, the standard here is a reasonable belief that your target is outside the United States. What does that mean when you get information that might lead you to believe the contrary? It means you can’t ignore it. You can’t turn a blind eye to somebody saying: ‘Hey, I think so and so is in the United States.’ You can’t ignore that. Does it mean you have to completely turn off collection the minute you hear that? No, it means you have to do some sort of investigation: ‘Is that guy right? Is my target here?” he says.

“But, if everything else you have says ‘no’ (he talked yesterday, I saw him on TV yesterday, even, depending on the target, he was in Baghdad) you can still continue targeting but you have to keep that in mind. You can’t put it aside. You have to investigate it and, once again, with that new information in mind, what is your reasonable belief about your target’s location?”

The broad nature of the court’s oversight role, and the discretion given to NSA analysts, sheds light on responses from the administration and internet companies to the Guardian’s disclosure of the PRISM program. They have stated that the content of online communications is turned over to the NSA only pursuant to a court order. But except when a US citizen is specifically targeted, the court orders used by the NSA to obtain that information as part of Prism are these general FAA orders, not individualized warrants specific to any individual.

Once armed with these general orders, the NSA is empowered to compel telephone and internet companies to turn over to it the communications of any individual identified by the NSA. The Fisa court plays no role in the selection of those individuals, nor does it monitor who is selected by the NSA.

The NSA’s ability to collect and retain the communications of people in the US, even without a warrant, has fuelled congressional demands for an estimate of how many Americans have been caught up in surveillance.

Two US senators, Ron Wyden and Mark Udall – both members of the Senate intelligence committee – have been seeking this information since 2011, but senior White House and intelligence officials have repeatedly insisted that the agency is unable to gather such statistics.

Background Articles and Videos

Stellar Wind

Stellar Wind was the open secret code name for four surveillance programs by the United States National Security Agency (NSA) during the presidency of George W. Bush and revealed by Thomas Tamm to The New York Times reporters James Risen and Eric Lichtblau.[1] The operation was approved by President George W. Bush shortly after the September 11 attacks in 2001.[2] Stellar Wind was succeeded during the presidency of Barack Obama by four major lines of intelligence collection in the territorial United States, together capable of spanning the full range of modern telecommunications.[3]

The program’s activities involved data mining of a large database of the communications of American citizens, including e-mail communications, phone conversations, financial transactions, and Internet activity.[1] William Binney, a retired Technical Leader with the NSA, discussed some of the architectural and operational elements of the program at the 2012 Chaos Communication Congress.[4]

There were internal disputes within the Justice Department about the legality of the program, because data are collected for large numbers of people, not just the subjects of Foreign Intelligence Surveillance Act (FISA) warrants.[4]

During the Bush Administration, the Stellar Wind cases were referred to by FBI agents as “pizza cases” because many seemingly suspicious cases turned out to be food takeout orders. According to Mueller, approximately 99 percent of the cases led nowhere, but “it’s that other 1% that we’ve got to be concerned about”.[2] One of the known uses of these data were the creation of suspicious activity reports, or “SARS”, about people suspected of terrorist activities. It was one of these reports that revealed former New York governor Eliot Spitzer’s use of prostitutes, even though he was not suspected of terrorist activities.[1]

In March 2012 Wired magazine published “The NSA Is Building the Country’s Biggest Spy Center (Watch What You Say)” talking about a vast new NSA facility in Utah and says “For the first time, a former NSA official has gone on the record to describe the program, codenamed Stellar Wind, in detail,” naming the official William Binney, a former NSA code breaker. Binney went on to say that the NSA had highly secured rooms that tap into major switches, and satellite communications at both AT&T and Verizon.[5] The article suggested that the otherwise dispatched Stellar Wind is actually an active program.

PRISM

PRISM is a clandestine national security electronic surveillance program operated by the United States National Security Agency (NSA) since 2007.[1][2][3][Notes 1]PRISM is a government codename for a data collection effort known officially as US-984XN.[8][9] It is operated under the supervision of the United States Foreign Intelligence Surveillance Court pursuant to the Foreign Intelligence Surveillance Act (FISA).[10] The existence of the program was leaked by NSA contractor Edward Snowden and published by The Guardian and The Washington Post on June 6, 2013.

A document included in the leak indicated that the PRISM SIGAD was “the number one source of raw intelligence used for NSA analytic reports.”[11] The President’s Daily Brief, an all-source intelligence product, cited PRISM data as a source in 1,477 items in 2012.[12] The leaked information came to light one day after the revelation that the United States Foreign Intelligence Surveillance Court had been requiring the telecommunications company Verizon to turn over to the NSA logs tracking all of its customers’ telephone calls on an ongoing daily basis.[13][14]

According to the Director of National Intelligence James Clapper, PRISM cannot be used to intentionally target any Americans or anyone in the United States. Clapper said a special court, Congress, and the executive branch oversee the program and extensive procedures ensure the acquisition, retention, and dissemination of data accidentally collected about Americans is kept to a minimum.[15] Clapper issued a statement and “fact sheet”[16] to correct what he characterized as “significant misimpressions” in articles by The Washington Post and The Guardian newspapers.[17]

History

Slide showing that much of the world’s communications flow through the US

Details of information collected via PRISM

PRISM is a “Special Source Operation” in the tradition of NSA’s intelligence alliances with as many as 100 trusted U.S. companies since the 1970s.[1] A prior program, the Terrorist Surveillance Program, was implemented in the wake of the September 11 attacks under the George W. Bush Administration but was widely criticized and had its legality questioned, because it was conducted without approval of the Foreign Intelligence Surveillance Court (FISC).[18][19][20][21] PRISM was authorized by an order of the FISC.[11] Its creation was enabled by the Protect America Act of 2007 under President Bush and the FISA Amendments Act of 2008, which legally immunized private companies that cooperated voluntarily with US intelligence collection and was renewed by Congress under President Obama in 2012 for five years until December 2017.[2][22] According to The Register, the FISA Amendments Act of 2008 “specifically authorizes intelligence agencies to monitor the phone, email, and other communications of U.S. citizens for up to a week without obtaining a warrant” when one of the parties is outside the U.S.[22]

PRISM was first publicly revealed on June 6, 2013, after classified documents about the program were leaked to The Washington Post and The Guardian by American Edward Snowden.[2][1] The leaked documents included 41 PowerPoint slides, four of which were published in news articles.[1][2] The documents identified several technology companies as participants in the PRISM program, including (date of joining PRISM in parentheses) Microsoft (2007), Yahoo! (2008), Google (2009), Facebook (2009), Paltalk (2009), YouTube (2010), AOL (2011), Skype (2011), and Apple (2012).[23] The speaker’s notes in the briefing document reviewed by The Washington Post indicated that “98 percent of PRISM production is based on Yahoo, Google and Microsoft.”[1]

The slide presentation stated that much of the world’s electronic communications pass through the United States, because electronic communications data tend to follow the least expensive route rather than the most physically direct route, and the bulk of the world’s internet infrastructure is based in the United States.[11] The presentation noted that these facts provide United States intelligence analysts with opportunities for intercepting the communications of foreign targets as their electronic data pass into or through the United States.[2][11]

According to The Washington Post, the intelligence analysts search PRISM data using terms intended to identify suspicious communications of targets whom the analysts suspect with at least 51 percent confidence to not be United States citizens, but in the process, communication data of some United States citizens are also collected unintentionally.[1] Training materials for analysts tell them that while they should periodically report such accidental collection of non-foreign United States data, “it’s nothing to worry about.”[1]

Response from companies

The original Washington Post and Guardian articles reporting on PRISM noted that one of the leaked briefing documents said PRISM involves collection of data “directly from the servers” of several major internet services providers.[2][1]

Initial Public Statements

Corporate executives of several companies identified in the leaked documents told The Guardian that they had no knowledge of the PRISM program in particular and also denied making information available to the government on the scale alleged by news reports.[2][24] Statements of several of the companies named in the leaked documents were reported by TechCrunch and The Washington Post as follows:[25][26]

Slide listing companies and the date that PRISM collection began

Microsoft: “We provide customer data only when we receive a legally binding order or subpoena to do so, and never on a voluntary basis. In addition we only ever comply with orders for requests about specific accounts or identifiers. If the government has a broader voluntary national security program to gather customer data we don’t participate in it.”[25]

Yahoo!: “Yahoo! takes users’ privacy very seriously. We do not provide the government with direct access to our servers, systems, or network.”[25] “Of the hundreds of millions of users we serve, an infinitesimal percentage will ever be the subject of a government data collection directive.”[26]

Facebook: “We do not provide any government organization with direct access to Facebook servers. When Facebook is asked for data or information about specific individuals, we carefully scrutinize any such request for compliance with all applicable laws, and provide information only to the extent required by law.”[25]

Google: “Google cares deeply about the security of our users’ data. We disclose user data to government in accordance with the law, and we review all such requests carefully. From time to time, people allege that we have created a government ‘back door’ into our systems, but Google does not have a backdoor for the government to access private user data.”[25] “[A]ny suggestion that Google is disclosing information about our users’ Internet activity on such a scale is completely false.”[26]

Apple: “We have never heard of PRISM. We do not provide any government agency with direct access to our servers, and any government agency requesting customer data must get a court order.”[27]

Dropbox: “We’ve seen reports that Dropbox might be asked to participate in a government program called PRISM. We are not part of any such program and remain committed to protecting our users’ privacy.”[25]

In response to the technology companies’ denials of the NSA being able to directly access the companies’ servers, The New York Times reported that sources had stated the NSA was gathering the surveillance data from the companies using other technical means in response to court orders for specific sets of data.[13]The Washington Post suggested, “It is possible that the conflict between the PRISM slides and the company spokesmen is the result of imprecision on the part of the NSA author. In another classified report obtained by The Post, the arrangement is described as allowing ‘collection managers [to send] content tasking instructions directly to equipment installed at company-controlled locations,’ rather than directly to company servers.”[1] “[I]n context, ‘direct’ is more likely to mean that the NSA is receiving data sent to them deliberately by the tech companies, as opposed to intercepting communications as they’re transmitted to some other destination.[26]

“If these companies received an order under the FISA amendments act, they are forbidden by law from disclosing having received the order and disclosing any information about the order at all,” Mark Rumold, staff attorney at the Electronic Frontier Foundation, told ABC News.[28]

Slide showing two different sources of NSA data collection. The first source the fiber optic cables of the internet handled by the Upstream program and the second source the servers of major internet companies handled by PRISM.[29]

On May 28, 2013, Google was ordered by United States District Court Judge Susan Illston to comply with a National Security Letter issued by the FBI to provide user data without a warrant.[30] Kurt Opsahl, a senior staff attorney at the Electronic Frontier Foundation, in an interview with VentureBeat said, “I certainly appreciate that Google put out a transparency report, but it appears that the transparency didn’t include this. I wouldn’t be surprised if they were subject to a gag order.”[31]

The New York Times reported on June 7, 2013, that “Twitter declined to make it easier for the government. But other companies were more compliant, according to people briefed on the negotiations.”[32] The other companies held discussions with national security personnel on how to make data available more efficiently and securely.[32] In some cases, these companies made modifications to their systems in support of the intelligence collection effort.[32] The dialogues have continued in recent months, as General Martin Dempsey, the chairman of the Joint Chiefs of Staff, has met with executives including those at Facebook, Microsoft, Google and Intel.[32] These details on the discussions provide insight into the disparity between initial descriptions of the government program including a training slide which states “Collection directly from the servers”[29] and the companies’ denials.[32]

While providing data in response to a legitimate FISA request approved by FISC is a legal requirement, modifying systems to make it easier for the government to collect the data is not. This is why Twitter could legally decline to provide an enhanced mechanism for data transmission.[32] Other than Twitter, the companies were effectively asked to construct a locked mailbox and provide the key to the government, people briefed on the negotiations said.[32] Facebook, for instance, built such a system for requesting and sharing the information.[32] Google does not provide a lockbox system, but instead transmits required data by hand delivery or secure FTP.[33]

Post-PRISM Transparency Reports

In response to the publicity surrounding media reports of data-sharing, several companies requested permission to reveal more public information about the nature and scope of information provided in response to National Security requests.

On June 14, 2013, Facebook reported that the U.S. Government had authorized the communication of “about these numbers in aggregate, and as a range.” In a press release posted to their web site, Facebook reported, “For the six months ending December 31, 2012, the total number of user-data requests Facebook received from any and all government entities in the U.S. (including local, state, and federal, and including criminal and national security-related requests) – was between 9,000 and 10,000.” Facebook further reported that the requests impacted “between 18,000 and 19,000” user accounts, a “tiny fraction of one percent” of more than 1.1 billion active user accounts.[34]

Microsoft reported that for the same period, it received “between 6,000 and 7,000 criminal and national security warrants, subpoenas and orders affecting between 31,000 and 32,000 consumer accounts from U.S. governmental entities (including local, state and federal)” which impacted “a tiny fraction of Microsoft’s global customer base”.[35]

Google issued a statement criticizing the requirement that data be reported in aggregated form, stating that lumping national security requests with criminal request data would be “a step backwards” from its previous, more detailed practices on its site transparency report. The company said that it would continue to seek government permission to publish the number and extent of FISA requests.[36]

Response from United States government

Executive branch

Shortly after publication of the reports by The Guardian and The Washington Post, the United States Director of National Intelligence, James Clapper, on June 7 released a statement confirming that for nearly six years the government of the United States had been using large internet services companies such as Google and Facebook to collect information on foreigners outside the United States as a defense against national security threats.[13] The statement read in part, “The Guardian and The Washington Post articles refer to collection of communications pursuant to Section 702 of the Foreign Intelligence Surveillance Act. They contain numerous inaccuracies.”[37] He went on to say, “Section 702 is a provision of FISA that is designed to facilitate the acquisition of foreign intelligence information concerning non-U.S. persons located outside the United States. It cannot be used to intentionally target any U.S. citizen, any other U.S. person, or anyone located within the United States.”[37] Clapper concluded his statement by stating “The unauthorized disclosure of information about this important and entirely legal program is reprehensible and risks important protections for the security of Americans.”[37] On March 12, 2013, Clapper had told the United States Senate Select Committee on Intelligence that the NSA does “not wittingly” collect any type of data on millions or hundreds of millions of Americans.[38] In an NBC News interview, Clapper said he answered Senator Wyden’s question in the “least untruthful manner by saying no”.[39]

Clapper also stated that “the NSA collects the phone data in broad swaths, because collecting it (in) a narrow fashion would make it harder to identify terrorism-related communications. The information collected lets the government, over time, make connections about terrorist activities. The program doesn’t let the U.S. listen to people’s calls, but only includes information like call length and telephone numbers dialed.”[15]

On June 8, 2013, Clapper said “the surveillance activities published in The Guardian and The Washington Post are lawful and conducted under authorities widely known and discussed, and fully debated and authorized by Congress.”[40][10] The fact sheet described PRISM as “an internal government computer system used to facilitate the government’s statutorily authorized collection of foreign intelligence information from electronic communication service providers under court supervision, as authorized by Section 702 of the Foreign Intelligence Surveillance Act (FISA) (50 U.S.C. § 1881a).”[10]

The National Intelligence fact sheet further stated that “the United States Government does not unilaterally obtain information from the servers of U.S. electronic communication service providers. All such information is obtained with FISA Court approval and with the knowledge of the provider based upon a written directive from the Attorney General and the Director of National Intelligence.” It said that the Attorney General provides FISA Court rulings and semi-annual reports about PRISM activities to Congress, “provid[ing] an unprecedented degree of accountability and transparency.”[10]

The President of the United States, Barack Obama, said on June 7 “What you’ve got is two programs that were originally authorized by Congress, have been repeatedly authorized by Congress. Bipartisan majorities have approved them. Congress is continually briefed on how these are conducted. There are a whole range of safeguards involved. And federal judges are overseeing the entire program throughout.”[41] He also said, “You can’t have 100 percent security and then also have 100 percent privacy and zero inconvenience. You know, we’re going to have to make some choices as a society.”[41]

In separate statements, senior (not mentioned by name in source) Obama administration officials said that Congress had been briefed 13 times on the programs since 2009.[42]

Legislative branch

In contrast to their swift and forceful reactions the previous day to allegations that the government had been conducting surveillance of United States citizens’ telephone records, Congressional leaders initially had little to say about the PRISM program the day after leaked information about the program was published. Several lawmakers declined to discuss PRISM, citing its top-secret classification,[43] and others said that they had not been aware of the program.[44] After statements had been released by the President and the Director of National Intelligence, some lawmakers began to comment:

Senator John McCain (R-AZ)

June 9 “We passed the Patriot Act. We passed specific provisions of the act that allowed for this program to take place, to be enacted in operation,”[45]

June 9 “These programs are within the law”, “part of our obligation is keeping Americans safe”, “Human intelligence isn’t going to do it”.[46]

June 9 “Here’s the rub: the instances where this has produced good — has disrupted plots, prevented terrorist attacks, is all classified, that’s what’s so hard about this.”[47]

June 11 “It went fine…we asked him[ Keith Alexander ] to declassify things because it would be helpful (for people and lawmakers to better understand the intelligence programs).” “I’ve just got to see if the information gets declassified. I’m sure people will find it very interesting.”[48]

Senator Susan Collins (R-ME), member of Senate Intelligence Committee and past member of Homeland Security Committee

June 11 “I had, along with Joe Lieberman, a monthly threat briefing, but I did not have access to this highly compartmentalized information” and “How can you ask when you don’t know the program exists?”[49]

Representative John Boehner (R-OH), Speaker of the House of Representatives

June 9, “This is well beyond what the Patriot Act allows.”[51] “President Obama’s claim that ‘this is the most transparent administration in history’ has once again proven false. In fact, it appears that no administration has ever peered more closely or intimately into the lives of innocent Americans.”[51]

Representative Mike Rogers (R-MI), a Chairman of the Permanent Select Committee on Intelligence.

June 9 “One of the things that we’re charged with is keeping America safe and keeping our civil liberties and privacy intact. I think we have done both in this particular case,”[46]

June 9 “Within the last few years this program was used to stop a program, excuse me, to stop a terrorist attack in the United States we know that.