A man looks at his cellphone as he walks in downtown Madrid on Thursday after allegations emerged that Spain was a target for surveillance by the NSA. | AP

Omnipresent: The headquarters of the National Security Agency in Fort Meade, Maryland, are seen from the air in January 2010. | AFP-JIJI

Security vs. privacy: Demonstrators protest outside of the U.S. Capitol in Washington during an Oct. 26 rally to demand that Congress investigate the National Security Agency's mass surveillance programs. | AP

LONDON – The National Security Agency gathers intelligence to keep America safe. But leaked documents reveal the NSA’s dark side — and show an agency intent on exploiting the digital revolution to the full.

U.S. President Barack Obama hailed United Nations Secretary-General Ban Ki-moon as a “good friend” after the two had sat down at the White House in April to discuss the issues of the day: Syria and alleged chemical weapons attacks, North Korea, Israel-Palestine and climate change. But long before Ban’s limousine had even passed through the White House gates for the meeting, the U.S. government knew what the secretary-general was going to talk about, courtesy of the world’s biggest eavesdropping organization, the NSA.

One NSA document — leaked to The Guardian by whistle-blower Edward Snowden just a month after the meeting and reported in partnership with The New York Times — boasts how the spy agency had gained “access to U.N. secretary-general talking points prior to meeting with POTUS (president of the United States).” The White House declined to comment on whether Obama had read the talking points in advance of the meeting.

Spying on Ban and others at the U.N. is in contravention of international law, and the U.S., forced on the defensive last week over the Snowden leaks about worldwide snooping, ordered an end to surveillance of the organization, according to Reuters.

That the U.S. spied on Ban is no great surprise. What is a revealing is that the disclosure is listed in the NSA’s “top-secret” weekly report from around the world as an “operational highlight.”

It sits incongruously alongside other “operational highlights” from that week: details of an alleged Iranian chemical weapons program; communications relating to an alleged gas attack attack in Syria and a report about the Mexican drug cartel Los Zetas.

Bracketing the benign, U.S.-friendly Ban alongside drug traffickers and weapons in the Middle East and Central Asia points to a spy agency that has lost its sense of proportion.

The incident is consistent with the portrait of the NSA that emerges from the tens of thousands of documents leaked by Snowden. Page after page shows the NSA engaged in the kind of intelligence gathering it would be expected to carry out: eavesdropping on Taliban insurgents planning attacks in remote Afghanistan valleys, or listening in on hostage-takers in Colombia.

But the documents reveal, too, the darker side of the NSA. It is indiscriminate in the information it is collecting. Nothing appears to be too small for it. Nothing too trivial. Rivals, enemies, allies and friends — U.S. citizens and “non-Americans”— are all scooped up. The documents show the NSA, intent on exploiting the communications revolution to the full, developing ever more intrusive programs in pursuit of its ambition to have its surveillance net cover of the whole planet: total command of what the NSA refers to as the “digital battlefield.”

Mission creep

When the NSA was founded in 1952, its task was primarily to target the Soviet Union. And so it did, decade after decade, until the fall of the Berlin Wall in 1989 and the end of the Cold War soon afterward.

With the collapse of the Soviet Union, the NSA entered a decade of uncertainty. Morale slumped. The mood is caught in a document dated February 2001, only a few months before 9/11. In it, the agency admitted its capacity for intercepting electronic communications had been eroded during the ’90s.

“NSA’s workforce has been graying and shrinking. The operational tools have become antiquated and unable to handle the emerging signal structure,” it says.

“Ten years ago, we had a highly skilled workforce with intimate knowledge of the target and the tools to analyze the data.

“We have now reached the point of having a workforce where the majority of analysts have little-to-no experience.”

Tellingly, in the light of the attacks on New York and Washington six months later, the document complained about a lack of linguists and analysts covering Afghanistan. The pool of experts covering Afghanistan as a whole were the same that “assist NSA’s Office of Counter-terrorism in following the Taleban-Usama bin Laden relationship,” it said, using alternative spellings for the terrorist group and the former al-Qaida leader.

9/11 spurs shift

The attacks on New York and Washington ended the NSA’s decade of torpor. Suddenly, it found funding, and staff recruitment was no longer a problem. Since 9/11, expansion has been rapid. The NSA was one of the main beneficiaries of the doubling of the intelligence budget since the Sept. 11, 2001, attacks.

Its proposed budget allocation for 2013 is $10.8 billion, with 35,000 staff and bases in Georgia, Texas, Colorado, Hawaii and Utah adding to its headquarters at Fort Meade, Maryland. Its antennae can be found on the rooftops of 80 American embassies around the world.

It has large posts in the U.K., Australia and Japan, but also operates elsewhere, sometimes covertly. In one country, Americans are secretly present at a base where exposure of their presence would provoke a major diplomatic incident, as it is in breach of an international treaty signed by the NSA’s host nation. Agency staff visiting the base have to hide their real identities, posing as contractors working on communications equipment and carrying fake business cards to back up their story.

A PowerPoint briefing warns staff heading to this secret base: “Know your cover legend.” It urges them to “sanitize personal effects” and to send no postcards home. Nor should they take souvenirs home with them. The NSA briefing makes an exception for jewelry, because “most jewelry does not have markings identifying it” as coming from that country.

The NSA refers to the people it serves as “external customers”: the White House, the State Department, the CIA, the U.S. mission to the U.N., the Defense Intelligence Agency and others. Its remit for those customers has become ever more complex.

New ‘customers’

During the Cold War, the NSA mainly targeted state institutions: the political, military and intelligence structures of Russia and Eastern Europe. Today, the main targets — al-Qaida and its related groups — are much more diffuse and elusive.

The NSA sets out its mission statement in its current five-year plan. In it, the agency insists Sigint (signals intelligence, or the interception of communications) will adhere to the highest standards. “Sigint professionals must hold the moral high ground, even as terrorists or dictators seek to exploit our freedoms. Some of our adversaries will say or do anything to advance their cause; we will not.” Summing up the reason for its existence, it says: “Our mission is to answer questions about threatening activities that others mean to keep hidden.” But its actual scope goes well beyond that. It is hard to see where surveilling Ban Ki-moon or German Chancellor Angela Merkel fits into answering questions about “threatening activities.”

At a news conference in August, Obama defended the NSA and defined its role in narrow terms. He described the agency’s remit purely as counterterrorism. “We do not have an interest in doing anything other than that,” he said.

The remark was striking. Counterterrorism has been the justification for huge budget increases, but the agency is involved in much more than that. The NSA discloses in one leaked document that only 35 percent of available resources are dedicated to the “global war on terrorism.”

Obama later amended his statement. The NSA was not only engaged in counterterrorism, he said, but also cybersecurity and combating weapons of mass destruction. Even this does not begin to capture the sheer variety and reach of NSA operations.

Its own list of strategic targets includes: support for U.S. military in the field; gathering information about military technology; anticipating state instability; monitoring regional tensions; countering drug trafficking; gathering economic, political and diplomatic information; ensuring a steady and reliable energy supply for the U.S.; and ensuring U.S. economic advantage. It boasts that it can collect information from “virtually every country.”

NSA success stories

Hundreds of the documents show the NSA engaged in activities that would generally be applauded. One credits the NSA’s Texas base as intercepting 478 emails that helped to foil the “Jihad Jane” plot to kill Swedish artist Lars Vilks over his depiction of the Prophet Muhammad.

Another shows the NSA, during a deadly takeover of the Intercontinental Hotel in Kabul by the insurgent Haqqani network, able to listen in, minute-by-minute, to what the gunmen were saying.

There is an account, too, of the NSA’s part in disrupting a human-trafficking racket based in Fuzhou, China. It led to two arrests at New York’s Kennedy Airport. One of those arrested allegedly carried details of the smuggling routes in his pocket.

Remote surroundings might fool some into thinking they are beyond snooping. An alleged cocaine smuggler might have thought he was relatively safe aboard a yacht in the Caribbean. But he failed to take into account the fact that his partner, also on board, was chatting on Facebook, providing valuable information about the boat’s location and planned landfall; information intercepted by one of the NSA’s intelligence partners.

Nor is the Iranian leadership beyond reach. In 2009, the NSA was able to track almost every move made by Iran’s supreme leader, the Ayatollah Ali Khamenei, on a rare visit outside Tehran to the mountainous Kurdistan province.

The most valuable service the NSA has provided for America and its allies since 9/11 is in support of the military in Iraq and Afghanistan. A 2007 NSA file, called “State of the Enterprise,” is typical of many of the spy agency’s documents that list wartime successes.

“Specific results included the identification and location of a sniper targeting personnel inside Baghdad’s Green Zone; the confirmation that a CIA asset was operating as a potential ‘bad actor.’ ” Other intelligence agencies such as the CIA complain privately about the degree of cooperation from the NSA in sharing intelligence, but in the end, like most other intelligence agencies, it is generally thankful for it. There are complaints, too, from soldiers in the field that live information is not always transferred to them fast enough, but they, too, express gratitude for snippets passed on about potential Taliban attacks.

The NSA, according to one document, overheard a Taliban figure, Mullah Rahimullah Akhund, known on the U.S. military’s kill-or-capture list by the code name Objective Squiz Incinerator, instructing an associate to buy and organize components for a roadside bomb, suicide vests and a Japanese motorbike.

The appreciation of Americans and their allies in Afghanistan for such information is summed up in this letter back to headquarters: “You guys/gals probably have no idea how much we rely on your tool for enabling our CT (counterterrorism) capture operations in Afghanistan. It really does help us get our enemies off the playing field, so to speak.”

Casting a wide net

When the NSA, the CIA and other parts of the intelligence community spied illegally on U.S. antiwar protesters, civil rights leaders and trade unionists in the 1970s, there was at least a technical limitation of their actions. The difference today is that technological revolution allows them to spy on almost everyone.

The expansion in surveillance that accelerated under President George W. Bush has continued under Obama. And this growth has not been matched by any corresponding reform of the legal framework or political oversight.

While there are frequent warnings in the documents reminding NSA staff of rules for protecting the privacy of Americans, other documents show repeated violations. Such violations are almost inevitable given the way the NSA collects so much, the technology and analysts unable to distinguish between data on foreigners and U.S. citizens.

The NSA says in public it only collects a tiny percentage of Internet traffic, smaller than “a dime on a basketball court.” But there is a gulf between what the NSA says in public and what it says in documents, in which technicians and analysts express their glee at finding novel ways of cracking into electronic communications and expanding their reach in ever more imaginative ways.

The question critics of the NSA raise is: Just because it has the technical ability to do these things, should it? One document shows the NSA engaged in a massive snooping operation targeting a 2007 U.N. climate change conference in Bali.

Ban, speaking at the conference, which attracted thousands from around the world, described combating climate change as “the moral challenge of our generation.”

However, the NSA’s Australian base at Pine Gap was less interested in combating climate change than collecting the numbers of Indonesian security officials in case of a future emergency.

“Highlights include the compromise of the mobile phone number” for one senior Balinese official, an NSA report boasted. “Site efforts revealed previously unknown Indonesian communications networks and postured us to increase collection in the event of a crisis.” This effort-filled collection of the cellphone number falls under the category of information that spies have always gathered. The rationale is: Should there be an attack at the conference or some future outrage, such numbers could be valuable. The counterargument is that Indonesia is a friend of the U.S. and might be expected to share information in the event of an attack, so why does the NSA devote grand resources to harvesting such numbers?

One of the biggest criticisms of bulk data collection is that the agency cannot look at, let alone analyze, all the data it is collecting. One document echoed the problems the agency faced in 2001 when it lamented the lack of linguists pre-9/11. An officer, after checking some messages that might have been from a terrorist group, admitted: “Most of it is in Arabic or Farsi, so I can’t make much of it.”

The eyes have it

The NSA operates in close cooperation with four other English-speaking countries — the U.K., Canada, Australia and New Zealand — sharing raw intelligence, funding, technical systems and personnel. Their top level collective is known as the “Five Eyes.”

Beyond that, the NSA has other coalitions, although intelligence-sharing is more restricted for the additional partners: the “Nine Eyes,” which adds Denmark, France, the Netherlands and Norway; the “14 Eyes,” including Germany, Belgium, Italy, Spain and Sweden; and 41 Eyes, adding in others in the allied coalition in Afghanistan.

The exclusivity of the various coalitions grates with some, such as Germany, which is using the present controversy to seek an upgrade. Germany has long protested at its exclusion, not just from the elite Five Eyes but even from Nine Eyes. Minutes from the U.K. signals intelligence agency the Government Communications Headquarters note: “The NSA’s relationship with the French was not as advanced as GCHQ’s . . . the Germans were a little grumpy at not being invited to join the Nine Eyes group.”

The NSA’s closest ties are with GCHQ. Documents suggest the British contribution is significant. In a random selection of NSA documents monitoring weekly reports, the British agency is frequently listed alongside the U.S. agency’s biggest regional bases such as Texas and Georgia.

GCHQ operates a vast Internet tapping operation based on partnerships between the U.K. government and telecoms firms based in the U.K. and overseas. This allows the NSA to “touch” about 90 percent of the traffic crossing the U.K.

Given the U.K.’s location, this is a huge proportion of the Internet: Britain hosts one of the major trans-Atlantic Internet cables, as well as numerous cables connecting Europe and the Middle East. Each day, a quarter of all Net traffic traverses the U.K. The information collected and stored by the program, code named Tempora, is stored by GCHQ for up to a month, with NSA analysts granted direct access to the intelligence.

Lack of self-regulation

The NSA — in theory at least — operates inside a legal framework that requires warrants to target Americans. But the Foreign Intelligence Surveillance Act (FISA) court turns down few such requests.

GCHQ operates in an even looser environment. One GCHQ document, referring to U.K. oversight, says: “So far they have always found in our favor.” A GCHQ legal briefing suggests some of the distinctions stressed in policy documents and public statements by staff of both agencies may not be so rigorously enforced in practice. A lengthy legal training slide show includes several slides explaining the often-complex differences between content and “metadata” — including phone numbers dialed and the length of calls — which requires substantially different handling, especially under U.S. law.

However, the notes for the presentation say: “GCHQ policy is to treat it pretty much all the same, whether it’s content or metadata.” The blurred boundaries are acknowledged, too, in NSA documents, one of which states: “It is often unclear whether individual communication elements, particularly content-related metadata (CRI) — information derived from the message body — is content or metadata? For example, are email subject lines metadata or content? What about an email’s signature block or telephone numbers within a message? Questions like these are not necessarily clear-cut.”

Gaining access to the huge classified data banks appears to be relatively easy. Legal training sessions — which may also be required for access to information from Australian, Canadian, or New Zealand agencies — suggest that gaining credentials for data is relatively simple.

The sessions are often done as self-learning and self-assessment, with “multiple choice, open-book” tests done at the agent’s own desk on its “iLearn” system. Agents then copy and paste their passing result in order to gain access to the huge databases of communications.

The NSA, once the most secretive of the 16 U.S. intelligence agencies but now embarrassed as a result of Snowden, is facing more scrutiny that at any time since its founding, even more than during the domestic spying scandals of the 1970s.

It is being challenged in Congress. It is being challenged in the courts by an unholy alliance of the liberal American Civil Liberties Union and the right-leaning National Rifle Association. It is coming under pressure from Internet companies to be more transparent. And there is review panel announced by Obama in August. There is also pressure from Germany and France, Mexico and Brazil.

In spite of the furor, reforms may prove modest. The agency is hardly likely to easily relinquish its newfound capability of snooping almost everywhere.

In one of the leaked State of the Enterprise documents from 2007, an NSA staff member says: “The constant change in the world provides fertile ground for discovering new targets, technologies and networks that enable production of Sigint.” The official happily embraces this: “It’s becoming a cliche that a permanent state of change is the new standard. It is the world we live in — navigating through continuous white water.” It’s an environment in which the NSA thrives, the official says. And adds: “Lucky for us.”