All it took for Edward Snowden to grab roughly 1.7 million classified documents from the National Security Agency’s network was an open-source Web crawler and a few scripts, according to a New York Timesreport on Sunday. An investigation of Snowden’s activities at the NSA outposts in Hawaii apparently found that he was able to retrieve millions of classified documents in an automated fashion using what the Times described as “low-cost” software. That software was likely based on the open source GNU Wget utility.

Intelligence officials would not say what the tool was, but said they believed it was "more powerful" than Wget. The anonymous sources don’t add much to the narrative of Snowden’s extraction of secret documents, though they do start to put a number on the volume of what officials believe he made off with. But the real sting of the latest data is that the NSA’s internal IT operations are portrayed as even more fast and loose than before. Anyone with admin access might have been able to do what Snowden did.

Walking through the spider webs

Wget is the tool that was used by Chelsea Manning (formerly Bradley Manning) hundreds of times to retrieve classified files off Department of Defense networks that she later provided to WikiLeaks. It can be used to download a series of interlinked files from websites—downloading a webpage and then every document linked from that page, as well as every document linked from subsequent pages. Wget can be and is often used to set up “mirror” websites. And it's free and open source—only the overhead associated with a support contract would push it into the realm of “low-cost.”

Further Reading

As Ars has previously reported, the NSA has little if any internal compartmentalization of its classified documents since the organization shifted to a culture of sharing in the wake of the September 11, 2001 attacks on the World Trade Center and Pentagon. That minimal security gave Snowden access based on his clearance to a significant amount of the NSA’s internally shared documents. And despite efforts following the WikiLeaks scandal to better monitor the activities of cleared users within the NSA’s networks, the NSA’s moves to increase security never reached the facilities in Oahu where Snowden worked. Bandwidth limitations on the NSA’s internal network made the deployment of the software there impractical.

It’s not clear whether the sources who spoke to the Times were simply trying to discredit the mythos of Snowden as some “uber-hacker” who used elite skills to defeat the NSA’s internal security or if the backchannel was for internal political reasons. But the report reveals nothing really new about Snowden’s collection efforts. It simply highlights how extraordinarily bad the NSA’s internal security regime was.

Root has its privileges

Anyone with a Top Secret/Sensitive Compartmented Information (TS/SCI) clearance at the NSA would have had access to nearly everything Snowden touched. But because of the extraordinary level of leeway given to system administrators by the NSA, Snowden was able to scoop most of it up, put it on USB drives, and carry it off without generating much suspicion at all. Snowden was allegedly able to explain away his large-scale, scripted accessing of the data on the NSA’s WebWorld intranet as part of his job as a system administrator. And his activities were allowed to continue with little to no oversight.

According to one official who spoke to the Times anonymously, Snowden’s activities were “challenged a few times.” But on each occasion, Snowden offered what seemed to be legitimate excuses for the access—he was, after all, in charge of moving content into a newer, more secure system, according to NSA Director of Technology Lonny Anderson.

Wget would be exactly the type of tool Snowden would have used to do that sort of task—Anderson said he was part of the team moving content to a new “tagged” system that would allow for greater auditing and control over who accessed documents in the WebWorld intranet.

Using some fairly simple scripts, Snowden would have been able to execute crawls of targeted parts of the NSA’s intranet without the need to hang around and hunt for data manually. He could go about his daily business as an NSA system administrator while a computer downloaded the contents of the NSA’s network of Wikis and other Web-based document stores.

Insider blind spot

Intelligence officials have claimed that Snowden was able to do all this largely because the Oahu NSA facilities had not gotten the software purchased to prevent insider threats in the wake of WikiLeaks. “He was either very lucky or very strategic” to get the positions he held in Hawaii, one official told the Times. But it’s also entirely possible that his activities would have gone unchecked in any case, simply because of his system administrator status.

The software deployed to prevent insider threats focused largely on client machines to watch for “exfiltration”—the removal of data from the network. But Snowden would have been able to download the data directly from servers and find other ways to dodge auditing, all while chalking up the activity to sysadmin duties.

Share this story

Sean Gallagher
Sean is Ars Technica's IT and National Security Editor. A former Navy officer, systems administrator, and network systems integrator with 20 years of IT journalism experience, he lives and works in Baltimore, Maryland. Emailsean.gallagher@arstechnica.com//Twitter@thepacketrat