NSA officials declined to say which web crawler Snowden had used, or whether he had written some of the software himself.The software is widely available and designed to search, index and back up a website, but was used by Snowden to harvest classified data.

The news has deepened concerns that Snowden was not discovered and stopped in the light of the fact that the NSA is responsible for defending sensitive US computer systems from cyber attack.

Snowden’s insider attack, by contrast, was hardly sophisticated and should have been easily detected, investigators said.

Weaknesses exploited

Snowden had broad access to the NSA’s complete files because he was working as a technology contractor for the agency in Hawaii, helping to manage the agency’s computer systems.Almost three years earlier, a similar technique was used to harvest data from the US State Department, that was passed on to WikiLeaks.

An NSA official told the NYT that Snowden had been “challenged a few times” but he had been able to persuade investigators his actions were in line with his work as a system administrator.

Investigators say Snowden exploited the fact that, while the NSA had built enormously high electronic barriers to keep out foreign invaders, it had rudimentary protections against insiders.

He also exploited the fact that he was working at an NSA outpost in Hawaii that had yet to be equipped with modern monitors, which might have sounded the alarm.