from the perfect-audits? dept

Following on our earlier story about how Ed Snowden covered his tracks -- showing that the NSA's vaunted "auditability" of its systems is a complete joke -- comes the news that there are approximately one thousand sys admins with Snowden's authority, who can basically go through any document without any trace. Even more incredible: they can "appear as" anyone else when doing things on the system. In other words if a sys admin wanted to frame an NSA analyst, it sounds like that would be quite easy. The report also notes that, for all of the talk about how great the NSA is at cybersecurity, and the fact that part of the point of CISPA was to try to have the NSA in charge of the nation's cybersecurity, the agency does a piss poor job protecting itself:

“It’s 2013 and the NSA is stuck in 2003 technology,” said an intelligence official.

Jason Healey, a former cyber-security official in the Bush Administration, said the Defense Department and the NSA have “frittered away years” trying to catch up to the security technology and practices used in private industry. “The DoD and especially NSA are known for awesome cyber security, but this seems somewhat misplaced,” said Healey, now a cyber expert at the Atlantic Council. “They are great at some sophisticated tasks but oddly bad at many of the simplest.”

That last sentence really means: "they are great at hacking stuff, but crap at protecting stuff."

As for the thousand or so sys admins on staff, it appears that they have no restrictions or tracking of what they do:

As a system administrator, Snowden was allowed to look at any file he wanted, and his actions were largely unaudited. “At certain levels, you are the audit,” said an intelligence official.

He was also able to access NSAnet, the agency’s intranet, without leaving any signature, said a person briefed on the postmortem of Snowden’s theft. He was essentially a “ghost user,” said the source, making it difficult to trace when he signed on or what files he accessed.

If he wanted, he would even have been able to pose as any other user with access to NSAnet, said the source.

Remember how the NSA at one point said that there were only 35 analysts who could run certain queries? And that all of the queries were tracked and audited. It seems they left out the thousand or so sys admins who could do whatever they wanted with no tracking at all. Does anyone honestly think that none of those sys admins ever was involved in a "LOVINT" situation? Or something much worse?

So we're left with an agency that collects a ridiculous amount of info, and has around 1,000 employees (who are mostly actually employed by outside contractors) who can look through anything with no tracking, leaving no trace, and we're told that the data isn't abused. Really? Do Keith Alexander, James Clapper, President Obama, Dianne Feinstein and Mike Rogers really believe that none of those 1,000 sys admins have ever abused the system? And, do they believe that none of the people whom those thousand sys admins are friends with haven't had their friend "check out" information on someone else? Hell, imagine you were someone at the NSA who understood all of this already. If you wanted to abuse the system, why not befriend a sys admin and let him or her do the dirty work for you -- knowing that there would be no further trace?

Basically, it seems clear that the NSA has simply no idea how many abuses there were, and there are a very large number of people who had astounding levels of access and absolutely no controls or way to trace what they were doing.

Reader Comments

They gave 1000 people system administration privileges? Wow as a system administrator you can pretty much do as you please then remove any trace you were ever there. Also yes you can't automate system admins. Its way better to have trustworthy people then a computer which can be hacked and then there's completely no way to know.

How many of those 1000 sys admins are paid to spy for other governments?

Here's a bigger question, with 1,000 people who can do whatever they want and get their hands on almost any data they want it sounds like, how many of them have been bribed to work for foreign governments?

If they can access this information so easily like Snowden could, and they can cover up their tracks so easily like Snowden did, that makes ALL of them a very tempting target for a foreign government to bribe.

Re: Disgruntled admins..

I wouldn't say I would worry about disgruntled admins as much as the fact that they still have access after they've been fired. What's more is that they've got only 100 people to remove the security credentials on what is likely thousands of computers.

Re: Re: Disgruntled admins..

I'd hope an organisation the size of the NSA has some kind of centralised security management and other restrictions. If they're manually setting the access credentials for each individual machine, they're asking for trouble and you might as well assume that the data's compromised anyway.

The more realistic question is how many of those admins left themselves backdoors or other ways of accessing data that nobody else knows about. I'll bet there's several, and now they lack the manpower to audit machines for anything not detectable by standard intrusion detection and auditing procedures.

How many of those 1000 sysadmins are working for the NSA?

And how many of them are working for the Russians, the Chinese, the Japanese, the British, the Germans...or worse, the Mafia, the Zetas, etc.?

Surely nobody of the slightest intelligence is going to suggest that 1000 out of 1000 are absolutely loyal. The odds against that are staggering. Not when Snowden has provided a demonstration proof that -- with just a little care -- it's possible to stroll out of the NSA with a staggering amount of information. Surely someone who only needs to take a little information...but just the right information...and sell it to some very interested buyers who are willing to pay top dollar/yen/euro/rupee for it, will have no trouble doing so.

Another way of looking at it: the NSA is very busy building an information repository for lots of other people besides the United States government.

Have those 1000 sysadmins actually been terminated?

If so, here are a few questions:

1 - Did they leave any back doors into the systems?
2 - Did they create some other accounts for later use?
3 - Did they already dump all the files they could find into a safe place?
4 - Who are they blackmailing already?

It sounds like a lot of these servers are UNIX or Linux based and the folks that administer those systems tend to be very creative.

Re: Have those 1000 sysadmins actually been terminated?

A sysadmin or a group of sysadmins who know they are facing the axe but then don't threaten to wreck the IT systems if they're fired don't deserve the title. No matter what your organization, NEVER piss off your sysadmins. Since it's been proven here that any of them can leave no trace, then there really is nothing stopping them from going "Fuck this, there goes my salary, might as well get myself set for life and sell a few secrets to the Chinese or Russians or whoever".

Re: Re: Have those 1000 sysadmins actually been terminated?

Real Sysadmins don't spend months or years creating a beautiful, functional system just to tear it down at the threat of termination. It would be like cutting the left pinky-finger off of your child. It's sick.

If this wasn't the NSA, this would be hysterical.
Sadly, this is the NSA and I don't know about anyone else but I feel fuckloads less safe seeing how inept and incompetent they are.
I'm terrified that Congresscritters and others actually think these people are the best of the best.

This is what happened with every "good" idea we put into motion, they attach wads of cash to their corporate sponsors and it goes to shit and needs more and more money.

Another 900 Snowdens ready to go wild

They don't know what Snowden took and now they're firing another 900 just like him. What are they leaving with?

OK, I'm not paranoid but.... I think everyone needs to make themselves as small a target as possible. Start encrypting phone calls, emails, text messages, browsing. Stop storing files on Dropbox, in Gmail, in iCloud, etc., and stash everything in a Cloudlocker (www.cloudlocker.it) which stays in your house where they still need a warrant to look inside.

What a shame it's come to this, but we have to protect ourselves from the people who are supposed to protect us.

Basically, it seems clear that the NSA has simply no idea how many abuses there were, and there are a very large number of people who had astounding levels of access and absolutely no controls or way to trace what they were doing.

And now 900 of them are going to get laid off.
I've heard of big businesses being brought to their knees at the hands of one disgruntled sysadmin. One.
They fired. Nine. Hundred. Sysadmins.
Sysadmins who were in charge of an enormous database of potentially dangerous information, all locked behind paper-thin security.

This cannot possibly end well for the NSA.
They're not going to be able to drum up much sympathy when the other shoe drops, either. With the public in a state of sustained outrage, and congressmen clamoring for them to be defunded, they may well be shut down altogether in the wake of whatever disaster befalls them.

Guesses about NSA tech

As tech-heavy as they are, the NSA is a very large and (in tech terms) relatively old organization, and a government one at that. From those characteristics, we can extrapolate from similar (but not secret) organizations to make certain guesses. For example, they've probably had an attempted upgrade turn into a major screwup with huge cost overruns and lots of management ass-covering. It happens in the military, it happens in government, it happens in private industry.

We can also guess that they have lots of legacy tech to maintain. Specialized old hardware and software that has to be kept in place for specific intelligence-collecting missions, but that doesn't integrate well with more up-to-date systems. Keeping these systems working would require specialized employees with significant system privileges.

We can also guess that their bureaucratic structure and security requirements will sometimes delay new technology. If, for example, they were using a custom in-house linux fork, then any improvements and bugfixes have to jump through the obvious hoops.

That's all straightforward, and can be streamlined. But it also means that those old systems -- the ones that mostly work, don't seem like a big risk, and would be a huge effort to replace, may be continued nearly indefinitely. It also suggests to me that remodeling old tech might sometimes be bureaucratically preferred over new construction. How many lines of Fortran 77 do you reckon the NSA still has deployed? Of Cobol? Ada?

My guess is the NSA has some really nasty system integration/ESB/API-genre problems. In addressing these, upper management lacks the technical expertise, but demands solutions, so middle management looks the other way when nerds cut corners on security stupidity (cf. sysadmin privs) to get results.

Fired 900

I bet the statement about getting rid of 900 sysadmins is like the other "least untrue" hairsplitting -- it means they'll have different job titles, but still be getting a paycheck. Maybe even a promotion!

I'll also bet that everything that can easily be automated has already been automated. Remember, upper management is always completely fucking clueless.

Re:

This long? What took yous?

That was one of the first things I realized. You have data piling up, both on the organization, and on its victims. You have people of all different shades of political belief and technological ability. And you have - in every organization - someone disgruntled with something or other.

Plus you're working in the Security/Intelligence field, where terminating employment sometimes means terminating the employee as well, even in the spy novels (One of the books I read during my misspent youth was about working in Security/Intelligence, and one thing the author pointed out was that anyone with scruples about terminating another spy to save his own life, was worth nothing as a spy.)

So the chance that someone else will have insurance lying around somewhere, ready to snap the NSA like dry-rotted wood if anything should turn against him? Practically 1.00 confidence interval.

The chance that someone will then use that information to spruce up his life? .95 confidence interval.

The likeliehood that nothing Snowden has leaked so far is news to anyone else? Practically 1.00 confidence interval.

So, what the h*** took yous? You've had all the pointers ... the persecution of Snowden's got nothing to do with security vulnerability as such - it's got everything to do with security theatre.

Thank you captain obvious...

Yes, the intelligence agencies are very good at hacking - it's their equivalent of the money-making part of their enterprises. As a result, data-collection is where they spend the bulk of their money and place the most brain-power/skills. However, like the vast majority of enterprises, IT administration is simply another cost-center. As such, those costs are controlled to as great a degree possible - moreso given current budgetary trends. This means that administration is happening at "lowest bid" rates.

The plus side of this is that, while there may be 1000+ individuals that have sufficient access to do something Snowden-esque, there's a very small fraction that have the skills to even begin to try to take advantage of that access. Smaller still is the number that, even if they had the skill to take advantage of that access would not voluntarily nor could be induced to do so.

You can see this reflected in any IT services organization: the billable guys tend to be the most clueful folks while the back-office guys tend to be lower-tier skills. Think of the guy doing desktop support at a Fortune 500 and that's what the vast majority of the 1000 SAs are like in government IT.