Posted
by
timothyon Sunday February 09, 2014 @09:25AM
from the what-would-christopher-boyce-do? dept.

An anonymous reader writes with this excerpt from the New York Times: "Intelligence officials investigating how Edward J. Snowden gained access to a huge trove of the country's most highly classified documents say they have determined that he used inexpensive and widely available software to 'scrape' the National Security Agency's networks, and kept at it even after he was briefly challenged by agency officials. Using 'web crawler' software designed to search, index and back up a website, Mr. Snowden 'scraped data out of our systems' while he went about his day job, according to a senior intelligence official. 'We do not believe this was an individual sitting at a machine and downloading this much material in sequence,' the official said. The process, he added, was 'quite automated.'"

Slightly more powerful than wget to me is a wrapper around wget. Perl and Bash scripts are way beyond the average users. To politicians scripts can be used to claim "voodoo" or "saintly" depending on who writes the scripts. The NSAs scripts are obviously saintly, while anybody else is probably voodoo.

"Slightly more powerful than wget to me is a wrapper around wget. Perl and Bash scripts are way beyond the average users. To politicians scripts can be used to claim "voodoo" or "saintly" depending on who writes the scripts. The NSAs scripts are obviously saintly, while anybody else is probably voodoo."

Even funnier is the assertion that such "web crawling" would be easy to detect. As someone who has done remote automation and data scraping for a living, I can tell you that it doesn't look any different than any other web traffic.

About the only way to detect it is to do traffic analysis, to see if the same IP address is hitting nodes a lot, or hitting many nodes in a short period of time, and especially if they are rapid-fire.

But the latter is easy to get around. I won't say just how here, because even if it's not hard to figure out it's still something of a trade secret.

[Chelsea Manning] had used a program called âoewgetâ to download the batches of files. That program automates the retrieval of large numbers of files, but it is considered less powerful than the tool Mr. Snowden used.

The very first program (after "Hello World") I wrote in Java was a website scraper. I used it to download all of Sun's API and tutorial pages for Java and rewrite links to be relative. Younger and dumber. This created two copies of each set of docs: The scraped version and the compressed version, which I only discovered existed after having scraped the downloadable offline version of said docs.

Point being: My scraper was written in a few hours and far less powerful than wget.

The chance of a random westerner being killed in a terror attack is miniscule. The chances that you or I will meet the same fate as Swartz are even less. You and the Runaway1956 are both engaging in partisan hyperbole similar to that displayed by Schwartz prosecutors, it might feel good to vent at the injustice but the hypocrisy of your methods will almost certainly undermine the validity of your point.

They tried to use an automated tool in developing healthcare.gov but were told it was classified. Someone argued, I think, but top management fired him. Automated tools are no way to get more direct reports, you know. And we need unemployment to go down, not up.

Did you know you can double your "lines of code" output with just a few keystrokes? Write for more info!

Well if you knew a SIEM system had rules which might trigger alters if a database backup is started off hours or if the backup files are accessed for one. As apposed to normalish get query logs with 2XX results, its likely been trained to ignore.

Questioning if who you might eventually leak the data to will have the technical chops and resources put the information together from the database file, as opposed to just reading through a bunch of handy precomputed html pages and office documents for another.

According to the BBC [bbc.co.uk] the new Director of the NSA says:"There's no place where it's an analyst and a database and you can search for whatever you like and there's no record and no after the fact," Mr DeLong says.

So it should be pretty easy for them to figure out which information Snowdon got and when. Unless nowhere means unless outside of Fort Meade...

Now that we have documented proof of a rogue government agency that ignores the law and then lies about it, I'm waiting for some enterprising criminal defense attorney to realize they've got the perfect patsy. Regardless of what crime their client is alleged to commit, just deny involvement in the crime and claim that it was committed by the NSA.

I'm not a lawyer, I just play one on Slashdot. But it seems to me that should be sufficient to raise reasonable doubt.

I think many here are missing the point. Point the Moon with your finger and the fool will look at the finger.

The entire point about the use of automated tools to scrape data here and there on the NSA network is that Snowden wasn't going at the only data he needs to prove the point he says he wants to make. He was just grabbing a full load of data hoping for some of it to prove something that could make him a credible whistleblower. This is playing against him if he would have to convince a judge he is a "l

I like to think of this article as evidence the NSA is manned by ineffectual morons who should be banned from sharp table cuttlery.I, living in this nation, feel less secure because of the existence of this agency, ironic, isnt it?Is this really what, Im paying for? Bullshit! I want ALL my tax dollars BACK! Including Medicare and Social security!I can see that it all an elaborate bait and switch conducted by charlatans like President Clinton and her sidekick Omama.Snowden COULD be the first President electe

You mean to tell me that an NSA tech contractor used wget or something, rather than loading up IE6 and clicking until his fingers fell off?

Knock me over with a feather, spooks. You fucking hired people to build what is probably the largest collection of signals intelligence scraping systems on the planet, targeted at a wide variety of differently structured systems. Why would you even consider, except as a last resort, the notion that you are dealing with a bunch of noobs?

(Oh, incidentally, maybe you should spend a bit less time reading everybody's email and work on that 'hilarious leaked diplomatic calls' problem, I'm told that sort of thing used to be your job at some point in the past...)

Agency officials insist that if Mr. Snowden had been working from N.S.A. headquarters at Fort Meade, Md., which was equipped with monitors designed to detect when a huge volume of data was being accessed and downloaded, he almost certainly would have been caught. But because he worked at an agency outpost that had not yet been upgraded with modern security measures, his copying of what the agency's newly appointed No. 2 officer, Rick Ledgett, recently called "the keys to the kingdom" raised few alarms.
"Some place had to be last" in getting the security upgrade, said one official familiar with Mr. Snowden's activities. But he added that Mr. Snowden's actions had been "challenged a few times".

So they knew he was doing it, even questioned him, and he still got away with the data. To the people who maintain the NSA has the best and brightest security people perhaps they (NSA security) should use that expertise to improve their own security instead of weakening everyone else's.

And yes, this is precisely why they must not be trusted with the data they are gathering due to mass surveillance.

Why would you even consider, except as a last resort, the notion that you are dealing with a bunch of noobs?

Because all the middle management MBAs don't have flying clue how computers work and feel the need to compensate for their perceived inferiority in the face of the technical employees who deliver real value. Demonizing them is the easiest way to accomplish that.

Yep, that's how I'd do it, if I had access to those kinds of networks and an overwhelming curiousity (or suspicion). Not that I'd ever do anything like that of course [glances over shoulder].. or even think of harming one of our most valuable national assets [glances at black helicopter over trees and discounts]...

I suspect the posturing about 'zOMG, Snowden is clearly working for the commie russians and/or chinese taleban!!!!' to be the purest of bullshit; but if I were a member of the US clandestine services, I'd be shitting myself wondering about the existence of people who are working for somebody and running up against the same... impressive... security measures. If there are any actual moles, it is not looking good for what they were likely able to get their hands on.

If the network can't identify that something accessing the network sporadically and in repeated succession is a bot and should be stopped maybe the NSA shouldn't have access to this much data to start with....

What if a legitimate foreign hacker was able to get in and do the exact same thing? Obviously, they have very shitty standards when it comes to network security - you'd expect thousands of honey pots, ability to intercept attempted attacks, flat out network filtering of these kinds of requests. But alas, that would make sense!

If the network can't identify that something accessing the network sporadically and in repeated succession is a bot and should be stopped maybe the NSA shouldn't have access to this much data to start with....

What if a legitimate foreign hacker was able to get in and do the exact same thing? Obviously, they have very shitty standards when it comes to network security - you'd expect thousands of honey pots, ability to intercept attempted attacks, flat out network filtering of these kinds of requests. But alas, that would make sense!

This is the other big glaring issue... Forgetting that the constitution prohibits the sort of forced collection of people's data for a second, which we really really shouldn't forget... with the ability and clear intention to eventually put all the business and communications data in the US into one giant repository the NSA is saying they can protect access to it, but the Snowden leaks are a very big glaring example that some policy isn't in control, people are.

Fortunately people such as yourself don't make these decisions. Out of curiosity, does this sort of logic apply uniformly? I mean if the plane to HK was full of US citizens would their deaths be within "accepted limits"?

Seems many terrorist organizations seem to feel the same way as you, what makes you different from them?

Our rulers do make exactly such decisions, things like invading an entire country on a made up pretext, or using drones to blow up weddings. The kinds of people in control of the Executive branch have staged a soft coup, usurping legislative and judicial powers (e.g., signing statements, due process free execution/detention, executive declared wars, extremely broad executive orders). To think that such evil motherfuckers would not blow up a plane is ridiculous.

You know, whether you agree or disagree with what Snowden did, that in no way justifies killing him without a... oh, what was that quaint thing we used to require? That's right, a trial. Rule of law, and all that. I think that's what the country was based on originally.

Of course, it's embarrassing for the NSA that Snowden waltzed out with so much confidential information, and arguably he should have been ARRESTED within 24 hours of "that flight to Hong Kong", but killed? To even think that sort of thing is disturbing.

Having said that, I am glad he managed to get away, since his revelations are shining an absolutely necessary light on the murky behavior of our government and its actions. An educated populace is necessary to ensuring our freedoms and for too long the government has been hiding its wrong-doings from the ones it purports to serve. Whether Snowden acted as a foreign agent, or for his own advantage, or out of idealism, his actions were necessary and should not be so readily scorned.

(oh right, and fuckbeta and all that jazz. It's gonna be hard doing that boycott tomorrow)

You know, whether you agree or disagree with what Snowden did, that in no way justifies killing him without a... oh, what was that quaint thing we used to require? That's right, a trial. Rule of law, and all that.

If there were to be a trial it is almost certain they would exclude pretty much all avenues of defense [techdirt.com] that support what he actually did and why. Rule of Law is no more in this country. Just ask Aron Schwartz, Bradley Manning and the host of other whistle blowers prosecuted by the self proclaimed most open administration in history. If they want you gone they simple twist the millions of laws that exist and make up new interpretations if that's not enough. But you will be gone.

His job was to find juicy tidbits in data scraped from external sources. His job was definitely NOT to find juicy tidbits in internal NSA documentation. The fact that he could easily and massively access this documentation without anyone seriously questioning his activities is a huge problem.The assumption could be made that internal documentation and externally sourced data are stored on the same servers, and accessed using the exact same methods.

Jerry: David Berkowitz, Ted Bundy, Richard Speck...Alice: What about them?Jerry: Serial killers. Serial killers only have two names. You ever notice that? But lone gunmen assassins, they always have three names. John Wilkes Booth, Lee Harvey Oswald, Mark David Chapman...Alice: John Hinckley. He shot Reagan. He only has two names.Jerry: Yeah, but he only just shot Reagan. Reagan didn't die. If Reagan had died, I'm pretty sure we probably would all know what John Hinckley's middle name was.

The idea of military specialists of whatever type being employed against the society they belong to, is treasonous and fucking retarded no matter what legal acrobatics are employed in their defense.

You may have some sort of mystic devotion to the law, but I believe laws are made by (generally corrupt) men for their own interests, and I am familiar enough with the world outside the borders and political influence of the United States to know there is an enormous difference between legality and rightousness. The U.S.A. may not be the kind of country where you are expected to bribe every public official however minor -- we generally reserve that for higher office. It takes a special kind of idiocy to use military forces against their homeland, though.

Government at its core is the body to which we have delegated our inherent right to violence -- a right being defined in this case as something which cannot be taken from you. We delegate this right to others, specialized in its use, with the express understanding that [a] as applied to civilian life, the exercise of violence by police will be applied fairly and equally as men can manage, and [b] that the unrestricted expression of this (as embodied by military force) be only employed against our enemies. War is hell, and we do not bring hell home.

Snowden is a patriot, and the NSA is treasonous -- whether or not the law can be made to serve whichever purpose. Beyond all other argument, potentially felonious violation of the law is so common with the continual proliferation of laws that lawfulness cannot be the only measure of either justice or rightousness. May all those who support the NSA have a fair trial.

Oddly, government is complaining that people will be able to take the various facts that he assembled and figure out what we're really up to. You know, the kind of thing they say they can't do with our metadata.

Now the question is, how many other NSA contractors / staff / moles / spies have been doing the same thing, without Snowden's intention to disclose their behavior?

I'm sure the NSA assumes they have moles, and none of the data Snowden released is a surprise to the Russians or Chinese.The NSA was just not prepared for the truth to leak to their real enemy - the general public.

Why? I'm quite sure that most governments at the very least had a general idea of what the NSA was up to before Snowden's leak, it's only members of the general public that would get painted as conspiracy theorists and ignored whenever the they tried to draw attention to this sort of thing.

How so? The real enemy of any bureaucracy is those who would dismantle, or at least limit it, in terms of power and growth. No foreign power is going to do that to the NSA. If (and I'm afraid it's a big if) anything limits the power and growth of the NSA it will be the outrage of the American people. Therefore the American people are the true enemy of the bureaucracy.

It's a logical conclusion based on the available evidence: No safeguards were in place to defend against an analyst stealing data and giving it to someone else, despite this being an obvious threat the NSA could not possibly have been unaware of.

No such measures were taking until someone (i.e. Snowden) leaked this information to the public. Add this to the extremely negative way in which the NSA and the entire administration talks about journalists reporting on this, and the

"Agency officials insist that if Mr. Snowden had been working from N.S.A. headquarters at Fort Meade.. he almost certainly would have been caught. But because he worked at an agency outpost that had not yet been upgraded with modern security measures, his copying.. raised few alarms."

This is retrospective ass-covering cyberbullshit. It is precisely at the edge that the security attacks would come from. What they were doing putting such material on Web servers and Wikis beggers credulity. Didn't senior management not realize that as keepers of the nations secrets they would be subject to attacks both internally and externally. Given the state of non-security at the NSA I would suspect that Snowden wasn't the only hostile with access to the “the keys to the kingdom”.

These people are part of the same defense establishment that thought it was a good idea to make CD writers available on "secured" networks where people like Manning could access them. The people working for defense contractors have fairly well secured and locked down systems because there are serious financial penalties for unintentional disclosures. Within the DoD institutions themselves, however, it's an anything goes wild-west in the name of expediency. In that sphere, it's all a good 'ol boys club where

There's absolutely zero reason to believe anything the NSA says about how Snowden got the documents, or indeed, about anything. They believe they are entitled to lie to congress, so the public isn't even a question.

Shouldn't the shock and horror be that Snowden was able to scrape the juiciest pages from the NSA information dump on basically everyone, without so much as a 403 error?

It was the intranet - I guess they trusted everybody with an AD account;-)

I believe, though, it's no coincidence that Snowden ended up in the HW office. He was probably aware of the lack of security when he requested the transfer.
God only knows how many guys have downloaded that data before him and sold it to the highest bidder.