Search

Subscribe

NSA Hiring Data Miners

The Algorithm Developer will work with massive amounts of inter-related data and develop and implement algorithms to search, sort and find patterns and hidden relationships in the data. The preferred candidate would be required to be able to work closely with Analysts to develop Rapid Operational Prototypes. The candidate would have the availability of existing algorithms as a model to begin.

In huge datasets you will probably find just about any relationship you care to look for. I wonder if a good understanding of the scientific method is a job requirement; or possibly just a working knowledge of the Dan Brown books :-)

It was not until I'd had a graduate course in datamining that I truly appreciated Bruce's criticisms thereof. Datamining can only yield useful statistics, and can only benefit on a statistical level. It might, for example, help a retailer decide to cluster various items in a way that boosts revenue in candybar sales by 5%. It will not allow the retailer to prevent Alice from purchasing the cheap dog food and nothing more; it will not ensure Bob will grab a 6-pack of beer with his frozen pizza. It might increase early detection of various cancers via genetic screening, reducing the overall cancer fatality rate by 10%; it will not say, at birth, which of Carl, Dave, or Earl will develop cancer (although based on a vast collection of data including true positives and true negatives; a cost of screening, a cost of false positivies; a cost of false negatives; datamining might say that after the age of 30, Carl should be screened every 3 years; Dave every 10 years; and Early every 20 years).

The NSA does not have nearly the data needed to even make that kind of prediction. Plus, I'm guessing the NSA isn't looking to datamining for statistical reductions in terror-related fatalities; they want it to sift through data of 300 million Americans and say, "These 6 guys are plotting to blow up the Sears tower!". I'm also guessing they will pin all of their inevitable failures on, you guessed it, insufficient data due to cumbersome judicial oversight processes.

An issue I'd like to see addressed in public is the likelihood that our ever-expanding surveillance society coupled with effective data-mining will undermine the international business and investing foundations.

If our more clandestine govt. entities have resorted to triangulating with a sworn enemy to fund revolutionary ideals (Iran-Contra) what is the likelihood that these same organizations, as well as organizations with government sanctioned eavesdropping and data-mining abilities (AT&T, AOL, etc.), will not tap into this as a source of funds? Imagine the ease with which one with this 'ultimate-insiders' access can generate funds!

The net-result will be lowered return on capital to the rest of the investing community and an incredible temptation to those well-connected.

I smell a scandal brewing - but still, it may provide an incentive that surpasses even Google stock-options!

Today, rapid deployment can stop a massive problem, if only
given the program.
I'd been disappointed if they didn't have this
down to a science many years ago.
What ever happened to the idea, be 15 years ahead of industry?