Intelligence Agencies Still Drowning in Data

More than a decade after the 9/11 terrorist attacks, US intelligence agencies are doing a much better job sharing information about terrorism and other national security threats. Their challenge now “is largely one of information overload,” says a new report by the Congressional Research Service, published last week.

“Analysts now face the task of connecting disparate, minute data points buried within large volumes of intelligence traffic shared between different intelligence agencies. According to a [Director of National Intelligence] statement from July 2010, ‘Terabytes of foreign intelligence information come in each day, vastly exceeding the entire text holdings of the Library of Congress, which is estimated at 10 terabytes.’ In the additional views section of the Senate report on the Christmas day bombing attempt, Senators Saxby Chambliss and Richard Burr noted that analysts who could have connected the dots prior to the incident struggled to search the large volume of terrorism-related intelligence available to them. The same problem was identified at the FBI in the aftermath of the 2009 Fort Hood shooting.”

The crippling dilemma of information overload is not news. (See here, here, here.) But in the context of the Boston Marathon bombings, and the inevitable questions that will follow about who knew what when, it’s important to keep this salient fact in mind: The US intelligence community does not have a problem collecting information. It has a problem understanding much of it.

(Thanks, as always, to Steve Aftergood at the Federation of American Scientists for posting this CRS report, among many others.)