America's top spies go up against a crowd

WASHINGTON — Nine years ago, Congress blocked a Pentagon agency from setting up a website that would have allowed anyone with a credit card to bet on the likelihood of foreign assassinations, coups and terrorist attacks.

The idea was to take advantage of the "wisdom of crowds," a social science maxim that contends the average of a group of forecasters, under certain circumstances, tends to be more accurate than even the most knowledgeable single forecaster.

But lawmakers worried the proposed predictions market could allow terrorists to profit from their own misdeeds. Congress forced the Defense Advanced Research Projects Agency -- the military's cutting-edge research arm, known as DARPA -- to scuttle the program.

Now terrorism futures are back.

DARPA's sister agency -- the Intelligence Advanced Research Projects Activity, which funds experimental projects for the U.S. intelligence community -- is running a four-year, $50-million program that pays people willing to predict major world events, including wars and terrorist strikes. Unlike the earlier scheme, participants can't profit from their predictions.

For The Record Los Angeles Times Thursday, August 23, 2012 Home Edition Main News Part A Page 4 News Desk 2 inches; 60 words Type of Material: Correction CIA crowd-sourcing: An article in the Aug. 21 Section A about a federal study into the "wisdom of crowds" said that participants were asked to assess the likelihood of future terrorist strikes, among other topics. Questions about terrorist attacks are prohibited in the program, said Cherreka Montgomery, spokeswoman for the Intelligence Advanced Research Projects Activity, which is conducting the study.

Now in its second year, the so-called crowd-sourcing project involves competing corporate and university teams, including UC Irvine. Each team includes more than a dozen social scientists and as many as 2,000 participants, who can answer hundreds of questions each if they want.

The study, known as Aggregative Contingent Estimation, is designed to see whether the 17 agencies in the U.S. intelligence community can aggregate the judgment of its thousands of analysts -- rather than rely on the expertise of just a few -- to issue more accurate warnings to policy makers before and during major global events.

Participants in the project give their best guesses on whether the Free Syrian Army will gain control of Aleppo, for example, or whether Kim Jong Un will resign as the leader of North Korea before April 1, 2013, according to project websites.

Initial results are promising, according to Philip Tetlock, a University of Pennsylvania professor who leads one of the teams.

His 2005 book, "Expert Political Judgment: How Good Is It? How Can We Know?" showed that political pundits often fared worse in predictions than the proverbial dart-throwing monkey.

"The idea that the average of a group of forecasters is more accurate than any one forecaster goes back 100 years," Tetlock said. In 1906, British scientist Francis Galton determined that the average of 787 guesses about the weight of an ox at a county fair was startlingly accurate.

The teams are trying to beat the average by using algorithms that, over time, give greater weight to the most successful crystal gazers among their participants and weed out the worst.

Tetlock's group, the Good Judgment Project, is in the lead so far.

"In year one, we beat the unweighted average by about 57%, which was big," he said. A control group, run by Mitre Corp., averages scores without giving weight to participants who tally the best results.

Predicting world-changing events is never easy. U.S. intelligence agencies failed to foresee the fall of the Soviet Union, to predict the terrorist attacks of Sept. 11, 2001, and, more recently, to anticipate political upheaval across the Arab world.

The CIA and other agencies tend to rely on small groups of experts who hash out consensus judgments. If there is strong disagreement among the analysts, or between different agencies, an intelligence report will note that.

Crowd-sourcing would mean, in theory, polling large groups across the 200,000-person intelligence community, or outside experts with security clearances, to aggregate their views about the strength of the Taliban, say, or the likelihood that Iran is secretly building a nuclear weapon.

For now, there is deep skepticism in intelligence circles.

"I don't believe in the wisdom of crowds," said Mark Lowenthal, a former senior CIA and State Department analyst (and 1988 "Jeopardy!" champion) who now teaches classified courses about intelligence. "Crowds produce riots. Experts produce wisdom."

Others argue that intelligence analysis can benefit from data-driven social science.

The CIA's venture capital arm, In-Q-Tel, was an early investor in a private company called Recorded Future, headquartered in Cambridge, Mass., and Goteborg, Sweden, which uses data on the Internet, from securities filings to Facebook posts to government statistics, to make connections and spot trends.

A National Academy of Sciences report last year commissioned by former National Intelligence Director Dennis C. Blair found that intelligence practices "have been only weakly informed by the behavioral and social sciences." The report recommended that "science and evidence-based protocols" be introduced and evaluated alongside the spy agencies' traditional reliance on expert judgment.