January 10, 2014

George Mason University launched today, Jan. 10, the largest and most advanced science and technology prediction market in the world: SciCast.

The federally funded research project aims to improve the accuracy of science and technology forecasts. George Mason research assistant professor Charles Twardy is the principal investigator of the project.

SciCast crowdsources forecasts on science and technology events and innovations from aerospace to zoology.

For example, will Amazon use drones for commercial package delivery by the end of 2017? Today, SciCast estimates the chance at slightly more than 50 percent. If you think that is too low, you can estimate a higher chance. SciCast will use your estimate to adjust the combined forecast.

Forecasters can update their forecasts at any time; in the above example, perhaps after the Federal Aviation Administration (FAA) releases its new guidelines for drones. The continually updated and reshaped information helps both the public and private sectors better monitor developments in a variety of industries. SciCast is a real-time indicator of what participants think is going to happen in the future.

“Combinatorial” prediction market better than simple average

How SciCast works (Credit: George Mason University)

The idea is that collective wisdom from diverse, informed opinions can provide more accurate predictions than individual forecasters, a notion borne out by other crowdsourcing projects. Simply taking an average is almost always better than going with the “best” expert. But in a two-year test on geopolitical questions, the SciCast method did 40 percent better than the simple average.

SciCast uses the first general “combinatorial” prediction market. In a prediction market, forecasters spend points to adjust the group forecast. Significant changes “cost” more — but “pay” more if they turn out to be right. So better forecasters gain more points and therefore more influence, improving the accuracy of the system.

In a combinatorial market like SciCast, forecasts can influence each other. For example, forecasters might have linked cherry production to honeybee populations. Then, if forecasters increase the estimated percentage of honeybee colonies lost this winter, SciCast automatically reduces the estimated 2014 cherry production. This connectivity among questions makes SciCast more sophisticated than other prediction markets.

“With so many science and technology questions, there are many niches,” says Twardy, a researcher in the Center of Excellence in Command, Control, Communications, Computing and Intelligence (C4I), based in Mason’s Volgenau School of Engineering.

Forecasters discuss the questions, and that discussion can lead to new, related questions. For example, someone asked,Will Amazon deliver its first package using an unmanned aerial vehicle by Dec. 31, 2017?

An early forecaster suggested that this technology is likely to first be used in a mid-sized town with fewer obstructions or local regulatory issues. Another replied that Amazon is more likely to use robots to deliver packages within a short radius of a conventional delivery vehicle. A third offered information about an FAA report related to the subject.

Any forecaster could then write a question about upcoming FAA rulings, and link that question to the Amazon drones question. Forecasters could then adjust the strength of the link.

“George Mason University has succeeded in launching the world’s largest forecasting tournament for science and technology,” says Jason Matheny, program manager of Forecasting Science and Technology at the Intelligence Advanced Research Projects Activity, based in Washington, D.C. “SciCast can help the public and private sectors to better understand a range of scientific and technological trends.”

Collaborative but Competitive

More than 1,000 experts and enthusiasts from science and tech-related associations, universities and interest groups preregistered to participate in SciCast. The group is collaborative in spirit but also competitive. Participants are rewarded for accurate predictions by moving up on the site leaderboard, receiving more points to spend influencing subsequent prognostications. Participants can (and should) continually update their predictions as new information is presented.

SciCast has partnered with the American Association for the Advancement of Science, the Institute of Electrical and Electronics Engineers, and multiple other science and technology professional societies.

Mason members of the SciCast project team include Twardy; Kathryn Laskey, associate director for the C4I and a professor in the Department of Systems Engineering and Operations Research; associate professor of economics Robin Hanson; C4I research professor Tod Levitt; and C4I research assistant professors Anamaria Berea, Kenneth Olson and Wei Sun.

Thanks for the clarification. I will give it another try next week. You may want to add a comment about the blueberries so other people are not similarly put off. Or, I suppose, if it is part of a test, you may not.

Seems there may also be a self fulfilling prophesy effect inherent in crowdsourcing. If people rely on these predictions to make R&D budget allocations and other business decisions then these actions contribute to making it happen. I think this is a good thing overall as good ideas will be more likely to be successful, but it’s hard to tease out cause and effect in a feedback scenario.

This may be a great idea but it needs some work in the implementation. The main problem is that the crowd gets to suggest questions. Many of the questions are exceedingly lame — a whole series on the price of Blueberries (apples, apricots, etc) this year. Nearly all of the questions deal only with the immediate future, i.e., 2014, because of the way it’s structured to reward people who give good predictions (who want to wait). So there are questions like “Which facility will announce productive fusion this year?” instead of “Which process will eventually prove productive for fusion.” What we really want are a select number of long-term predictions which are relevant to our future.

Please do help write questions! There is a tension between the really interesting long-term questions, and meeting performance metrics for a yearly funding cycle. We’re aiming for half the questions to resolve within 2 years (with 100+ resolving by May) and half resolving in 2-50 years.

(Er, about the blueberries… We needed a large number of verifiable, related, short-term questions for a study. You might prefer Mathematics and Space Sciences offerings. Also, look for dozens of new non-fruit questions next week. )