Forecasting in everyday life serves a messy mix of psychological and political functions, with accuracy only one of many goals. Forecasting tournaments are normative mechanisms for simplifying life. All that matters is accuracy at assigning accurate probability estimates to real-world events that often provoke strong political passions. This presentation will describe what has been learned from 33 years of research on forecasting tournaments that test skill at assigning probability estimates to a wide range of political-economic events, from Mikhail Gorbachev to Kim-Jong-un. These tournaments allow us to discover “who is more accurate about what?” and to explore the underlying cognitive and social drivers of accuracy. I conclude by exploring the latest directions that forecasting tournament research has taken, with special focus on the challenge of judging not only the accuracy of answers but also the quality of questions. I use as a test case the recent wave of speculation about a looming “Fourth Industrial Revolution that will be driven by strong forms of Artificial Intelligence and that will cause major dislocations in white-collar labor markets.”

Abstract 2:Judging Judgment: Philosophical and Scientific Challenges

What exactly do people mean when they claim that someone has “good judgment”? I explore four major perspectives—logical-coherence, empirical-correspondence, ethical-moral and pragmatic—and discuss how far scientific methods have taken us in assessing each conception of good judgment. The most significant advances have been in developing logical-coherence and empirical-correspondence benchmarks for judging judgment. There are however serious in-principle limits on how far purely value-neutral scientific methods of inquiry can advance our understanding of the ethical-moral and pragmatic conceptions of good judgment. These limits do not however imply no progress can be made. Ethical-moral dimensions can be partly captured in signal-detection models that specify how observers from clashing schools of thought set their tolerance thresholds of proof for making false-positive and false-negative classification judgments. And pragmatic conceptions can be partly captured by simulation models that capture how observers make judgments about counterfactual history, about what would have happened if liked or disliked leaders had made different choices at key event junctures. But “good judgment” can never be fully objectified.