Tuesday, June 26, 2012

The Fundamental Uncertainty of Science

While I have not had much time for the mini data-gathering/research projects that I usually try to post on this blog, I found the recent flurry over Professor Jacqueline Stevens' New York Times editorial "Political Scientists are Lousy Forecasters" (and the follow-up on her blog) worth commenting on a bit more.

The political science blogosphere has since respondedinfull-force (and snark). I agree entirely with the already stated criticisms and will try not to repeat them too much here. The editorial is at best a highly flawed and under-researched critique of quantitative political science and at worst a rather cynical endorsement of de-funding all NSF political science programs on the grounds that the NSF tends to fund studies using a methodological paradigm that Professor Stevens does not favor. I'll err on the side of the former.

But one quote from the piece did irk me quite a bit:

...the government — disproportionately — supports research that is amenable to statistical analyses and models even though everyone knows the clean equations mask messy realities that contrived data sets and assumptions don’t, and can’t, capture. (emphasis mine)

This statement is on-face contradictory. The entire point of statistical analysis is that we are uncertain about the world. That's why statisticians use confidence levels and significance tests. The existence of randomness does not make all attempts at analyzing data meaningless, it just means that there is always some inconclusiveness to the findings that scientists make. We speak of degrees of certainty. Those who use statistical methods to analyze data are pretty clear that none of their conclusions are capital-T truths and the best political science tends to refrain from any absolute statements. Indeed, this is a reason for why a gap tends to exist between the political science and the policymaking communities. Those who enact policy want exact and determinate guidance while political scientists are cautious about making such absolute and declarative statements. It is depressing to see these sorts of caricatures of quantitative methods being used to denounce the entire field. Simply put, just because physics is very quantitative and physics appears to describe very clean and determinate relationships does not mean that all uses of math in social science result in only simple, exact and absolutely certain conclusions.

But putting aside that highly inaccurate picture, Professor Stevens' definition of what constitutes scientific knowledge is remarkably limiting. Prof. Stevens is staking out a very extreme position by implying that the existence of randomness - "messy realities" as she calls it - makes all attempts at quantification meaningless. She argues for a very radical version of Popper's philosophy of science, positing that any theory should be considered falsified if it is contradicted by a single counter-example. It's unfortunate that Prof. Stevens glosses over the extensive philosophical debate that has followed in the eight-or-so decades after Popper, but this is inevitable given the space of a typical NYT column. Nevertheless, it is very disappointing that the OpEd gives the impression that Popperian falibilism is the gold standard of scientific method and the philosophy of science, when in fact, the scientific community has moved far beyond such a strict standard for what constitutes knowledge. While I won't go into a full dissection of Popper, Kuhn, Lakatos, Bayesian probability theory, and so on, it suffices to say that Stevens' reading of Popper would discount not only political science, but most modern sciences. Accounting for and dealing with randomness is at the heart of what so many scientists in all disciplines do.

By rejecting the idea that probabilistic hypotheses could be considered "scientific," Professor Stevens is perpetuating another caricature - one of science as a bastion of certitude. It's a depiction that resonates well with the popular image of science, but it is far from the truth. I'm reminded of a quote by Irish comedian Dara O Briain:

"Science knows it doesn't know everything, otherwise, it would stop."

All science is fundamentally about uncertainty and ignorance. Knowledge is always partial and incomplete. There was actually an interesting interview with neuroscientist Stuart Firestein on NPR's Science Friday on this topic a few weeks back, where he offered this valuable quote:

...the answers that count - not that answers and facts aren't important in science, of course - but the ones that we want, the ones that we care about the most, are the ones that create newer and better questions because it's really the questions that it's about.

Ultimately, I would argue that probabilistic hypotheses in the social sciences still have scientific value. Events tend to have multiple causes and endogeneity is an ever-present problem. This does not automatically make systematic, scientific and quantitative, inquiry into social phenomena a futile endeavor. Making perfect predictions the standard for what is "science" would dramatically constrain the sphere scientific research. (See Jay Ulfelder's post for more on predictions). Climate scientists constantly debate the internal mechanics of their models of global warming - some predict faster rates, some slower. Does this mean that the underlying relationships described by those models (such as between CO2 concentration and temperature) should be ignored because the research is too "unsettled"? While deniers of climate change would argue yes, the answer here is a definite no.

Or to take an example from the recent blog debates about the value of election forecasting models. Just because Douglas Hibbs' "Bread and Peace" model (among other Presidential election models) does not perfectly predict President Obama's vote percentage in November, does not mean that we can learn nothing from it. One of the most valuable contributions of this literature is that systemic factors like the economy are significantly more relevant to the final outcome than the day-to-day "horserace" of political pundits.

What should be said, then, about Prof. Stevens concluding suggestion that NSF funds be allocated by lottery rather than by a rigorous screening process? Such an argument could only be justified if there were no objective means to distinguish what is and is not "scientific" research. If the criteria for what passes for real political science is simply the consensus of one group of elites, then from the standpoint of "knowledge," there is no difference between peer review and random allocation. This in fact would be the argument made that Thomas Kuhn, Popper's philosophical adversary, made about all science. But while Kuhn's criticism of a truly "objective" science was a useful corrective to 20th century scientific hubris, it too goes too far in this case, justifying an anything goes attitude towards scientific knowledge that is all too dangerous. Penn State Literature Professor Michael Bérubé' wrote a rather interesting article on this exact topic as applied to science at-large, noting the worrying congruence of the highly subjectivist approach to "science studies" adopted by some in leftist academia and the anti-science rhetoric of the far-right.

But now the climate-change deniers and the young-Earth creationists are coming after the natural scientists, just as I predicted–and they’re using some of the very arguments developed by an academic left that thought it was speaking only to people of like mind. Some standard left arguments, combined with the left-populist distrust of “experts” and “professionals” and assorted high-and-mighty muckety-mucks who think they’re the boss of us, were fashioned by the right into a powerful device for delegitimating scientific research. For example, when Andrew Ross asked in Strange Weather, “How can metaphysical life theories and explanations taken seriously by millions be ignored or excluded by a small group of powerful people called ‘scientists’?,” everyone was supposed to understand that he was referring to alternative medicine, and that his critique of “scientists” was meant to bring power to the people. The countercultural account of “metaphysical life theories” that gives people a sense of dignity in the face of scientific authority sounds good–until one substitutes “astrology” or “homeopathy” or “creationism” (all of which are certainly taken seriously by millions) in its place.

The right’s attacks on climate science, mobilizing a public distrust of scientific expertise, eventually led science-studies theorist Bruno Latour to write in Critical Inquiry:
Entire Ph.D. programs are still running to make sure that good American kids are learning the hard way that facts are made up, that there is no such thing as natural, unmediated, unbiased access to truth…while dangerous extremists are using the very same argument of social construction to destroy hard-won evidence that could save our lives. Was I wrong to participate in the invention of this field known as science studies? Is it enough to say that we did not really mean what we meant? Why does it burn my tongue to say that global warming is a fact whether you like it or not?

Why can’t I simply say that the argument is closed for good?
Why, indeed? Why not say, definitively, that anthropogenic climate change is real, that vaccines do not cause autism, that the Earth revolves around the Sun, and that Adam and Eve did not ride dinosaurs to church?

In the end, Bérubé calls for some sort of commensurability between the humanities and sciences, and I think this kind of coming together that is actually becoming the norm in political science academia, particularly as political theorists and quantitative political scientists still tend to fall under the same departmental umbrella:

So these days, when I talk to my scientist friends, I offer them a deal. I say: I’ll admit that you were right about the potential for science studies to go horribly wrong and give fuel to deeply ignorant and/or reactionary people. And in return, you’ll admit that I was right about the culture wars, and right that the natural sciences would not be held harmless from the right-wing noise machine. And if you’ll go further, and acknowledge that some circumspect, well-informed critiques of actually existing science have merit (such as the criticism that the postwar medicalization of pregnancy and childbirth had some ill effects), I’ll go further too, and acknowledge that many humanists’ critiques of science and reason are neither circumspect nor well-informed. Then perhaps we can get down to the business of how to develop safe, sustainable energy and other social practices that will keep the planet habitable.