Friday, January 27, 2017

ALTERNATIVE FACTS: Scientists got 'em too!

“WHAT”, I hear you saying, “That is precisely the point. Science has
the facts and the facts are the facts – no alternatives. Anyone who makes
statements not supported by the data are presenting fiction, not alternative
facts.”

Seen in an Ottawa book store.

Yet maybe we scientists shouldn't be so smug about our FACTS.
As the editor and reviewer of countless papers, I am struck by how often two
(or more) fully qualified, insightful, and fair scientists can make opposite
assertions of fact based on the same data and analysis. Most obviously, data
are often interpreted in the light of preconceived expectations (we are all
Bayesians) and, so, faced with a series of mixed results, scientists consciously
or unconsciously emphasize the subset of results that support their expectations,
largely ignoring or discounting or explaining away the rest. “Ah”, I hear you saying “Those are
interpretations, not facts. Facts belong in the results, interpretations are in
the discussion.”

I am not so sure. Everyone knows that the same data can be
analyzed in multiple ways, each yielding different outcomes. Hence, the statements
and statistics in the results are interpretations too, not facts. “Ok, then,
the DATA are the facts.” Yet, data are not collected with error, and –
sometimes – are biased. Data collected in different ways at different times in
different years can yield different outcomes. Different measure of central tendency
(mean, median, etc.) yield different numbers. Different data transformations yield
different outcomes. Given that data have no meaning without interpretation,
even in the narrowest possible sense of a central tendency, data are not facts
either.

Alternative facts?

“Ah, but this is what measures of uncertainty are for.” Indeed,
standard errors, confidence intervals, credibility intervals and so on can –
and should – be calculated but, again, they have no meaning without interpretation.
Anytime you look at a result and a credibility interval and say “I therefore
conclude the evidence more strongly supports option 1 than option 2.” Sure, but
some other scientist can come along and say “I require a higher degree of
confidence, so my conclusion is different.” Or. “I would use a different
measure of uncertainty.” Or. “You did not include all sources of uncertainty.”
Or “You didn’t consider option 3.” And, of course, we ultimately need to accept
an outcome one way or the other or we never act on the information we have
collected.

So, are there any real facts out there, any “objective
truths?” Yes, of course, there are many of them, but many, perhaps most, of
those are – ultimately – unknowable. Instead, they are “latent variables” about
which we try to make inferences based on our imperfect surrogate measures.

Andy Gonzalez and I with our alternative facts

Alternative facts, which I have just argued are pervasive in
science, depend on the level of inference. “Evolution isn’t a theory, it is a
fact.” Sure, evolution as a process
is a fact. However, evolution in any given case (a particular population facing
a particular environmental change) might not be. Don’t believe me? Just look at
the knots people studying adaptation to climate change or fisheries have tied
themselves into trying to say that evolution has or has not occurred in any
given instance. “Look at allele frequency changes,” you might say. But no one
cares if those are changing (and they ALWAYS are), we care about the evolution
of the traits. “Ok, then, measure the genes influencing those traits.” Yes,
that works but what matters is the effect size: how much of the change in
phenotype is the result of evolution? That is extremely hard to measure. Climate
change is a fact, certainly, but stating that it is a fact is not informative
or helpful. We need to know the effect size (rate of change) and its consequences
– and then it depends on the place, the time, and the measure taken. Plenty of
room for “alternative facts” in this context.

Alternative facts?

So, maybe us scientists shouldn’t be so smug. If we are ever
going to state conclusions as facts (as opposed to facts being unknowable),
then room exists for alternative “facts” in the same sense. Otherwise we wouldn’t
have debates and arguments. Otherwise, we wouldn’t need to replicate our
studies, and those of other scientists. Otherwise, we wouldn’t have
retractions.

The difference between science and propaganda is that scientists
are willing to give up our “facts” when enough evidence suggests they aren’t “facts”
after all.

--------------------------

This post was motivated by reading scientific papers where authors are testing an hypothesis, obtain results that are mixed with respect to that hypothesis, emphasize only those results supporting the hypothesis, and then conclude in favor of the hypothesis. Presumably this sequence of events usually transpires without INTENT to deceive - and, regardless, the consequences are rarely dire. The same cannot be said for the current US government.

2 comments:

I'm glad that you added the paragraph at the bottom. I still find this post to be in extraordinarily poor taste at a moment when everybody with any moral compass at all is freaking out about Trump's crusade against truth. From the title of this post on down, you persistently draw a false equivalence between the "alternative facts" of Trump and his ilk, and the sometimes problematic relationship between absolute truth and science's iterative, flawed attempt at approximating that truth. But Trump's "alternative facts" are really simply lies – deliberate, bald-faced lies, told for reasons of power and profit. Science, on the other hand, has no truck with such lies; any scientist caught doing such things (deliberately falsifying data, for example) gets kicked out of the club. So there is no equivalence there at all, and using the term "alternative facts" to refer to what scientists do is deeply offensive, and potentially quite damaging as well. When people at government agencies like NASA and the EPA are in fear of being muzzled and losing their jobs because of Trump's "alternative facts", and the whole world is at risk of suffering catastrophic effects from climate change because of "alternative facts", and Muslims and Mexicans are being reviled and barred from the U.S. because of "alternative facts", this post seems more than a little tone-deaf. I feel sure that your heart is in the right place, Andrew, which is why I'm not more angry about this. But I don't think you understand how wildly inappropriate this post really is.