Journalists tend to struggle with science. One might expect fact check journalists to do a better job handling science. And maybe they do, but it still often looks like fish out of water.

Case in point:

At the request of The Associated Press, eight climate and biological scientists graded for scientific accuracy what a dozen top candidates said in debates, interviews and tweets, using a 0 to 100 scale.

The AP’s 0 to 100 scale gives every appearance of being a subjective scale. We have nothing in the AP’s presentation to suggest scientists were given any objective frame of reference for applying the grades. A “50” from one scientist might correspond to a “14” from a different scientist. Social science offers some methods for helping draw scientific conclusions from these types of judgments. But it doesn’t look like the AP bothered with anything like that. The AP is a press organization, after all, not a social science organization.

Note also the AP chose a mix of climate and biological scientists to judge the accuracy of statements about climate science. What’s the rationale behind using biological scientists to judge scientific statements outside their area of expertise? The fact check gives no reason behind its approach, so we are left to wonder.

The conservative Weekly Standard criticized the AP fact check for soliciting the opinion of climate scientist Michael Mann. Mann carries one of the more recognizable names in climate science, thanks in part to his role in the “Climategate” hacked email story from a few years ago. The Weekly Standard scores a point for noting that Mann likely fails to count as an impartial judge. The Standard illustrates that point readily enough with the AP’s quotation of Mann:

“This individual understands less about science (and climate change) than the average kindergartner,” Michael Mann, a Pennsylvania State University meteorology professor, wrote of Cruz’s statements. “That sort of ignorance would be dangerous in a doorman, let alone a president.”

Has Mann familiarized himself with scientific studies of the climate science understanding of kindergarteners? That’s very doubtful. It’s doubtful any such studies exist. The AP quotes Mann offering his unscientific opinion.

But the root problem with the AP’s fact check is not Mann’s involvement. The problem is the AP putting the label “fact check” on the unscientific opinions of scientists. The quotation from Mann is just a glaring symptom of that root problem. Presenting the opinions of the scientists as numbers on a scale from 0 to 100 doesn’t change the fact that the numbers represent opinions.

The Associated Press embarrasses itself with this deeply unscientific “fact check.” And the scientists who willingly participated in this sham share the shame.

2 Comments

AP did not embarrass itself and neither did the scientists who spoke about their field. No, the AP’s fact check is not scientific research published in a peer-reviewed journal. It was a news organization asking experts what they thought about something they study professionally.

Oh, how controversial.

For people interested in the scientific research on cultural conflicts like this and the mechanisms behind it, check out Dan Kahan’s work: http://www.culturalcognition.net/