Friday, April 22, 2011

CJR's Campaign Desk: "Testing the Truth-O-Meter"

Amid the controversy over a recent PolitiFact rating of a Democratic Party Medicare ad, Joel Meares of the "Campaign Desk" at Columbia Journalism Review recycles a handful of my stock criticisms of PolitiFact in an April 21 post:

(M)ore interesting than the details of this mini controversy about a standard-issue political ad are the questions it raises about the PolitiFact method and the use of its Truth-o-meter. The truth, or at least the truthiness, of some matters, seems to lie in degrees that Adair’s innovative six-level gauge may not quite capture.

That sounds a bit familiar ("Unless PolitiFact comes up with a more precise way of grading, the "Truth O Meter" should be tossed on the scrap heap"--June 1, 2008).

Meares:

But PolitiFact’s “Pants on Fire” assessment still feels unduly harsh for some reason. What makes this, for instance, not simply “False”? Or, given that it is a missing phrase away from seeming quite reasonable, “barely true?” Or, if you don’t buy that—that the numbers are wrong and the vote is not binding is pretty damning—wouldn’t misleading be a more accurate description than “outright lie”?

Meares doesn't sufficiently credit PolitiFact for its trickiness. It doesn't label anything as an "outright lie" unless one counts the annual "Lie of the Year." The "Pants on Fire" label denotes a "ridiculous" claim while communicating "lie!" based on the "liar, liar, pants on fire" rhyme that all children seem to learn at some point. As for the difference between "False" and "Pants on Fire," I've regularly pointed out that devising an objective method of determining the property of ridiculousness is probably beyond human capacity. In addition, PFB highlighted a complementary assessment of PolitiFact from the "Engineering Thinking" blog where Ed Walker made similar points.

Bottom line: The "Truth-O-Meter" is subjective. We shouldn't be surprised that it doesn't operate consistently.

Unfortunately, Meares whiffs on his opportunity to slam another PolitiFact problem:

The concise rulings, says Adair, “are a tremendous reader service.” They cut through the political rhetoric to issue concise judgments and when pooled, like on this page breaking down the statements of Michele Bachmann, they can reveal quite a lot about a candidate or organization’s commitment to accuracy. “The tradeoff,” admits Adair, “is that sometimes complexity doesn’t fit neatly into our six ratings.”

There it is. Adair is still advocating using the collected rulings for individuals as a type of report card as though PolitiFact is immune to selection bias. Meares ought to subject that idea to some hard questions like those Eric Ostermeier has asked.

Though Meares is probably too easy on PolitiFact and PolitiFact's Bill Adair, he puts a helpful focus on PolitiFact's flawed methods--methods that help amplify newsroom bias in PolitiFact's fact checking.