‘Pundit Watch’ May Need Its Own Watchdog

Pundits are prized for making bold predictions, but they’re usually not held accountable for their forecasting accuracy. A news-prediction Web site is trying to change that with a pundit-tracking game, pitting the likes of MSNBC commentator Chris Matthews against technology blog TechCrunch and entertainment blogger Perez Hilton. While the idea is a good one, the execution is flawed.

Pundit Watch is a new spinoff of Hubdub, a site that lets users create questions about news and current events, and then predict outcomes. (ie, “Will there be more than 200 deaths by tornado in the US in 2008?”) Correct predictions yield positive returns in a fake currency – the size of the payout is dependent on how many other users answered the same way. HubDub’s founders pitch the free service as the news cousin of fantasy sports leagues, whose participants tie their fantasy fortunes to the real performance of chosen athletes.

For Pundit Watch, HubDub created accounts to track the predictions of nine “famous” pundits. Each week, HubDub posts up to three predictions each pundit has made and then scores their predictions just like other users. So far, each of the pundits has made between nine and 22 predictions, and between 1 and 12 of them have settled — meaning the question they were answering has been decided. Six of the nine have accounts in the black — led by the Silicon Valley blog Venture Beat — while three are in the red, with TechCrunchbringing up the rear.

“We follow all these pundits,” Nigel Eccles, chief executive of U.K.-based Hubdub, told me about his motivation for Pundit Watch. Prior to launching HubDub this year, he worked for Betfair, a sports-betting site, and for Johnston Press, an Edinburgh-based newspaper publisher. “We were wondering, how accurate are they?”

But Pundit Watch’s numbers are based on questionable interpretations of punditry. As Mr. Eccles has found, most pundits he initially wanted to track, including the Journal’s Walter Mossberg and Sports Illustrated’s Peter King, didn’t make many testable predictions. And much of what he counts as predictions — which should be about future, unknown events — are more like scoops and rumors, which insiders may already know to be true.

The criticism has spanned the spectrum of the leaderboard. First-place Venture Beat’s MG Siegler wrote on his blog, “A lot of the predictions PunditWatch has us making are simply us reporting rumors but not saying definitively one way or another if we believe them.” (Mr. Eccles responded that he interprets such writing “as we expect most readers would” interpret it.)

Hollywood blogger Nikki Finke, who is in fifth place but hadn’t heard her predictions were being monitored before I contacted her, told me about some of her supposed forecasts, “I was reporting on news. I never made the predictions they claim I made.” For example, Ms. Finke wrote that “TV sources tell me that [New York Times writer Bill] Carter has been saying he’d like to write a sequel to Late Shift…” Pundit Watch converted that into a Finke prediction that Mr. Carter would write the sequel this year.

And Erick Schonfeld, co-editor of TechCrunch — a site whose predictions Mr. Eccles called “generally terrible” — told me that “PunditWatch is generally terrible in distinguishing between opinion and predictions.” As an example, he cited the translation of the statement, “If I were Microsoft, I’d place a new bid for Yahoo at $33 per share, and let the offer stand for three days” into the prediction, “Microsoft coming back to the table by end of May.”

Mr. Eccles defended his approach. “To report a rumor is to give it some credence. If they report a rumor and it [turns out to be] true, they always say, ‘As we reported a month ago…” But in light of the criticism, he added, “If enough of the A-list pundits want to take over control of their own account, then we may allow them to do that.” Ms. Finke, for one, said she’s game: “I think columnists like myself should be accountable for what they report.”

Further reading: “Pundits shouldn’t lose or win gigs on the basis of how many of their predictions come true but whether they write interesting copy,” Slate’s Jack Shafer wrote earlier this year. I wrote two years ago about a more-rigorous attempt to evaluate pundits, which found that they better be writing interesting copy.

About The Numbers

The Wall Street Journal examines numbers in the news, business and politics. Some numbers are flat-out wrong or biased, while others are valid and help us make informed decisions. We tell the stories behind the stats in occasional updates on this blog.