“While there is a long tradition of measuring news reports’ accuracy post hoc … substantially less work has examined the processes by which journalists seek to attain accuracy,” write Canadian journalism researchers Ivor Shapiro, Colette Brin, Isabelle Bédard-Brûlé and Kasia Mychajlowycz in their newly published paper, “Verification As A Strategic Ritual: How journalists retrospectively describe processes for ensuring accuracy.”

It’s perhaps the first paper to offer a look at how working journalists view and practice verification.

The researchers found that verification is widely seen as essential and core to a journalist’s work. But at the same time, the methods for achieving accuracy vary from one journalist to the next. There is no single standard for verification, and not every fact is treated the same.

“A small, easily checkable fact needs to be checked; a larger but greyer assertion, not so much — unless it is defamatory,” they write. “Thus, verification for a journalist is a rather different animal from verification in scientific method, which would hold every piece of data subject to a consistent standard of observation and replication.”

The method

To gather data, the researchers interviewed 28 Canadian journalists (men and women; French and English), half of whom had recently won an award for a piece of work; the other half were selected after the authors chose “14 stories semi-randomly from a constructed population of texts commensurate in length with the set of award-winning stories.”

They met with the journalists and spoke about the verification practices used to produce the stories.

Like good researchers, they noted one weakness to this approach.

“We were relying entirely on the subject journalists’ own accounts of their work, with no available means to verify (!) the truth of those accounts,” they write. “Apart from the possibility of the subjects varnishing their verification efforts, we also were limited by the capacity of their memories.”

Variations in verification

One theme in the paper is that different journalists practice verification in different ways — though they all agree it’s hugely important.

‘‘There’s no point in being a journalist if you’re not going to relay accurate, correct factual information to the public,” one interviewee said.

Shapiro told me by email that, “while journalists see the norm of verification as quite pivotal to their professional identity, the recognition of this norm is not quite matched by the kind of methodological discipline that Kovach and Rosenstiel speak of.”

Rosenstiel, a member of Poynter’s National Advisory Board, contributor to Poynter.org, and the executive director of the American Press Institute, disagrees with Shapiro’s assessment of how the findings relate to the treatment of verification in “The Elements of Journalism.”

“Far from challenging what we found in Elements, the study reinforces it,” he told me. “We conclude, as they do from their 28 interviews, that journalists aspire to accuracy and being truth tellers but lack standard routines or sufficient intellectual training for seriously doing it. We note that these routines are highly individualized and idiosyncratic. We also outlined some of those individual routines as a way to suggest how to make this discipline more conscious and more serious. That, indeed, is the point. Elements is a call for journalists to live up to their aspirations with more rigor, not a celebration of current practice.”

So why are journalists unable to systematize their aspirations of accuracy? In that regard, the paper shared an important fact: there’s little specific guidance for verification offered in journalism textbooks. They write:

Many journalism textbooks are devoid of references to verification or fact-checking (e.g. Frost 2002; Gaines 1998; Harcup 2004; Harris and Spark 1997; Spark 1999) or confine themselves to only the briefest references to the importance of double-checking basic facts such as names, ages and locations, and the necessity for more than one source where a charge of misconduct is made (Lanson and Stephens 2008).

I asked Shapiro, who is chair of the School of Journalism at Toronto’s Ryerson University, why something so important has not been standardized and incorporated into core teaching material.

“Columbia [University’s journalism school] has a course called ‘evidence and inference’; Ryerson has a course called ‘Exactly so’, and I am sure many schools try to teach verification strategies and critical thinking…. but textbooks are another matter,” he said. “There are probably some great ones out there that do a good job of tackling this question, but I haven’t seen them yet.”

This partly explains the varying ways journalists practice verification: They come into the profession having been taught in different ways, or perhaps not much at all.

And yet, we place accuracy on a pedestal.

“We actually found the interviews journalists’ passion for accuracy often inspiring,” Shapiro told me. Journalists “have to juggle competing priorities in delivering products that their employers can sell. Not one of our 28 interviewees gave us any reason to think that s/he does not take the responsibility for accuracy very seriously.”

“They are professionals, but they are artists, not scientists – and, mostly, artists on a deadline,” Shapiro added.

One positive trend I’ve seen in recent years, which wasn’t part of this specific research, is that the increasing use of user-generated content by newsrooms has resulted in organizations creating a defined verification process. (Read more about the processes in place at Storyful, the BBC, AP and CNN’s iReport.)

This is encouraging, particularly as new technologies and media continue to transform newsgathering and fact finding at a rapid pace.

How journalists verify

“There’s no hard and fast rule about any of this stuff,” one interviewee told the authors. “You have to exercise your judgment all the time.”

‘‘[T]o me, verification is much more rooted in the actual reporting process, step by step and looping back in upon itself,” said another.

Among the interviewees there were those who applied a disciplined approach to verification.

“Some arrived for the interview armed with indexed binders full of source materials; some had clearly refreshed their memories of the reporting by reviewing their notes, and related articles, before their meetings with us; one checked additional facts and followed-up via email,” the researchers write.

A commitment to verification may be shared across journalists, but this research suggests any shared norms are combined with variations in practice.

“Methods for ensuring accuracy varied greatly, with some factual statements relayed, with or without attribution, based on a single subject’s word, while others were rigorously triangulated,” the authors write. “Strongly idealistic statements about the need for verification were often made during the course of the same interview as were indications of methodological ambiguity.”

In terms of specifics, here are a few findings from the research:

On checking names: “An almost universal practice among participants is asking sources to spell their own name to ensure correct spelling, either at the beginning or the end of the interview.” Some journalists also check names against official sources.

On offering sources pre-publication review: “Despite some evidence in the literature that partial pre-publication review is not the taboo it used to be (Stoltzfus 2006; Carr 2012), our subjects displayed a strong sense that it was a discouraged practice.”

On quotes: “…methods for checking the accuracy of quotes vary greatly. Some reporters routinely record and transcribe interviews, while some record but rarely transcribe and others rarely use recorders at all. Some check quotes against tapes only if there is a specific concern, such as difficulty hearing, or the threat of libel litigation.”

On a source’s personal history: “Facts relating to a source’s personal history are considered not to require verification … or are simply not verified because there is no practical way to do so.”

A ‘Strategic Ritual’

In the end, the researchers likened the journalists’ commitment to verification to a doctor’s oath to “do no harm”:

Put another way, using the language of professional identity rather than professional ethics, verification might be seen as a ‘strategic ritual’, as Tuchman (1972) said of the aforementioned (and perhaps now rather old-fashioned) idea of ‘objectivity’ — something that legitimizes a journalist’s social role as being demonstrably different from other communicators.

Now let’s work on giving them better guidance and practices to make the ritual even more real, and reliable.

As Rosenstiel told me, “the aspiration to vet the news is an essential goal of most journalists, but … the processes for living up to that goal are not well-defined and not rigorous enough. And for journalism to survive, much more needs to be done to give the process of verification more throw weight.”

As a former NZ news reporter and chief reporter, I found this interesting. I do think, however that the study cannot be generalised beyond the borders of Canada.

Here is an NZ perspective the four points you make at the end of the article:

1) On checking names: This is absolutely necessary. Every journalist encounters a Jon Smyth at some point in their careers.

2) On offering sources pre-publication review: This is not only discouraged in NZ, but banned at most newspapers. Certainly it is banned at the NZ Herald.

3) On quotes: It is not possible to transcribe taped interviews in a modern newsroom – there simply isn’t time. NZ journalists are expected to be proficient in shorthand (at least 80wpm), and it is certainly much faster to flick through a notebook than it is to scroll backwards and forwards through a recording device. My understanding is that most North American journalists have no training in shorthand.

4) On a source’s personal history: I think most good NZ journalist would check an interviewee’s background if it is necessary and possible to do so within a reasonable timeframe (usually the same day as the interview).

S

http://twitter.com/JohnHamer John Hamer

Excellent column, Craig, and this study is indeed interesting. However, even the authors admit a major weakness: relying on the journalists’ own accounts of how hard they worked to verify facts. What if the researchers had asked the subjects of the stories how accurate they were and how hard the reporters worked? Might that not have been even more enlightening?

Here at the Washington News Council (http://wanewscouncil.org) in Seattle, we now have a solid 15-year track record of reviewing and ruling on complaints from people who feel they have been damaged by inaccurate, unfair or unethical stories about them or their organizations. To see a particularly egregious recent example, read the “Leschi Elementary Community vs. KIRO7 Eyewitness News” case on our website. Watch the hearing at which we carefully examined the accuracy of the KIRO broadcasts and voted in public on how well the reporter did his job. We also invited audience members to vote at the hearing and citizens to vote later online. That’s true accountability.

it’s all well and good to give journalists “better guidance and practice” on verification, but in today’s high-pressure newsrooms with diminished resources, fewer seasoned editors and constant digital deadlines, “aspirations” of accuracy may not be enough. Add in some journalists’ inexperience, biases, preconceptions, biases, egos, vindictiveness — plus the perennial pursuit of journalism prizes — and you have the potential for real damage. We’ve seen far too many such cases, despite the Society of Professional journalists’ Ethics Code admonitions to “Minimize Harm” and “Be Accountable” — often honored only in the breach.

Kovach and Rosenstiel’s more recent book, “BLUR: How to Know What’s True in the Age of Information Overload,” offers lots of constructive caveats for readers, viewers and listeners on how to assess and verify what they find in the media. John McManus’ new book, “Don’t Be Fooled: A Citizen’s Guide to News and Information in the Digital Age,” is equally good. He suggests that every story needs to pass the “SMELL” test: a cool tool. Check it out. Bottom line: News consumers today must be more active, skeptical and engaged. They should talk back to journalists and hold them publicly accountable for inaccurate, unfair or unethical stories. It takes more work, but it’s vital for journalism, the public — and democracy.

Clayton Burns

One suggestion that I have is careful aftermath analysis layered through Google advanced search for various time periods.

An example might be the idea that the brain is composed of 100 billion neurons, which appeared again and again in The NYT recently before a realization transpired.

How is it possible that so many copy editors at the paper could have made the same mistake?

You have to hard wire it into your bones–I hope that is not a mixed metaphor–that aftermath analysis through Google is critically important because it is so easy to be oblivious to salient reactions, for example. It seems to be a ritual at many papers.

On the spectrum, though, The NYT’s Mark Thompson “error” is more significant. Despite the recent developments in the UK, the story has gone somewhat quiet at The NYT. There is no significance to that, of course.

http://twitter.com/ivorshap Ivor Shapiro

Thanks for this attention to our study, Craig. I think every line of your piece was verified and accurate! (Phew.) I also agree with everything that Tom Rosenstiel is quoted as saying, but would like to comment on the famous line from The Elements of Journalism stating that “the essence of journalism is a discipline of verification.”

It is true that, as Rosenstiel points out, The Elements describes quality journalism at its best – providing exemplary models of what one might call elite practice – rather than presenting a snapshot of journalism as it actually is. Therefore, it is noteworthy that more than half of our sample of interviewees consisted of award-winning leading journalists. (Half of the interviewees were semirandomly selected as winners of leading national awards, while the “control” sample also happened to include some award-winners.)

The picture of strategic compromise that our study paints, therefore, is interestingly different from the more definitive Kovach/Rosenstiel idea that the “essence” (not “ideal”) of journalism “is” (not “should be”) a “discipline of” (not “selective attempts at”) verification.

But yes, we also found ample evidence for accuracy as a professional norm to which journalists universally subscribe with considerable vigour in practice. Therefore I think the closing comments (by you and by Rosenstiel) are spot on. As we scientists like to say: nailed it!

http://www.facebook.com/gwolin Glenda Wolin

In my 40 years as a journalist, about 35 of which were as an editor, I found that many reporters don’t verify at all. They often don’t ask sources how to spell their name, even when a diacritical mark may or may not be used. They take for granted much of what a source says if it’s not a controversial subject, or they look on one website. And with a requirement to cq all names, reporters cq when they think they know instead of checking. I remind them that the cq is your pledge that you checked the accuracy, which essentially makes you a liar if it’s wrong. But many reporters are so focused on the bigger picture — which is essential, too — they let the smaller things go. I always tell them that if you can’t get such an easily verifiable fact as the name right, readers can’t help but wonder what bigger elements are wrong.