Assessing the effect of news source ratings on news content

Gallup and Knight Foundation’s 2017 Survey on Trust, Media andDemocracy found that Americans believe the media landscape is becoming harder to navigate. A majority of Americans say the plethora of information and news sources available makes it harder, rather than easier, to be informed today. The proliferation of online news sources that fail to adhere to the basic journalistic standards of accuracy and accountability contributes to the challenge of determining what is true or important.

An April 4-11, 2018, Gallup/Knight Foundation survey experiment of 2,010 U.S. adults sought to test the effectiveness of a news source rating system designed to bolster online news consumers’ ability to identify misinformation, or so-called “fake news,” meaning false or misleading content. The system identifies news organizations as reliable (by showing a green cue) or unreliable (using a red source cue) based on expert evaluations of their work, funding and other factors.

Three key findings emerged from this survey experiment:

The news source rating tool worked as intended. Perceived accuracy increased for news headlines with a green source cue and decreased for headlines with a redsource cue. Participants also indicated they were less likely to read, like or share news headlines with a red source cue. The source rating tool was particularly effective for participants who correctly recalled that experienced journalists devised the ratings, compared with those who did not recall that information.

The source rating tool was effective across the political spectrum. The perceived accuracy of news articles with a red source cue decreased similarly among Republicans and Democrats, with the sharpest decline occurring when the headlines had a clear political orientation that matched the users’ political beliefs.

The source rating tool did not produce known, unintended consequences associated with previous efforts to combat online misinformation. Our experiment did not produce evidence of an “implied truth effect,” an increase in perceived accuracy for false stories without a source rating when other false stories have a source rating, or a “backfire effect,” a strengthening of one’s false beliefs following a factual correction.

This survey experiment offers compelling evidence that the use of an online tool to indicate news organization reliability increases healthy skepticism when individuals consume news online. Gallup and Knight Foundation acknowledge support for this research from the Ford Foundation, the Bill & MelindaGates Foundation, and the Open Society Foundations.