Authors

Disclosure statement

The authors do not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and have disclosed no relevant affiliations beyond their academic appointment.

But is this what the survey is actually telling us? To us as researchers studying science communication and public understanding of science, factors in the survey itself and in the way the media report on it raise questions about how much to read into these findings.

Survey’s results aren’t publicly available

Researchers are trained to look for the original methods whenever they read a new study, especially if the results are surprising. Learning how the study was done provides information that helps determine whether the science is sound and what to make of it.

The chocolate milk survey is described as a nationally representative survey of 1,000 American adults, but this is impossible to verify without seeing how respondents were selected. Likewise, how the survey was conducted – whether it was a phone or online survey, for instance – can have significant impacts on its accuracy. Research suggests that phone surveys may be less accurate than online surveys because they require people to give their responses out loud to another person instead of quietly clicking away in privacy.

For instance, someone who holds racist views may feel comfortable checking a box about it but might avoid openly professing those opinions on the phone to a stranger. It’s unlikely the chocolate milk survey ran into such problems, but depending on the questions asked, other challenges may have presented themselves.

Likewise, it’s difficult to interpret the results of the chocolate milk question without seeing how it was worded. Poorly phrased or confusing questions abound in survey research and complicate the process of interpreting findings.

Social scientists call this a problem with validity: the question doesn’t really measure what it’s supposed to measure. Of course, without seeing how the question was worded, we can’t know whether the chocolate milk question had validity.

Indeed, early media coverage focused on the 7 percent statistic but left out the fact that 48 percent of respondents said they don’t know where chocolate milk comes from. This gives context to the 7 percent number. While it’s conceivable that 7 percent of the population doesn’t know that chocolate milk is just milk with chocolate, the idea that a full 55 percent — over half of adults — don’t know or gave an incorrect response begins to strain credulity. This points toward a confusing survey question.

We reached out to Lisa McComb, the senior vice president of communications for Dairy Management, Inc., about the survey. She confirmed that it’s not publicly available. “The purpose of the survey was to gauge some interesting and fun facts about consumers’ perceptions of dairy, not a scientific or academic study intended to be published,” she told us.

Story feeds a popular narrative — and media missed it

Questions about the original findings aside, there’s reason to explore how the media covered the chocolate milk survey.

The results were instantly shared and republished by a mind-boggling number of outlets (a Google Trends search for “chocolate milk” and “brown cows” shows a spike beginning June 15th). This factoid likely garnered such massive attention because it feeds into a popular narrative about American ignorance and science illiteracy.

Studies suggest we are more likely to accept new information when it confirms what we already want to believe. In this case, the chocolate milk statistic fits well with the notion that Americans are fools, so it’s accepted and republished widely despite the numerous red flags that should give scientifically minded people pause.

But the fact remains that many reporters and news outlets decided to run the story without having seen the original results, instead citing one another’s reporting. This led to some interesting challenges when trying to fact-check the survey: The Washington Post links to Food & Wine’s coverage, which linked to the Innovation Center’s website, which originally publicized the survey results. The Innovation Center, in turn, links to a story on Today.com, which linked right back to the Food & Wine article. This type of circular reporting without seeking out the original source can lead to the spread of misinformation. Unfortunately, as news stories quickly pop up and go viral online, it’s all too likely that we will continue to see such problems in the future.

Importantly, none of this disproves the notion that some adults believe chocolate milk comes from brown cows. It certainly does nothing to undermine the need for increased science education in the United States or suggests that a better understanding of our food production system wouldn’t be beneficial to society. All of these points are still valid. Likewise, this isn’t necessarily evidence that the survey itself is flawed. As McComb notes, the survey is not a scientific one and isn’t meant to be taken as evidence of Americans’ knowledge (or lack thereof) of dairy products. The problem is that it’s being reported on as though it is.

So this survey did point out a lack of science understanding. Ironically, rather than showing Americans’ ignorance of chocolate milk’s origins, the fact that media coverage of this survey was reported so widely and with so few caveats instead showed that many people are not skeptical of the science they read.