Talking about scientific results without overstating the findings

Sometimes the most productive conversations at scientific conferences happen outside the formal sessions. That was the case for Psychology Professor Maureen Callanan, who attended a meeting a few years ago during which she participated on a panel about research on diversity in developmental psychology.

Afterwards, she and her colleague Susan Gelman, a professor of psychology and linguistics at the University of Michigan, discussed researchers' tendency to take a simple finding and state it as a universal truth, generalizing beyond the scope of an individual study.

"It's the difference between saying, 'What the 5-year-olds in this study did…' and 'What 5-year-olds do,'" explained Callanan, a developmental psychologist. "To imply that all children behave a certain way ignores that most research in developmental psychology is based on studies of white, middle-class children."

This use of "generic language" gives the impression there's no nuance, and it can be misleading, Callanan and Gelman agreed. And they started to wonder how prevalent it is in the literature.

Fast forward to August 26, when Callanan and Gelman published an article they coauthored with Jasmine DeJesus and Graciela Solis in the Proceedings of the National Academy of Sciences, "Generic Language in Scientific Communication." Their analysis of nearly 1,150 psychology journal articles published in 2015-16 revealed that generic language was used in 89 percent of the research summaries. The use of such "universal" language is particularly troublesome given that 73 percent of those articles made no mention of the race or ethnicity of their participants, one of several demographic factors that could limit the relevance of findings about one group to another; similarly, 79% of articles did not mention the income level of the sample, and 74% didn't note language background.

"Using generic language is especially problematic if researchers are overgeneralizing from selective or limited samples," said Callanan. The authors noted that the field of psychology even has an acronym that acknowledges the homogeneous nature of typical samples: WEIRD, which stands for participants from Western, educated, industrialized, rich, and democratic societies.

The authors identify several forces contributing to the use of generalizations, including a desire to make the work accessible to broad audiences—and appealing to funding agencies. However, glossing over findings and failing to describe study samples could lead to exaggerated conclusions, they caution.

"Because scientists are encouraged to reach broad conclusions, their writing may sacrifice precision in favor of bolder claims," said DeJesus, assistant professor at the University of North Carolina at Greensboro and the article's first author.

A follow-up study indicated research summaries that used generic language pack more punch than more nuanced summaries, at least among the undergraduates and others survey participants, who consistently rated them "more important."

"The challenge is figuring out how to report findings in a way that's honest and also gets the work noticed," said Callanan, who speculated that this phenomenon may also occur in fields other than psychology. "I'd like to see researchers do a better job of being accessible and concise, while also fully acknowledging variability and the limits of our evidence."

At a minimum, Callanan urged researchers to refrain from making sweeping generalizations and to consider referring to their participants' behavior in the past tense to emphasize that the findings are limited to the study and not necessarily applicable to other groups or a predictor of future behavior.

"A single study really isn't enough basis for a broad generalization," she said. "That type of language might be more appropriate when reviewing a whole literature and summarizing across multiple studies."

Even with a representative sample, it may not be appropriate to generalize, noted Callanan, who has wrestled with these issues in her own work. "I'm less focused on findings that can be generalized, because there is always nuance," she said. "There are always differences, even among white, middle-class kids."

Callanan also sees broader negative consequences of using generic language to talk about scientific findings that are based on samples from very specific groups—often middle-class, white families. "It makes people think that's the norm, and if findings among another group are different, it is considered a deficit," she said. "That's the danger."

Rather, it's important to study individual communities on their own terms, not as outliers or exceptions to the norm, she said. Avoiding generic language is part of a broader effort to be inclusive, an effort that has shaped Callanan's work over three decades. Her own research focuses on how children learn about science and the world around them, and she appreciates the tremendous range and variety she has observed.

"People have different ways of learning about the natural world," she said. "I'm not looking anymore for the one way children learn. I think that puts us on a false path. We should be looking not just for generalizable findings but also for variation, because there is going to be both in whatever topic we're studying."

Citation:
Talking about scientific results without overstating the findings (2019, September 6)
retrieved 15 September 2019
from https://phys.org/news/2019-09-scientific-results-overstating.html

This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no
part may be reproduced without the written permission. The content is provided for information purposes only.

User comments

Well, this is a step forward, used to be that psychologists worried that their preferred study populations were students. The remaining problems seems to be of random sampling, likely double blinded studies so that no observer bias creeps in, and limited population sizes. If your factors are independent (which is unlikely here) you need ~ square root of population size to extrapolate without a good hypothesis of a pattern (law).

Psychology lacks in many ways good hypotheses, so they would need ~ 100,000 individuals in their studies in order to be sure their finds has a chance to cover the ~ 10^10 people of Earth. Or they could try to find good hypotheses...

E-mail the story

Talking about scientific results without overstating the findings

Note

Your email address is used only to let the recipient know who sent the email. Neither your address nor the recipient's address will be used for any other purpose.
The information you enter will appear in your e-mail message and is not retained by Phys.org in any form.

Your message

Newsletter sign up

Get weekly and/or daily updates delivered to your inbox.
You can unsubscribe at any time and we'll never share your details to third parties.

Your Privacy

This site uses cookies to assist with navigation, analyse your use of our services, and provide content from third parties.
By using our site, you acknowledge that you have read and understand our Privacy Policy
and Terms of Use.