Team calls for creative approaches to accountability

By Eddy Ball

As she discussed the limitations of citation analysis, Sarli said, “There are so many more ways to tell the story.” She and Holmes have developed a hybrid solution with more than 80 indicators, but Sarli cautioned the audience, “I can’t give you an SOP — a standard operating procedure.”
(Photo courtesy of Michael Gaske)

“The Becker model is in perpetual beta,” Holmes said. “We’re always encouraging new input.” She pointed to future efforts in what she called ontology extension, — expanding the set of concepts within the research evaluation domain and the relationships between those concepts.
(Photo courtesy of Michael Gaske)

Librarian Cathy Sarli and bioinformaticist Kristi Holmes, Ph.D., urged scientists and administrators to think outside the citations analysis box to quantify research outcomes and translational impact, during an April 9 presentation at NIEHS.

The Washington University in St. Louis (WUSL) researchers are among several people now thinking divergently about a tracking system framework for evaluating biomedical research — one that includes indicators of impact on public and community health, economic benefit, medical practice, patient outcomes, regulation and legislation, new research directions, and even product development and insurance reimbursement.

Hosted by NIEHS Health Scientist Administrator Kristi Pettibone, Ph.D., who is also involved in high impact tracking for the Institute’s Partnerships in Environmental Public Health (PEPH) programs (see related story), Holmes and Sarli presented the results of ongoing efforts using the
Becker Medical Library Model for Assessment of Research Impact. (https://becker.wustl.edu/impact-assessment)
Their model challenges the bibliometric analytical paradigm for accountability long enshrined in tenure/promotion guidelines and grant/program review, with a novel multipronged
guide (https://becker.wustl.edu/sites/default/files/becker_model-reference.pdf)
with more than 80 indicators for metric analysis of biomedical research.

Beyond the assumption that a high citation count equates with high impact

Sarli, (https://becker.wustl.edu/about/library-info/staff/cathy-sarli)
who delivered the first part of the team presentation, referred early in her talk to a statement in the conclusion of a 2007
commentary (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1929109/?tool=pubmed)
by Robert Wells and Judith Whitworth, M.D., on assessing research outcomes. “It is no longer enough to measure what we can,” they wrote. “We need to measure what matters.”

For people involved in today’s biomedical research, Sarli explained, what can be measured easily — frequency of citation — is only a small part of what really matters. And what really matters — the meaningful research outcomes — she added, may be some of the most difficult information to find and quantify.

Spurred by a request in May 2007 by WUSL researcher
Mae Gordon, Ph.D., (http://ophthalmology.wustl.edu/Faculty/MaeOGordonPhD/tabid/337/Default.aspx)
for assessment of the impact of the Ocular Hypertension Treatment Study (OHTS), the Becker team began looking beyond citations for impact in other areas. Gordon had anecdotal evidence of the OHTS study impact, but she wondered if quantitative, independent indicators were available.

Sarli and Holmes, who joined the effort later on, began the search for novel indicators. Using resources from databases, personal anecdote, organizational guidelines, trade publications, and other so-called grey sources, the team began accumulating evidence that OHTS research had impacted something beyond the world of the PubMed-indexed literary canon, an odyssey they discuss in a 2010
article. (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2801963/?tool=pubmed)

Challenges for measuring what really matters

In her part of the presentation,
Holmes (http://vivo.wustl.edu/display/n4754)
reinforced Sarli’s assessment of the challenges of using the new evaluation paradigm. She conceded it’s a messy, time-consuming, and labor-intensive process that is non-linear and non-standardized.

“This is certainly not an exercise for the faint of heart,” Holmes told the audience. The supporting documentation may not be publicly available or easily accessible, requiring efforts beyond the standard search. And, it can take creative thinking to establish a direct correlation between the research and the outcomes.

“Researchers need to learn to tell their stories,” Holmes said, “be more proactive, and use strategies that enhance their research impact.” As budgets tighten and legislators look more critically at funding for scientific research, Holmes predicted, scientists need to demonstrate, clearly, the returns on investment of taxpayer dollars.

Holmes suggested that biomedical researchers begin thinking, at the outset, about potential effects and outcomes in terms of value and benefit beyond publications. She suggested
strategies (https://becker.wustl.edu/impact-assessment/strategies)
as simple as using the declarative voice in titles, using names consistently, and writing clearer abstracts with a general, as well as scientific audience, in mind.

Improved dissemination strategies, she added, can include such activities as setting up websites for research output, publishing in open access journals, creating public access repositories for data, taking advantage of social media and outreach opportunities, and tracking the impact of studies according to the kinds of global measures that will resonate with the public and show tangible outcomes in terms of improved health.

Wells R, Whitworth JA. (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1929109/?tool=pubmed)
2007. Assessing outcomes of health and medical research: do we measure what counts or count what we can measure? Aust New Zealand Health Policy 4:14.

In a testimonial posted on the Becker website, Gordon wrote, “Without the thorough analysis recommended by the Becker Model for Assessment of Research Impact, we would have never realized just how far-reaching the impact of our research has been.”
(Photo courtesy of WUSL)

As Pettibone, standing, moderated the question and answer session, NIEHS Program Analyst Liam O’Fallon, second from right, asked about measures that might apply to environmental justice and community-based participatory research programs that are part of PEPH.
(Photo courtesy of Michael Gaske)

The Environmental Factor is produced monthly by the National Institute of Environmental Health Sciences (NIEHS) (http://www.niehs.nih.gov/), Office of Communications and Public Liaison. The text is not copyrighted, and it can be reprinted without permission. If you use parts of Environmental Factor in your publication, we ask that you provide us with a copy for our records. We welcome your comments and suggestions. (bruskec@niehs.nih.gov)