A Troubled Trifecta: Peer Review, Academia & Tenure

We welcome Peter Frishauf as an author on our blog. Peter is on the Editorial Board [brief bio] of our Society’s Journal of Participatory Medicine, and as described below, has already authored some important material on this subject. His first post here is triggered by an article in Tuesday’s New York Times that generated much discussion on a vital subject: how we reality-check what we think we know, as a basis for science, and especially for medical advice. – e-Patient Dave

It’s a recitation of the obvious: without reliable, high-quality peer-review, medical information and patient care suffer.

Sadly, academia and broken peer review are also linked to our tenure system. We need to fix that.

I hate to condemn with a broad brush: many academics care deeply about better patient care and believe – genuinely – that tenure is a good thing. As Wikipedia describes it today, tenure “protects teachers and researchers when they dissent from prevailing opinion, openly disagree with authorities of any sort, or spend time on unfashionable topics.” But sadly, the vast majority of academics acknowledge that an even more powerful benefit of tenure is pretty damn good job security. Tenure has much more to do with money (lifetime job security) than academic freedom.*

Tenure and traditional peer review are joined in a pernicious union that should be smashed if universities are to be trusted to promote the public interest through good, published science and medicine. In the “publish or perish” environment of academic medicine, candidates for tenure are generally ranked not on how much information they create (by research or invention) or how well they teach (which counts for just about nothing), but by the prestige of the journal where they publish. It’s a system that might have made sense when a journal’s reputation was regarded as a proxy for an author’s scientific rigor and integrity. No more.

With the Internet, information is distributed efficiently and effortlessly. Reputations are created – and destroyed – quickly. Crowdsourcing experts review articles to improve their quality. Articles themselves aren’t published and forgotten, but continually curated over time. The old publishing and peer review model is obsolete in this environment.

In an excellent article this week in The New York Times, journalist Patricia Cohen describes a number of experiments to improve peer review:

Katherine Rowe, a Renaissance specialist and media historian at Bryn Mawr College, worked with the prestigious 60-year-old Shakespeare Quarterly to crowdsource/review articles by a panel of invited experts. The reviewers “were invited to post their signed comments on the Web site MediaCommons, a scholarly digital network.” Others could add their thoughts as well, after registering with their own names. “In the end 41 people made more than 350 comments, many of which elicited responses from the authors. The revised essays were then reviewed by the quarterly’s editors, who made the final decision to include them in the printed journal, due out Sept. 17.”

“Today a small vanguard of digitally adept scholars is rethinking how knowledge is understood and judged by inviting online readers to comment on books in progress, compiling journals from blog posts and sometimes successfully petitioning their universities to grant promotions and tenure on the basis of non-peer-reviewed projects.”

“Clubby exclusiveness, sloppy editing and fraud have all marred peer review on occasion. Anonymity can help prevent personal bias, but it can also make reviewers less accountable; exclusiveness can help ensure quality control but can also narrow the range of feedback and participants. Open review more closely resembles Wikipedia behind the scenes, where anyone with an interest can post a comment. This open-door policy has made Wikipedia, on balance, a crucial reference resource.”

Journalist Cohen also nails the tragic and under-reported association between traditional peer review and tenure:

“The most daunting obstacle to opening up the process is that peer-review publishing is the path to a job and tenure, and no would-be professor wants to be the academic canary in the coal mine. The first question that Alan Galey, a junior faculty member at the University of Toronto, asked when deciding to participate in The Shakespeare Quarterly’s experiment was whether his essay would ultimately count toward tenure. “I went straight to the dean with it,” Mr. Galey said. (It would.)

Well, good for the University of Toronto: Mr. Galey can get “tenure credit” and participate in an experiment to create high quality information. What a concept for a university!

Medicine isn’t a total laggard when innovating in this space. PLOS, several journals within the BioMedCentral group, and others, are experimenting with open, pre and post publication peer review. A growing group of medical publications – but still a tiny percentage – embrace open access, which makes articles freely available. U.S. federal law that mandates open access for research published with public grant money is speeding the trend exponentially.

Still, when you consider what’s at stake – healthcare, and at times a life-saving treatment – we should hang our heads in shame when one considers how slow medical publishers have been to improve peer review and access. And shouldn’t academia be leading the charge for better information rather than obstructing it through a system as self-serving and public-interest-damaging as tenure?

In more naïve times, most of us believed that the system of peer review used by trusted, traditional publications like The New England Journal of Medicine was our best bet for trusted sources of current medical information. Now we know better: numerous experts have provided extensive evidence that traditional peer review is unreliable. And using tenure to prop up the bad system is a black eye on our universities.

I have proposed3 that online Reputation Systems could be created to help assess the reliability of crowdsourced information, medical evidence, and pre and post-publication peer review. It’s just one idea, my horse in the race. Regardless of what we wind up with, here’s hoping that everyone who wants to improve health and patient care will innovate to fix our broken peer review system. And while we’re at it, let’s fix tenure, too!

* When you think about it, academic freedom and tenure shouldn’t even be linked: there are other ways (transparency, public exposure, good journalism by people who write about universities, even political pressure) that influence academic freedom more powerfully than tenure. Return to reference

Peter Frishauf is founder of SCP Communications, Inc. and Medscape, past President of the Healthcare Marketing and Communications Council, Inc., and former Director of the Association of Medical Publishers. He is a member of the editorial board of the Journal of Participatory Medicine, the non-executive Chairman of the Board of Directors of Crossix Solutions, Inc., and a director of MedPage Today, a site that covers breaking medical news peer-reviewed by the University of Pennsylvania. Peter is also an advisor to the Markle Foundation’s Connecting for Health initiative.

Comments

Let’s not forget, though, that not all peer-reviewed scientific studies are connected to tenure. Many are the products of the pharmaceutical industry. Following the case of Scott Reuben MD, whose hand was so vehemently slapped for a career of peer-review fraud, does anyone still trust the peer-reviews of pharmaceutical studies? Gilles Frydman said Reuben put “the last two handful of nails into the coffin of the scientific peer-reviewed process.”

So, on the one hand we have young scientists trying to fulfill their tenure requirements. On the other hand, we have all that money out there waiting to reward pro-pharma testers and reviewers. Crowd-sourcing offers a possible answer to the tenure-requirements problem, but what about the other? What about the guys who cheat on their clinical trials (or like Reuben, just do the whole-cloth thang)? Can anyone doubt that there are kickbacks going to some of the reviewers to ensure that those “results” get published?

I hate to sound like a conspiracy theorist, but I’m beginning to have genuine doubts about so-called evidence-based medicine. There’s no there there.

BillDog, with respect to bias from pharma-supported research: both good and bad science comes from academia and industry, and I know of no evidence that shows one is worse than the other. But a good peer review system would catch bad science whatever its source. For me, a more relevant question about how studies are conducted is that the inclusion and exclusion criteria in drug studies (mandated by the FDA, not industry) may not take into account how drugs work in the real world, where people often have constellations of diseases (ex: hypertension, diabetes, and respiratory disease). For a discussion of this, listen to the podcast athttp://www.jopm.org/multimedia/podcasts/2010/08/09/participatory-evidence-opportunities-and-threats/.

Excellent, thoughtful post. I think the operative quote in the NYT article is:

“What we’re experiencing now is the most important transformation in our reading and writing tools since the invention of movable type,” said Katherine Rowe

As applied to medicine, peer review has heretofore been restricted to a ‘good old boy’ network of the elites in academic medicine – who, predictably, have their own agendas and biases. Dissenters were confined to letters to the editor after the fact. Now, an entire article could be published, debated, and even perhaps revised in real time, for all to read. Could this reduce the lethal lag time which has plagued medical advances for so long?

Dave, I think the biased pharma-funded (and -designed, and even -written) research would be more exposed to sunshine with this new development – so I see this as cause for optimism, not pessimism.

She writes about how the peer review process slowed down publication of the findings and how members of the National Academy of Sciences are able to choose their own reviewers.

What stuck with me, however, is that some CFS patients knew about this blockbuster finding months ago, started on the new treatment, and found it to be beneficial. They didn’t wait for the journal article to be published.

How did they find out? How did they spread the news? If patients found out first, how did they present it to their doctors? If doctors found out first, how did they present it to their patients?

How do other CFS patients feel now, if they are just hearing about this potentially life-changing news? How long will it take for this treatment to spread to the wider population of CFS patients?

Alternatively, what are the downsides of rapid publication and dissemination?

Threaded throughout it all is the internet’s role, patient networks’ role, health professional networks’ role. This is a deep and significant effect of the internet and social networks on health and health care. It is not just CFS, but many condition groups who are engaging in this work to uncover answers faster and disseminate them more widely.

Harnad, S. (1997) Learned Inquiry and the Net: The Role of Peer Review, Peer Commentary and Copyright. Learned Publishing 11(4) 283-292. Short version appeared in 1997 in Antiquity 71: 1042-1048. Excerpts also appeared in the University of Toronto Bulletin: 51(6) P. 12. cogprints.org/1694/

[…] system to evaluate the quality of their content. My earlier contribution to this blog, “A Troubled Trifecta: Peer Review, Academia & Tenure,” discussed this in more detail, as well as a number of articles and podcasts I participated in […]

Leave a Reply

Welcome

Participatory Medicine is a model of cooperative health care that seeks to achieve active involvement by patients, professionals, caregivers, and others across the continuum of care on all issues related to an individual's health. Participatory medicine is an ethical approach to care that also holds promise to improve outcomes, reduce medical errors, increase patient satisfaction and improve the cost of care.