Science, policy, and politics. Focus on science communication and climate change. The Dake Page offers news, analysis and book reviews.

Thursday, February 19, 2015

How peer-review works…and doesn’t work (Part 3: Abusing the system)

This is a continuation of the series on how peer-review works…and doesn’t work. Part 1 looked at the basics of how the peer-review process works for
scientific papers - what it does, and what it doesn't do. You can read the entire Part 1 article here. Part 2 looked at what happens when peer-review goes wrong. You can read the entire Part 2 article here. Now we’ll take a look at some cases
where the peer-review system has been abused.

Before
starting Part 3, however, it must be stressed that any inadequacies so far discussed are exceptions to the
rule. Peer-review almost always does what it is supposed to do – a
first screen to make sure papers represent legitimate research and are
fully documented so that they can be assessed by the larger scientific
community. It’s rare that peer-review “fails” (see Part 2). It’s even
rarer that papers are retracted once they are published. A study
published in 2012 examined the 2047 retractions of papers indexed
in the PubMed database (mostly biotechnology papers). That sounds like a
lot until you realize that this was out of over 21 million published
papers in that database, meaning less than 0.01% of published papers
were retracted. Retraction is rare even though the bar for retracting papers has been lowered (i.e., it's much easier and faster to retract now than previously).

That
said, let’s look at the cases where papers have been published that
probably shouldn’t have been. The new problem of “open access journals,”
i.e., those journals who publish for a fee, was mentioned in Part 2.
The biggest concern here is that some of these “journals” are simply
predatory publishers that will post online anything that is sent to them
as long as the fee is paid. These predatory journals will likely disappear as people refuse to be
associate with them, especially since they obviously aren’t really peer-reviewed.
So while they may be a big headache right now, likely they will weed out
the bad eggs through, not ironically, peer pressure.

Which
gets us to the real problem. The following examples highlight some of
what can happen when unscrupulous people try to take advantage of the
system.

The most famous example of “pal review” as discussed in Part 2
is the publication of a climate related paper by Soon and Baliunas in the journal Climate Research
in 2003. The paper was shuttled through the review process by fellow climate denier Chris de
Freitas, an editor for the journal. Once published, the paper was
roundly criticized by the scientific community as unsupportable on its
face. Further review revealed that Soon and Baliunas
were funded by the fossil fuel industry, that the conclusions stated were
inconsistent with their own data (which were inconsistent with
reality), and that de Frietas
had a history of pushing through papers by climate deniers despite
their obvious failings. Details of the controversy can be read here.
Since then, Soon and a small group of lobbyist-associated authors have
been implicated in a series of questionable papers that misrepresent the
science. Often these papers are published in a journal called Energy and Environment, a non-science pal-review type of journal where the editor has acknowledged papers are published based on political motives.

Following publication of the Soon and Baliunas
paper described above, and also in one or two other cases where
apparently fraudulent papers were published in peer-reviewed journals, senior
editors chose
to resign. While reputations of any scientists involved can be severely
damaged, for some this doesn’t appear to matter much as long as the lobbyist funding continues (e.g., Soon was
recently accused of violating basic ethics conventions by failing to
disclose his fossil fuel industry funding in a paper he co-authored with the usual band of climate deniers).

There
isn’t much that can be done about such papers other than to keep
strengthening peer-review standards, a difficult proposition given the
thousands of journals that now compete for papers to publish. Sometimes
the papers are retracted, but as noted above, retractions are rare, though increasing.

This
latter point can actually work against legitimate scientists. In the
past, scientific papers were scrutinized and critiqued by other
scientists, and that feedback helped move the science along. Now the
papers are more accessible to the general public through blogs, the public is more likely to get a "spun" version of the paper than the actual science. While press releases by the scientific organizations may be poorly worded, the real problem is when bloggers, either intentionally or unintentionally, get the gist of the paper's findings wrong. So the public may be misinformed. Worse, the
papers are read by political and lobbying interests, which would be
okay if they honestly evaluated the science. But that isn’t the case.
Most political operatives
and lobbyists have a particular policy view and are not hesitant to
misrepresent the science if they feel doing so will help them achieve
their preferred policy action – which in most cases is no action at all.
These operatives and lobbyists can exert tremendous pressure on
journals that, at least in one recent case, can lead to legitimate, scientifically robust, papers being retracted solely because the journal feared an expensive legal battle with lobbyists. This sets a dangerous precedent.

In addition, there
are many cases of politicians saying things about science that are not
scientific. Senator James Inhofe is notorious for arguing that the science of man-made climate
change is all a hoax, originally basing this politically convenient opinion on the 2003 Soon and Baliunas paper, which many suggest was the main motivation for the paper being funded by the petroleum industry. Not surprisingly, Inhofe’s home state of Oklahoma is highly dependent on the oil and gas industry and that industry routinely lavishes upon him significant campaign funding. This
is true of other politicians as well. And yes, health and environmental
advocacy groups also financially support their preferred politicians
and feed them information that supports their advocacy. The main difference
is that health and environmental lobbyists generally pressure
politicians to listen to the scientists while fossil fuel lobbyists
generally pressure politicians to listen to, well, the fossil fuel
lobbyists and their small cadre of associated scientists who disagree
with the vast overriding consensus of the science.

But that’s a topic for another post.

To
recap, the last three weeks have taken a look at the peer-review
process – what it is, and what it isn’t. We’ve looked at some ways that
peer-review can “fail,” and some ways that people have abused the
process. Due to the space limitations of a blog format, these
discussions are necessarily incomplete. The
links provide more detail on some of the points being made, but there
are many others that could also be discussed in greater depth. The main
points to understand are that peer-review is merely the first step in
the scientific evaluation process, and only after publication can the
greater scientific community scrutinize the studies being presented.
Sometimes bad papers get published, but most of the time they are
inconsequential. Attempts at fraud do happen, and while relatively rare, can have
significant impacts (e.g., see Andrew Wakefield).

Overall,
peer-review works, and is necessary. There are challenges for the
future because of predatory practices related to the “open access”
nature of the worldwide web, but these are likely to be worked out so
that some combination of public access and quality assurance can be
achieved.

In Part 4 we'll take a look at how some papers that might have been
inconsequential in the past can now be artificially elevated into a level of importance they don't merit. We'll explore the role of the internet in making this
happen, both for good and for evil.

3 comments:

There isn’t much that can be done about such papers other than to keep strengthening peer-review standards, a difficult proposition given the thousands of journals that now compete for papers to publish. Sometimes the papers are retracted, but as noted above, retractions are rare, though increasing. This latter point can actually work against legitimate scientists.

Being a "legitimate" scientist, I would argue that we should not make peer review more strict. It is just a filter, some bad stuff gets through, but on average the quality is better afterwards. If we make the review more strict, we risk that good and especially very unconventional ideas will find it hard to get published. And like you write yourself, it is impossible to make peer review so strict that only "right" papers get published, whatever that means. The papers should just be worthwhile to read.

For science it would thus be best to keep the standards. What would not hurt would be to apply the standards more consistently. That is hard to do because the reviews are written by volunteers, but that would be worthwhile.

That has as consequence that we have to tell society and journalists that they should never rely on a single new paper. It only becomes science after being digested by the scientific community, that takes time. Great to see how this blog series contributes to that.

You make excellent points and I didn't mean to suggest that legitimate papers that have unconventional ideas be restricted. My suggestion that peer-review standards be strengthened is more focused on keeping the process free from abuse such as that described, and ensuring that some sort of peer-review is included in the new open-access journals.

This last point is probably the greater issue. Simply getting gibberish published because you paid the fee gets the notice, but there could be many "propaganda" papers also being published this way. Part 4 takes a look at papers that would usually be ignored because they lack robustness now, in the age of universal access of "information" (whether real or not) on the internet, inconsequential papers can take on undue importance. Especially when lobbyists (pro- or anti-industry) promote them.

Your last paragraph captures exactly a point that scientists need to keep emphasizing to journalists - one paper does not define a body of science, and most papers focus on one small aspect, a piece of a very large puzzle that only makes sense when you look at the puzzle as a whole. Journalists (and to an even greater extent, bloggers) routinely take some tiny paper and elevate to seminal without merit.

Traveler IQ

This Traveler IQ was calculated on Saturday, February 07, 2009 at 05:18PM GMT by comparing this person's geographical knowledge against the Web's Original Travel diary's 3,649,959 travelers who've taken the challenge.