The Scientific Publishing Sting: a Missed Opportunity?

Last week I received an email asking me if I wanted to apply to be an editor for a chemistry journal. This might seem a bit odd because I'm not a chemist (although I did do a couple of courses in biochemistry at university, which seems to be enough in the eyes of at least one Nobel committee), but nowadays this sort of email is a regular part of the scientist's inbox: the email from an obscure publisher asking us to join one of their editorial boards or submit papers (or even books) for publication.

If you dig a bit deeper, you usually find that the publishers' business model is to charge the authors a fee for publication, and then make the paper freely available on the web. This "open access" model is also followed by several perfectly respectable publishers, but it is easy to see how you could scam the system. Scientists want their papers published, so your journal accepts everything (or almost everything), sit back and rake in the money. It's become clear over the last few years that there are a lot of publishers trying this -- apparently successfully -- so John Bohannon, a journalist at the journal Science, decided to assess the scale of the problem.

Briefly, Bohannon wrote a fake (awful) scientific paper and submitted it to a lot of journals that have an author-pays, open access model. He chose these journals based on a couple of lists of open access publishers -- the Directory of Open Access Journals (DOAJ), which "aims to be comprehensive and cover all open access scientific and scholarly journals that use a quality control system [i.e. peer review] to guarantee the content", and Jeffrey Beall's list of predatory publishers, which lists all of the publishers that Beall has investigated and concluded are dodgy. This is what Bohannon found:

Of the 255 papers that underwent the entire editing process to acceptance or rejection, about 60% of the final decisions occurred with no sign of peer review. For rejections, that's good news: It means that the journal's quality control was high enough that the editor examined the paper and declined it rather than send it out for review. But for acceptances, it likely means that the paper was rubber-stamped without being read by anyone.

Of the 106 journals that discernibly performed any review, 70% ultimately accepted the paper. Most reviews focused exclusively on the paper's layout, formatting, and language. This sting did not waste the time of many legitimate peer reviewers. Only 36 of the 304 submissions generated review comments recognizing any of the paper's scientific problems. And 16 of those papers were accepted by the editors despite the damning reviews.

In other words, a lot of the journals either accepted the fake (awful) paper without meaningful peer review, or ignored the bad reviews it received.

The reaction from the open access crowd was rapid, and mainly consisted of trying to shoot the messenger rather than dealing with the message. The usual open access attack dogs criticised the news story, calling it an embarrassment, and suggesting Science's "credibility is shot to pieces" (in this example, mistaking a news piece for a scientific paper).

They criticised Science because they had published a bad paper three years ago (ahem), and they said the piece was flawed because it hadn't included a control group. Because everyone knows that if you want to run a news story about how many car drivers speed on the motorway, your story is invalidated unless you include stats for cyclists too. Or something like that. (Yes, there are reasons why a comparison with other journals would be interesting, but still.)

Another complaint was that the study's conclusions weren't an indictment of open access, but of peer review, presumably by people who didn't read as far as the final paragraph quoted above. So although there were some more measured responses from the open access community (e.g. from the DOAJ, whose list of journals was used in the sting), these were largely drowned out by all the noise.

It's obvious that a lot of these responses were from people who were trying to protect their interests -- i.e. open access -- and to push their agenda: "peer review is broken and anyone who doesn't embrace open access should be damned for all eternity to peer review philosophy papers." Or something like that.

But I think this reaction is misguided. The problem Bohannon highlighted in his news story is a problem for open access, and their "shoot the messenger" tactic is surely the wrong reaction. But I think the problem Bohannon highlighted in his news story can be successfully resisted if the open access fraternity takes it seriously.

Yes, you've guessed it, I am about to push my own agenda.

Why is this a problem for open access?

The model that predatory publishers is following relies on an author-pays model for generating income, and this works best with open access. Author-pays is necessary because the publisher needs to get their money from somewhere, and it is difficult to get from online advertising (as this newspaper knows), or from selling subscriptions (as some other newspapers have been finding out). You have to make sure your product is interesting if you want to generate advertising and subscription revenue, and this takes hard work.

But for scientists there is pressure to publish, both from a scientific point of view (if you don't communicate your research, nobody will know about it), and from a professional viewpoint (publications are a major way of judging performance). Given that pressure, and that scientists generally have money for research, a journal can extract funds from them in exchange for publishing their papers.

As I mentioned above, this scenario is not necessarily bad, but obviously it can be abused. Simply stated, a journal can accept (almost) everything, have a fig-leaf of peer review that asks for a few minor revisions, and then ask for the money. Once the journal has gotten that far, they might as well make the papers open access: few people will pay for a subscription (except perhaps the authors, but the journal has already separated them from their money), and putting up paywalls and collecting money is more effort than it's worth. Plus you can use open access as a selling point: after all, science wants to be free!

It is clear from the number of predatory open access publishers that this model works well to generate profit. The problem for the open access fraternity, who genuinely want to encourage publication of good science, is that bona fide open access publishers, like PLOS, BMC, and Hindawi (all three of which rejected Bohannon's paper), also use an author-pays model, and thus, they also need to accept a large number of scientific papers.

If you are not attentive, it is easy to link genuine publishers and predators. Clearly this link needs to be broken, and indeed there are large differences between predatory publishers and reputable open access publishers: good open access publishers do have a peer review system, and they do reject papers on the basis of these reviews.

This is good for the journals in the longer term: they exercise quality control because they need to maintain a reputation as reputable publishers (I meandered around these issues in a blog post last year). Using Bohannon's sting to emphasise these differences might have been a better strategy than attacking his piece: use it to get the message out that there are many good open access publishers.

What about non-open access publishers?

The sting was aimed only at author-pays open access publishers, and I think one should be careful about extrapolating beyond the obvious point that there are a lot of predatory open access publishers. In particular, it says nothing about non-open access publishers.

What would have happened if they had been targeted? This is the basis of the criticism that there wasn't a control group. In one sense this criticism is a bit silly because the differences that are interesting are not treatments that are applied: Bohannon didn't randomly make one set of journals author-pays and the others not. On the other hand, this criticism does raise an important wider question. Is this bad behaviour limited to predatory open access publishers, or does it also affect other publication models? Bohannon had originally intended to test this, but didn't have the resources.

My guess (admittedly without any evidence) is that fewer journals that don't charge authors for publication would have accepted the paper. My reasoning is that there is less incentive to publish rubbish. Journals that charge for access are competing with other journals for librarians' limited attention and funds. Journals that are more likely to publish papers that researchers want to read are more likely to attract subscriptions. But if your journal publishes rubbish, nobody's going to want to read it. So there is a stronger incentive for quality control. This does not mean that no journal would ever accept either a fake or an awful paper -- after all, one recently accepted a spoof paper that even cited Borat.

One issue at play here is the variation in quality found across all types of journals, and Bohannon only tested the bottom end of one publishing model. He had good reason for doing so: predatory open access has been expanding rapidly over the last few years, and his starting point was a researcher being unwittingly caught by one of these journals. These journals have a deservedly poor reputation, and it is good that this reputation sticks with them. But there are also open access publishers that deserve a good reputation, and they have to work hard to keep it and to distinguish themselves from their predatory competitors. To do this requires not just maintaining their own reputation, but also that of the open access movement.