29 March 2015

When I used to work in an office, my boss used
to say that any time he had a good idea, he could come to me and in ten
minutes he'd know everything that might go wrong with it. He often went
ahead anyway, and his ideas often worked, but at least he was
forewarned. So in that spirit, here goes.

I'm a little concerned by all the hype around Open Access (OA) journals.

Yes,
I know that traditional journal publishers are evil, and make more
money and higher gross margins and have bigger car parking spaces than Apple, and
I agree that when taxpayers fund research then taxpayers should
have access to it. However, I'm not sure that just because all of the
above may be true, that our current model of OA journals is necessarily the solution. I have a number of concerns of what may happen as the OA model takes hold and, as everybody tells me is going to happen, becomes dominant. This post is intended to start a discussion on those concerns, if anyone's interested.

1. It's the economy, stupid

One of
the strengths
of the traditional publishing model is that, to a first approximation
and allowing for all kinds of special circumstances, the editor-in-chief
of a half-decent journal doesn't have to worry about filling it.
Indeed, many journals proudly promote their rejection rate on their home
page, next to their average turnaround time. "We reject 90% of
submissions; don't waste our time unless you've got a good story to
tell", is the message (with, of course, all of the predictable effects
on publication bias that this implies). Doubtless the editor-in-chief
has some financial targets to meet, perhaps in terms of not blowing the
production budget on full-page colour pictures of kittens, but this is
not a job whose holder is principally tasked with revenue generation.
The money is coming in
pretty steadily from sales of packages of journals to institutions
around the world (even if some of these institutions are starting to
take exception). The editor gets to concentrate on, among other things, maintaining the journal's impact factor --- hopefully using methods that are a little less blatant than this.

With
OA journals, funded principally by
article processing charges paid by authors, things are likely to be a
little different. No matter how dedicated to academic integrity and the
highest possible scientific standards the editorial staff want to be,
money is right there in the equation every day, especially for an online journal
with almost no physical limits to its size. How many articles can we
get through the
review process this month? Can we upsell the author to the full-colour
package? Why do so many people want hardship waivers? (Oh, and I have yet to see any suggestion that OA journals will be less concerned about their impact factor than traditional journals.)

The
idea, of course, is that authors will not be reaching into their own
pockets to pay the article processing charges. The intention is that
the fee of $1,000 or so to publish the results should be budgeted for out of the project's funding. After all, it's only a waffer-thin thousand bucks, the kind of money some projects
probably have slopping around at the end anyway if the participants didn't eat all of the M&Ms. But once funding
agencies catch on, will they allow grant proposals to include specific line items for
OA publishing, when publication in one of the high-prestige traditional
journals --- which you promised them, earlier in the proposal, were
definitely going to be interested in your groundbreaking project --- is
free? And what about the independent researcher with no budget, who may
have something interesting to say, but no money? Should such a person
have to fund publication from their own pocket?

I'm afraid that the money always, always
finds a way to affect things. Someone, somewhere in the process, will be directly incentivised to
increase revenue. (In France, where I live, gambling is a state
monopoly, which means that whatever arms-length construction they have
put together, somewhere there is someone who essentially works for the
government and yet has a performance target to sell more
scratchcards to the urban poor, even though gambling is officially a social problem.) How
does this affect you as the editor-in-chief of an OA journal?
Maybe you ask your action editors to tell reviewers to be less picky
about certain things.
Maybe you suggest to an author that splitting these results into two
articles will be to everyone's advantage - after all, the publication
fee is coming out of the grant money, and as it stands it is a pretty
long paper manuscript for someone to have to wade through at one
sitting...

The corollary of this is that the PI presenting an article for publication is a paying customer. Now when I go to make a $1,000 purchase, I'm generally greeted with open arms. I certainly don't expect to have to pass quality control checks before I'm allowed to spend my $1,000. The psychology of the OA model is going to be interesting indeed. (Compare what happened in the UK when public universities started to charge tuition fees; all of a sudden, the idea of a student being given a failing grade became, for many people, a consumer protection issue. "I paid to come here and get a degree, how dare you tell me I can't have one?", ran the argument. Too many unhappy punters, and the Vice-Chancellor is touring the stricter departments to ask them to be a little more, um, flexible in their marking criteria.)

I found a pertinent example shortly before putting this post (which has taken a while to draft) online. Here
is a note from Nandita Quaderi, who is "Publishing Director, Open
Research" at Scientific Reports, which is part of Nature Publishing
Group. Nandita is pleased to announce that henceforth, "a selection of
authors submitting a biology manuscript to Scientific Reports
will be able to opt-in to a fast-track peer-review service". Needless
to say, this service comes "at an additional cost", being provided by a
for-profit organisation called Research Square. (An editor of Scientific Reports has resigned over this.) So now, I'm paying to publish, and I'm paying to have my article
reviewed. What could possibly go wrong with the objectivity and rigour
of the scientific process?

2. Access is not the biggest problem science faces right now

Another issue is that most
OA journals do
not address the ongoing problems of the peer review system. I would
argue that
currently, failures of peer review are a bigger threat to science than
paywalls.
If reviewers are allowing bad science through --- or erroneously
recommending rejection of good articles --- then getting free access to
the
resulting error-filled literature is the least of our problems; and I
have yet to see a coherent argument why the OA review process might be
inherently any more rigorous than that at traditional journals.

Some
online journals, such as The Winnower, have adopted a radical solution
to this: anyone can publish an article, without any prior review process, with the idea that people will come along and
review it afterwards. This seems attractive at first sight, except that people
typically have even less incentive to act as a reviewer once the article
is "out there", even if it doesn't yet have the status of a citable
article with a DOI (a status which, incidentally, the article's own
author decides to award it, at a time of his or her own choosing).

It
seems to me that OA journals are to some extent hitching a ride on the
back of the traditional journals, which have created (and still sustain)
the fundamental mode of operation that we know and love/hate: author sends in MS,
editor checks it, editor selects reviewers, reviewers approve or request
changes, editor finally accepts or rejects. This system more or
less works --- give or take the criticisms of peer review as "broken",
which have a lot of merit but which, as I noted above, it seems to me that OA (in and of
itself) doesn't do much to address --- because people generally
have confidence in it. Not necessarily absolute confidence, but we know how it's meant to work and how to spot when it isn't working. We (like to) believe that the editors do not
generally accept (too many) articles from themselves and their buddies (or at least, that they risk getting called out for it if they do), that they
select reviewers who are competent in the relevant subfields, that the
reviewers do an honest and unbiased job, etc. (Of course, the reviewer who is doing
"excellent quality control" with *your* article is an
incompetent idiot who has failed to understand even the most basic concepts
of *my* article, but that's part of the game.)

So, when something like Collabra,
the new OA mega-journal from the University of California, launches,
they can put pictures of respected people on the front page where they introduce their editorial board, thus
sending a message that the
review process will be every bit as rigorous as it is for a traditional
journal. Readers are reassured, and authors know they will need to
submit work of a high standard. But to me this only works because the
majority of people who are being held up as examples of the quality of
the journal have good reputations, which have been made within the
traditional process. How does this scale? What does the publication
process look like in 10 or 20 years time, if the traditional journals
have mostly gone and we make our reputations with OA (web-)publishing,
blogs, and social media presence? (Yes, impact factor is broken. But where is the dominant, credible alternative that everyone will be prepared to switch to?)

This doesn't mean that Collabra will be full of articles promoting
homeopathy after a few months. But over time, the relationship between authors, reviewers, and journals will change, in ways that we can't necessarily predict. That doesn't mean the sky will fall, but it does mean that there will be perverse situations that may or may not be worse than what we have to put up with now.

3. Ham, spam, and all points in between

I also worry that the line between "legitimate" and
"spam"
OA journals will start to blur. Currently we can all point and laugh at
the semi-literate invitations to publish in (or join the Editorial
Board of) those pseudo-journals with
plausible-sounding names, strange salutation styles in their e-mails,
and an editorial address in a Regus suite in
San Antonio, from which manuscripts are presumably forwarded to the
journal's real staff in Cairo
or
Mumbai. But these fraudulent (whatever that means...) journals will
improve, and it will become hard to tell the
"fake" from the "real".
A few weeks ago, I was asked to review an
article by an OA journal that was part of a London-based publishing
outfit. I genuinely couldn't decide
if they were spammers or genuine: the journals mentioned on their web
site all seem to exist, and about a third of them are indexed in
PubMed. How good or bad is that? I recommended rejection, as the
article
would have been of little interest to the
readers of
the journal, according to its own profile. I wonder what the lead author did next (assuming that my recommendation to reject was the editor's verdict as well)? Did he appeal, as a "paying customer", to the editor in chief? Or did he maybe send the article to another OA journal, on the basis that he will eventually find somebody, somewhere, who wants $1,000? (*)

I think,
though, that perhaps the bigger risk in the meeting of "legitimate" and
"spam" journals is through the trimming of standards at the "legitimate"
end. Look at what happened when the Saudis decided
to throw some money at education, and suddenly King Abdulaziz University is ranked #7 in the world in mathematics.
Uh-huh. Sure. So what happens when that university, or others with rather more money to
burn than academic integrity, starts
its own OA mega-journal? Exactly what will be the conditions of
scientific neutrality under which the editor-in-chief reviews articles
by, say, the children of minor members of the Saudi ruling family?
Perhaps someone will create an authoritative
clearing house to administer a sliding scale of which
journals are "real" versus "spam". But who would run such an
organisation? The AAAS? ISO? Standard & Poor's? Google?
And who would ultimately be responsible for the "legit"/"spam" decisions?

Historically, publisher-led journals seem
to have been mostly
spam-free; it would be interesting to establish why this was.
High barrier to entry in the world of ink and paper? Old-fashioned
academic and intellectual integrity, despite the profits? Risk of
reputational
damage if, say, Springer (cough) or Sage (cough) were to acquire a reputation for publishing garbage? I
don't know what the reasons are, but it created the current situation
whereby --- whatever the other problems in the system --- a journal that exists in a print edition is generally regarded, at
least by default, as having some degree of seriousness. I worry that
we will end up in a situation where we don't have a simple way to tell
whether we can take a "journal" (in the widest possible sense) seriously
or not. In such situations, humans tend to apply some simple
heuristics, which scammers have many centuries worth of experience exploiting.

4. A modest (and, as yet, barely sketched out) proposal

Do I
have an alternative? Well, when my boss came to me with his ideas, I
usually didn't, but in this case I do have a tentative suggestion. What
if the funding agencies ran a few journals? After all, these are (generally) the representatives of the
taxpayers, who --- as the Open Access movement is right to point out ---
pay for the research and ought to have free access to the results. Yet currently, they rely on "the system" to work, and for researchers to muddle their way through that system. In the traditional model, the readers pay, and in theOA model, the authors pay. Both systems have their deficiencies. Supposing we had a parallel model where nobody paid (except a general fund, set up to guarantee neutrality)?

Those of a libertarian bent might argue that the government shouldn't be
involved in academic publishing, but the stable door closed on that
when we started to take their money to do the research. Some might also argue that an funding agency-sponsored journal might be highly politicised, but then, /a/ why should it be more politicised than the handing out of the money, /b/ the Rind/Lilienfeld saga showed that politicians can pressure "independent" journal publishers into submission too, and /c/ there will always be other outlets; I'm just modestly proposing a "third way". (As a bonus, this would
seem to be a good fit with the aims of the pre-registration movement.)

Notes:

1. I'm aware that this is a rather long and at times rambling post. It started life in a frenzied evening of writing just
after I got out of hospital after a stay that lasted the better part of
three weeks, and that still shows. I should probably have scrapped it and started
again, or at least sat down and rearranged the paragraphs, but I wanted
to get the ideas out there within a reasonable time frame. I hope some of them are useful.

2. I want to thank Rolf Zwaan for some helpful discussions on an earlier draft of this post.
Rolf disagreed with much of what I had written, and I've only made a
few changes, so he probably still disagrees with a lot of it. I should point out that my use of the example of Collabra (for whom Rolf is an editor) above is not based on any specific criticism of that journal, but merely as a salient example; Rolf's tweet about his appointment as an editor at Collabra was the spark for my writing of this post.

(*) Update 2016-11-28: I was re-reading this post because reasons, and I noticed this dangling question. I googled the title of the article... sure enough, it was accepted, despite my recommendation to reject.