Slashdot videos: Now with more Slashdot!

View

Discuss

Share

We've improved Slashdot's video section; now you can view our video interviews, product close-ups and site visits with all the usual Slashdot options to comment, share, etc. No more walled garden! It's a work in progress -- we hope you'll check it out (Learn more about the recent updates).

Frequent contributor Bennett Haselton writes:
"A Harvard biologist was able to get an intentionally flawed
paper accepted for publication by a number of open-access
academic journals, included that had supposedly been vetted for quality by advocates of
open access. It seems the problem could be mitigated by consolidating journals
within a field, so that there are much fewer of them, publishing much more articles
per journal -- so the review processes take the same amount of labor, but you have fewer
journals that have to be audited for procedural honesty." Read on for the rest, including his idea to solve the problem of fraudulent submissions (or even just sub-par science) through simplification.

Harvard biologist John Bohannon wrote about his
experiment in an
article published
by Science Magazine. He submitted his deliberately bogus paper to 304 open-access
publishers, including 183 that were listed in the Directory
of Open Access Journals (DOAJ), which Bohannon calls the "Who's Who of credible open-access journals",
and whose quality is supposedly vetted by the DOAJ staff.

Of the 304 open-access journals targeted by the sting, 60% published the paper.
I think this mainly just shows that the average quality of open-access journals may always be low,
but that's not surprising since anyone in the world can set up an "open access journal".
That shouldn't be relevant to the reputation of the best open-access journals. If the best
open-access journals acquire a reputation for high standards and proper peer review, then that
will attract high-quality papers, whose publication will reinforce the reputation of the journal,
which enables it to confer prestige on the papers it publishes, which in turn will continue to
attract high-quality papers. The existence of other open-access journals with crummy standards,
should be irrelevant.

What's more disturbing, is that of the 183 journals listed in the Directory of Open Access Journals,
45% of those published the paper -- which, according to Bohannon's article, surprised and
disappointed the DOAJ founders.
But
perhaps if you're maintaining a database of thousands of allegedly reputable open-access journals,
there's no way to make sure that they're all telling the truth about their standards and their
practices. At a quick glance, all you can really say is that they would be good-quality journals
if they're telling the truth about how they operate, but it's hard to tell from the outside
whether they're being honest.

So perhaps a different solution is that we don't really need a huge number of good open-access journals.
Rather, in each field, you could get by with a small number of "super-journals" which have a lot of
reviewers on file, and which publish a high number of papers but apply uniformly high standards across
all of them.

Consider: you have two journals, A and B. Each has their own non-overlapping database of 20
reviewers. When they receive a paper, the standard practice for each of them is to send the
paper to 3 randomly chosen reviewers in their database. Each one receives 10 submissions per
month.

Now combine A and B to form one single journal which has 40 reviewers and gets 20 submissions
per month, and still sends out each submitted paper to three randomly chosen reviewers. The
total amount of work performed by the reviewers, doesn't change. But now, if you're auditing
the quality of a journal according to its adherence to its own practices, you only have to audit
one journal instead of two. By the same logic there's no reason in principle that any number of
journals in one field couldn't be subsumed into a few behemoths, which apply uniform standards
across all their papers.

You could do this without waiting for the traditional system to be dismantled. Somebody in the
field just assembles a list of people to be peer reviewers for the "virtual super-journal". That
list is public, so that anybody can audit it and see that it consists of people with a credible
reputation in their field. Anyone who pays the (nominal) fee can submit a paper to the VSJ, which
sends the paper to a random selection of n reviewers from that list. If the paper "passes" the
test, then it gets the stamp of approval of the VSJ, which says, "This paper was judged to be good
by a majority of a random sample of reviewers on our list, and you can see from this list that the
quality of our reviewers is pretty good."

And suppose someone wants to publish their paper in some other journal XYZ, and they also want
to publish it in the VSJ just to get a certification of its quality, but journal XYZ doesn't
allow them to simultaneously submit it to another journal for publication? In that case, you
can still submit your paper to get the stamp of approval from the VSJ -- just pay the normal
reviewing fee, and if it passes the VSJ's review process, they can list the paper on their
website, saying, "This paper was judged to be good by a majority of our reviewers. We can't
actually publish the paper here, because some other journal XYZ has exclusive publication rights,
but you can view the paper at this link in this other journal." You still have the
self-reinforcing cycle where the VSJ's stamp of approval maintains high standards, which attracts
high quality papers, which reinforce the reputation of the VSJ's stamp of approval. There's no
part of that cycle that requires the VSJ to actually "publish" the paper itself.

And people could subscribe to the VSJ's "stamp of approval" feed the way they subscribe to
any other publication -- the VSJ can send out the papers themselves that they have the right
to publish, or links to papers in other journals, saying, "This paper got our stamp of
approval, and follow the link to read it here."

You could even use this process to do a "hit job" on someone else's paper that got published
in another journal, but which you think is too low-quality to have been published. You can
submit it to the VSJ and if the VSJ rejects it, you can ask them to list it as a paper that
failed their review process. (Whether or not the VSJ would give you the option of doing this,
may depend on their policies. It "sounds mean", yes, but academics are supposed to keep each
other honest. I've never heard of a traditional journal doing that -- calling out a paper
published somewhere else and saying, "This sucked, we never would have published it.")

There should probably be multiple open-access journals
(or Virtual Super-Journals) within each field, so that the competition
between them keeps them honest. But there's no reason to have such a huge number of them
that the Directory of Open Access Journals can't keep track of what they're doing.

It's not about how to run a journal, it's about how non-experts distinguish between properly run journals and shitty journals just trying to scam some money.

If I am buying a product, I distinguish the good products from the bad by reading the user reviews on Amazon. Why not just have user reviews for journals? Even better would be to have reviews and ratings for individual articles. A reader could select articles to read based on either all ratings, or only on ratings from reviewers using their real name and affiliation, to ensure the process isn't being gamed. Just like Amazon reviews, the reviews could point out flaws in the article, or provide useful feed

It's not about how to run a journal, it's about how non-experts distinguish between properly run journals and shitty journals just trying to scam some money.

If I am buying a product, I distinguish the good products from the bad by reading the user reviews on Amazon. Why not just have user reviews for journals? Even better would be to have reviews and ratings for individual articles. A reader could select articles to read based on either all ratings, or only on ratings from reviewers using their real name and affiliation, to ensure the process isn't being gamed. Just like Amazon reviews, the reviews could point out flaws in the article, or provide useful feedback for the author.

While I see your point, that's pretty much exactly what the point of peer review is. The difference with peer review is that you're supposed to be an expert in order to review the paper. In your scenario, what you'll end up with is a group of people claiming to be experts doing 90% of the reviewing. The whole point of a peer reviewed journal is that somebody is verifying (to some degree at least) that the reviewer is actually an expert on the topic. This is very different than amazon reviews, where the

While I see your point, that's pretty much exactly what the point of peer review is.

No it isn't. When I read a journal, I can not see what the peer reviewer wrote. The peer reviews are not published with the article. If one of the peer reviewer expressed some doubts, or pointed out some problems with the methodology, that information is lost when the the article is published. Furthermore, no one has the ability to add new reviews if an article is later shown to be reproducible, or is even later confirmed. The information is the journal is fixed and static.

Just look at slashdot comments on nuclear power...everybody sounds like an expert in nuclear physics and in politics. Would you want those commenters reviewing a paper on the topic of nuclear physics, or would you want somebody actually verifying that the reviewers are nuclear physicists?

Why not just have user reviews for journals? Even better would be to have reviews and ratings for individual articles.

In a way that has already been done: impact factor.

http://en.wikipedia.org/wiki/Impact_factor

The number of times a journal article is cited by other articles/authors is a rough guide to how influential it is in the field, the same goes for the journals themselves. It's not foolproof, but not bad.

But it does make it wrong. The journals will usually only retract an article that is fraudulent, but not one that is wrong. So the article is still available to be read, but with no indication that it is wrong.

Lest we forget, a large number of submissions from the paid journals had data that was not reproducible [wsj.com]

I admit my eyes started to glaze over and I didn't finish reading TFS because it seemed like a lot of hand waving and busy work to no real efect, but isn't this proposal, well, a lot of hand waving and busy work to no real effect?

The only way I see to "hoax-proof" a journal is to require reproduction of the results during peer review.

But don't all serious fields have that already? Getting through the review process for publication is just the first step--not all published results are inducted into the cann

The only way I see to "hoax-proof" a journal is to require reproduction of the results during peer review.

But don't all serious fields have that already?

No, they do not. I have never heard of a reviewer trying to reproduce the results. I have reviewed plenty of papers. I will spend about 2-4 hours reviewing something that took the author months of work. All I do is read the paper, make a recommendation, write a few paragraphs of feedback, and email it back to the journal. That's it. This is an unpaid process. There is no way I am going to put my own work on hold for several months to repeat the experiment.

The only way I see to "hoax-proof" a journal is to require reproduction of the results during peer review.

But don't all serious fields have that already?

No, they do not. I have never heard of a reviewer trying to reproduce the results. I have reviewed plenty of papers. I will spend about 2-4 hours reviewing something that took the author months of work. All I do is read the paper, make a recommendation, write a few paragraphs of feedback, and email it back to the journal. That's it. This is an unpaid process. There is no way I am going to put my own work on hold for several months to repeat the experiment.

Well, I hope you do better with the papers you recommend than you did with my comment. If you continue to the sentence after the one you quote, you'll see I'm talking about what happens _after_ a paper is published--other researchers attempt to reproduce the results, either directly or indirectly.

I'm an ecologist. 2-4 hours sounds about right. It takes longer if they're using some fancy new statistic, or if it's not really my subfield, or if it's a particularly long and complex paper. Many papers get a 'reject' with much less time.

I'd say it's actually overly simplistic. It's a complex problem. A simple solution is doomed to only make things worse. "There are a lot of open access journals, and there are some shitty ones, so we should make there be less" is the basic suggestion here. The factors that motivated the journal numbers aren't going to go away just by consolidating them. Publishers of the shitty open access journals who are simply looking for a profit will still simply want a profit. Researchers who just want to churn

This is correct, but the question is how to make this scale to a large number of journals. The more journals you have, the more you run the risk that some are not following this procedure, and it's hard to audit them from the outside.

That's why I'm recommending having a smaller number of journals with a larger output and a larger number of reviewers on call, because then you have a centralized point where the procedures can be enforced.

He didn't set subtle traps that somehow slipped past vigilant reviewers. He included deliberate, very basic, glaring errors that showed no meaningful peer review had ever taken place. It's more like being fleeced by a con artist with 'This is a Con!' tattooed on his forehead - if you're taken in, you really only have yourself to blame.

The Journals all assume that the author is acting in good faith and believes his result. They are 'peer' reviewers, not Police.

Meaningful peer review demands an intelligent evaluation of the author's arguments and evidence and the clarity with which they are presented. Good faith does not imply good science. Belief does not imply good science. Neither faith or belief implies good writing and sound editing.

A major problem with open-access journals is that there is no motivation for them to reject submissions, If anything, the more they publish the more money they make. Likewise, peer reviewers (at least in my field --natural language processing and machine learning) are never paid to review them. This is not a good combination.
I cannot see any reason for journals nowadays. Either publish in conferences (which in some fields are competitive and very tightly reviewed) or better still publish them on arv

Yep, a lot of these "journals" are the academic journal equivalent of diploma mills. They accept any piece of garbage you submit to them, then you get to add said piece of garbage to your resume/c.v., postdoc/grant/tenure application, etc.

Most colleges and even some high-schools run student papers through software packages to detect plagiarism andstructural problems. In fact most careful students do the same thing before submitting papers.

You would think there would be a first layer of software detection that could throw up enough red flags to sent the papers back without any wasted efforts. Not JUST on plagiarism (which might be harder to check, since quoting other studies in a paper

Except those paid journals have also had serious hoaxes foisted on them. You have to go to really really really really big journals like science or nature before there's enough credibility to protect against fraud.

They seem to select for papers which will have a bit impact in the general press rather than necessarily for good ones.

Well, at least Nature admits as much in their Mission Statement:

First, to serve scientists through prompt publication of significant advances in any branch of science, and to provide a forum for the reporting and discussion of news and issues concerning science. Second, to ensure that the results of science are rapidly disseminated to the public throughout the world, in a fashion that conveys their significance for knowledge, culture and daily life.

Even big discoveries in small fields with no significant tie in to wider fields of science or daily life will not be accepted, which is fine. Their mission statement is fairly clear on that point. They are not focused field specific journals.

Except those paid journals have also had serious hoaxes foisted on them. You have to go to really really really really big journals like science or nature before there's enough credibility to protect against fraud.

Actually, no journal is fully immune, no matter how prestigious. Worse than that, top-notch journals like Science and Nature require sensational stories, which makes them more likely to publish skillfully hyped-up reports than honest ones that acknowledge their own limitations. The best science is often found in mid-range journals that accept longish and seemingly boring manuscripts, with nothing swept under the rug.

Except those paid journals have also had serious hoaxes foisted on them. You have to go to really really really really big journals like science or nature before there's enough credibility to protect against fraud.

Wait, you're saying Nature and Science don't get frauded?Seems to me both of those Journals were taken in by Jan Hendrik Schön.And paid journals also have turned down key papers, including Nature's total snub of Watson and Crick's 1953 paper on the structure of DNA.

Well, then, you don't recall very well some of the major issues for which Elsevier was targeted. Such as printing fake shill journals under the Elsevier label just for articles produced by Big Pharma PR, so that pharmaceutical companies could pass requirements for "peer reviewed" studies of their products with a citation to a "peer reviewed" study (in their privately purchased Elsevier journal). The Elsevier upper management are profiteering crooks, plain and simple, which is a pity, because they've bought

Well, then, you don't recall very well some of the major issues for which Elsevier was targeted

The fraud you are referring to is, indeed, reprehensible, but the publisher (and I remain convinced, other publishers were blamed too) was targeted simply for wanting profit. Indeed, you are doing just that right now:

Why yes, "profiteering" is a dirty word to me. Saying "we care less about the human lives potentially harmed by willfully publishing potentially fraudulent medical information in order to grab a few bucks" is among the most despicable things someone in their position would be able to do. Maybe you have lower standards for what constitutes crimes against humanity. Are you waiting for Elsevier's management to personally strangle your children before your eyes before passing judgment?

The boycott of Elsevier was primarily related to their "bundling" of journals---the act of forcing libraries to buy subscriptions to their low-impact, narrowly focused, but very expensive journals in order to have access to their high-impact, high-circulation journals. See http://www.nature.com/news/elsevier-boycott-gathers-pace-1.10010 [nature.com]

Think about this process.

Elsevier prints journals for which they receive their content FOR FREE from academic researchers, most of whom are funded by taxpayer money. They the

It doesn't matter if it is an "exclusive" journal or one that is open access. If a scientist submits fake data to a publication, shouldn't the scientific community take the time to verify his results? I'm pretty sure that had he just made up his findings someone somewhere would have called him on it and his cred amongst his colleagues will go down the toilet. Isn't this how cold fusion was proven false?

It doesn't matter if it is an "exclusive" journal or one that is open access. If a scientist submits fake data to a publication, shouldn't the scientific community take the time to verify his results?

Yes, once it's published. That means, BS in a "reputable traditional" journal will cause that, BS in an unknown open access journal will just be ignored.

But that's the case for published papers. Submitted papers are just vetted by a couple reviewers. And they might get paid the same for open access or traditional: zilch. The difference: Traditional has better connections to experts in the field. Who will reject anything that is out of the ordinary (either flawed or just against standing assumptions.)

Notice he only submitted his fake papers to open access journals. As a scientist, and especially as a biologist, he's perfectly aware of the importance of control groups. If he were honest, he would have submitted the same papers to closed, for-profit journals as well, even if it cost him money to do so.

This wasn't an experiment, it was a stunt to highlight a problem he saw.

Bohannon's not addressing a problem with open access, since closed-access journals have printed junk articles in the past as well. He's addressing his confirmation bias by setting them up for a James O'Keefe-style hit piece, minus the fake prostitute.

Notice he only submitted his fake papers to open access journals. As a scientist, and especially as a biologist, he's perfectly aware of the importance of control groups. If he were honest, he would have submitted the same papers to closed, for-profit journals as well, even if it cost him money to do so.

Yes, his study was not scientifically valid. Now his only option will be to publish his paper in an open access journal, instead of closed, for-profit journal* that only accepts scientifically valid papers.

*The summary was based on an article in Science that simply described his experiment, not a peer reviewed scientific paper in Science.

you have fewer journals that have to be audited for procedural honesty

Taking this to its logical conclusion, a monopoly is the most honest organization, right?

Once one of these "fewer" journals has an established reputation, it can obscure its procedures and refuse to be audited, while it turns corrupt for profit. Since it's still a well-known journal (because who really has time to monitor the procedural audits, anyway?) it will still get the submissions and readers, and it will stay relevant for many years after "everybody knows" its' corrupt.

Modern journals are all online. So, it would be easy to provide a traffic light system that indicates if a reference is found to be fraudulent or incorrect. In a correct paper, all references would have a green background. If one of those papers was found to be incorrect by a 3rd party, it would be flagged and its colour changed to amber. This would propagate to all papers that make a reference to that paper and the entire chain would become amber. This would force all authors to update their papers, or the author of the original flagged paper to correct their work. If a paper is just flat out wrong, discovered to be a fake, or fails to be updated after a period of time in amber, it would become red, which again would propagate to every paper that uses it as a reference.

This will keep the chain of dependencies clean throughout the entire scientific world and minimise the impact of improperly peer-reviewed work.

A little subtlety would be required. In a document I'm working on, a whole load of references are papers containing flat out wrong, erroneous information (honest scientific mistakes, not deliberate fraud, but disproven material nonetheless). In the text, I'm using these as examples of things that have gone wrong in the past history of the field. Would my paper, with its bibliography littered with "red" entries (that I put there because they were "red," in order to comment on the scientific process --- which involves making and correcting mistakes --- leading to the present-day state of the field) be tagged as "red"? What if I reference a paper in which some material has held up under later scrutiny, and other hasn't (basing my work on the more "correct" material)?

This paper has already been extensively critiqued. To me the biggest problem is that he didn't include any subscription journals.

Many intentionally flawed or nonsense papers have been submitted -- and published! -- to reputable journals in the past.

This latest demonstration by Bohannan just shows that the peer review system needs improvement. It does not show whether Open Access journals are better or worse than subscription journals in terms of quality and reliability of content.

Of the 304 open-access journals targeted by the sting, 60% published the paper. I think this mainly just shows that the average quality of open-access journals may always be low, but that's not surprising since anyone in the world can set up an "open access journal".

Is this information really meaningful without a similar test on the paid journals?

There have been plenty of incidences in which paywalled journals have also been shown to, um, let dodgy papers through. Although I agree, it would be interesting to see what happened with an identical test:)
Also, here's another take on the entire thing - it's not open access vs paywalled, it's peer review (and shows an awesome opportunity to improve it!). http://www.theguardian.com/science/grrlscientist/2013/oct/08/1 [theguardian.com]

If ten a month is the standard load for a reviewer, I think there's already a problem. Reading an article should probably be allotted at least an hour. Any fact-checking will take more. I read articles for the humanities, and that's pretty easy. You can spot a bullshitter pretty quickly, in a page or two. But I'd imagine science can be trickier. So, the half-hour or so it might take me to be sure that it's crap would probably double for science. At a minimum. I say minimum because reading a stack of a dozen

My point was that regardless of how much time it takes to properly review a paper, the total workload doesn't change if you have a smaller number of journals publishing a larger number of articles per journal, rather than a huge number of small journals. The only variables that matter are the number of incoming submissions and the number of reviewers per submission, and I was arguing you can keep those constant.

Anybody who expects peer review to let through only correct papers doesn't understand peer review. Science editors, of all people, should understand that, given the many bogus and fraudulent papers they have published over the years.

The publication process has gone so far downhill it's basically not recognizable as science any more. This is driven by the university tenure process. Being a tenured professorship is a sweet job. The hours are short and flexible and the work is interesting and varied. Pay is less than industry, but once tenured the pay is guaranteed. Benefits are usually top-notch. That's an appealing package for anyone of reasonable intellect, middling ambition, and a desire for ironclad security. Not surprisingly,

... then you would be able to vote his paper down for being nonsense. This would then reduce his reputation and would:
1) Reduce the rating of future publications
2) Reduce the affect of feedback that he leaves others
3) Reduce his funding.
I.e. Post bad work and reduce the chances of succeeding as an academic.

Ok, maybe "fraud" is a slightly strong term, but it's pretty close. There are thousands of "open access journals" created only to make money by sounding as if they were legitimate journals, getting people to send them articles, and then charging to publish them. I get spam from them on a daily basis. They aren't legitimate, they have no interest in quality, and they have no reason ever to reject an article.

Please don't confuse legitimate open access journals like PLoS with these scams.

Really good post. I think you've hit on the key issue, which is that peer review can be done better.

One problem is that peer review probably is done well in a lot of cases; we only hear about it when the system breaks down. From that perspective the most obvious solution is to start focusing on peer review and giving reviewers credit for the times where they do a good job. One way we're doing that is by assigning DOIs to post-publication reviews that the commun