Why bother publishing in a journal?

A couple of weeks ago I began writing a grant to get money to fund my trip to ScienceOnline 2012. The funding agency (UNM GPSA) specified that I write no more than 700 words and right around word 350 I hit kind of a writing block. I felt I had said everything there was to be said and I just couldn’t think on how to expand on that.

I didn’t want to just stop writing though. I put the article in Google Docs so I could share it and then I asked for feedback. First I asked some friends, but I quickly realized they wouldn’t be able to read it right away. So I did what I thought was the next best thing (but turned out to be the very best thing), I asked my Twitter followers for feedback.

Then the most amazing experience of my young scientific career happened. Within 20 minutes I had about 15 viewers and several of them had left comments on how to amend the grant. I was so excited. I evaluated the comments and made the changes that I thought were for the best (which turned out to be most of them). I even had a few ideas to expand on my original thoughts.

I followed this procedure again about a week later (a few days before the grant deadline) and got even more feedback. After implementing those changes and adding some of my own, I had a complete grant for submission. I’m now awaiting a decision.

I had just had an experience with peer review, it was anonymous peer review, but it was peer review nonetheless. Anyone who came across my article was a Twitter Follower (and all who follow me are either scientists, science writers, or involved in science in some way) or a follower of those followers. And it was normal peer review. This was real-time peer review, and it was SPECTACULAR!

Fast forward to yesterday.

I was having an email conversation with Mark Hahnel (FigShare founder) where I told him that story above. It had right then and there occurred to me… Why would anyone want to publish via the traditional peer review method when they could have the same amazing experience I had just a week ago? Why spend days, weeks, months to write a paper for it to sit in obscurity for months while someone else devalues all the hard work you put into the paper? Why deal with publishing cost and editorial staff when you can do it all yourself for free?

Why bother publishing in a journal at all?

How would peer review work without bothering with publishers? Would it even work? I think the answer is a resounding yes. Scientists don’t get paid to peer review an article. They sacrifice their time and contribute their effort to review articles for submission. Take out the middle man and you still have the exact same system, only in the case without journals the writer would have to hand pick peer reviewers to do the job.

Not only could you opt to pick your own peer reviewers, but you could also do what I did. Ask Twitter! In the case of Google Docs, any person not on the shared document list (ie those who show up to a public file) appear as “anonymoususer###.” To fix that issue, just ask that any potential commenters leave their name attached to their comments. This way you can judge the credibility of the comments. After all you don’t want a CEO of McDonalds to comment about the diffusion constants of GFP in differing viscous solutions, unless they have a background dealing with such problems.

The beauty of the Google Docs example is that your document, once made public, should get indexed by Google (search results!) so people can find your article. You won’t have to deal with editorial standards of peer review journals, if you want to discuss your topic for 1 page or 30 pages it makes no difference to Google Docs. The peer review process is much faster now (days or weeks vs months or even years). And finally, it’s free! No more paying publishers or worrying about the publisher’s sustainability (if you even do that anyway).

This is obviously not a perfect solution (yet), but with all the complaints I’ve heard about the peer review process and for-profit publishers, this seems quite ideal.

I’d also like to extend this notion to those of us in open notebook science.

I work really hard every day to compile all the scientific information I’ve accumulated all day and publish it in an easy to read format. Why should I have to do all that work again just to make the information more concise and less usable? As long as I make my information easy to navigate then I think the notebook is a much better publication platform than journals.

Not only that, but online tools like FigShare and BenchFly make it really easy to compile supporting information on the web. With BenchFly I can make videos that show exactly what my methods are, which may make my method posts even more clear. With FigShare I can post data sets that are citable(!) and link those sets to my notebook and link my notebook to those sets.

In my email conversation with Mark, I said:

FigShare has changed the way I think about publishing. Now I see endless possibilities… I can publish stuff in my notebook and organize the data on FigShare get a handle, cite the data, link to the methods, and write some conclusions and voila free publication in my notebook! Why would anyone want to do it any other way?

Seriously, why would they?

The point of peer reviewed publications is to get scientific information out to the world, but today that isn’t happening fast enough, nor is it simple enough. Why should anyone have to dig into an article to discover their real methods and reinterpret their data? I don’t think anyone should have to.

With an open notebook, someone could just follow the trail from their FigShare data (or whatever they decide to use to host their data), to their conclusions about that data, to their methods on how they came about the data, all the way to the ideas that led to the experiment in the first place. There is no interpretation, just a step-by-step guide to the science that led to your discoveries.

And with most online technology working to incorporate social media like aspects, peer review can happen in a notebook as well.

Like this:

Post navigation

The point of peer review is not to get you work out there… that is the goal of publishing your work. Peer-review is intended to evaluate the validity of the scientific methods used and the conclusions made, while providing feed-back to improve the manuscript. While I agree that the current journal-based system is inadequate in today’s research and internet based environment, I do not believe that the system can, at this time, be forgone completely.

I’d like to se an elimination of the closed-access and print versions of journals, with appropriately peer-reviewed articles published directly to open access online repositories combined with post-publication review/feedback. This would allow for “publication” of any scientifically valid studies including null-hypothesis work that’s not sext enough to be published in traditional journals.

EDIT: Also, you should not be able to pick who peer-reviews your work. I think even polling your twitter followers provides as potentially biased pool of reviewers for publication-style review.

http://www.iheartanthony.com Anthony Salvagno

A side effect of the things that I’m talking about here is that your work is out there, but it is only because you’ve opened your work to an open peer review model. One that has names attached to it.

Not only that but all the pluses of peer review are made better in the ways I discuss above because there is the potential for more opinions. Of course this may actually make it worse because you may have too many differing opinions.

http://stevekochscience.blogspot.com Steve Koch

What evidence is there that anonymous peer review (as has been around for only half a century-ish) adds value to publications? Or, more importantly, adds more value than it subtracts when accounting for the delayed release of information and the manpower spent performing and administering anonymous peer review. Over my career, I have seen no well-supported arguments in favor of the current anonymous peer review system. That’s not saying peer review has no value, but why stick with the broken, anonymous system we have now? Anthony is approaching this from the bottom up. Opening all his work to peer review. Certainly, most will be ignored, due to time constraints of all. But when he writes something that captures attention, it can be peer reviewed by anyone who wants. I agree with (both of you?) that it has much more value if it’s attributed. Nothing about open science prevents that. The extra piece of gold is that once something captures attention, if Anthony’s work is successfully open, anyone and all can click down into the details which are automatically available. Even with projects that I would call “top down,” trying to open up the peer review process, there is a lot of work that needs to be done in order for all the raw data, notebook entries, code, etc. to be available for the world to peer review. That’s why I think Anthony’s (and other’s) “bottom up” approach (aka “open notebook science”) will play a key part in revolutionizing and maximizing the impact of scientific publication.

Patrick Jurney

The value, as I see it, that anonymous peer reviewed journals have is in their monopoly on access to the qualified reviewers. Scientists in a field relating to a journal submission review manuscripts for a variety of reasons. Though I’ve never done it, I imagine they all lie somewhere between an altruist advancement of the body of human knowledge and squirreling capital with peers to be spent for their own advancement. These individuals, regardless of their motivations, are incredibly busy. Any reservations that I have about open peer review would be mitigated by a merit based system for reviewers. As an individual publishes and reviews increasingly cited works their value increases and what
pops out is a new world order so to speak with the “middle man” eliminated. Now, I have no access to whatever good peer reviewed journals may or may not do with the spoils of their system and I think it’s worth noting that it may be valuable.

http://www.iheartanthony.com Anthony Salvagno

I’m pretty sure reviewers don’t make money for their reviews. But the publishers make money from ads and sometimes (always?) from the submitters.

http://twitter.com/easternblot Eva

Submissions COST money rather than bring it in!
An unsuccessful paper costs a few hours of an editors time to figure out which reviewers to send it to, and process the results of that when it comes back. Journals that only charge authors upon acceptance of papers are basically charging successful authors for the cost of weeding out the unsuccessful papers. For journals with a low acceptance rate, the cost to the authors would be IMMENSE to break even. That’s why many OA journals adopt more lenient acceptance criteria (i.e. anything that is scientifically sound)

http://www.iheartanthony.com Anthony Salvagno

You don’t need to validate the cost of running a journal to me. I understand they need to sustain themselves. I’m not attacking the business practice of journals here. I get that someone has to pay for publication fees.

And you are right, the cost to authors would be immense which is why Science and Nature charge ridiculous fees to subscribe. If that stuff were open access they wouldn’t make much money.

But Open Access publishers have to charge money as well, and they choose to put the cost on the author. That’s just the premium the author accepts to have their article open access. Seems like that model is working pretty well right now. And from my understanding there are journals that have a selective acceptance (high impact) like PLoS Bio (from what I’ve been told). That’s all fine and dandy. Personally I like PLoS One’s ideals, who says any research isn’t high impact? If it’s useful then it needs to be out there in the cloud.

http://twitter.com/easternblot Eva

Not sure if it’s changed since this was written, but PLoS Bio doesn’t bring in as much as PLoS one for exactly that reason: http://www.nature.com/news/2008/080702/full/454011a.html

http://www.iheartanthony.com Anthony Salvagno

So self-hosting your own publications seems to be the way to go! Free for authors, free for readers. Sorry journals…

Like you and Jason suggest, there’s no need to couple all these things together, which are in fact often at odds (I was once told the best papers were those that could support the most astonishing claims with just enough data to suggest that it might be true, since they would be sure to generate more science than either more obvious or more definitive conclusions). So what will replace each of these tasks if we no longer rely on journals to provide them? You’ve outlined feedback, dissemination, and archiving solutions, but I think people worry most about “certification,” for the pubs and for scientists themselves. As long as we are evaluated by folks outside our area of expertise (which is probably a good thing) that don’t have time to read and understand all our work, we need metrics to know what to read, who to hire, fund, etc.

http://www.iheartanthony.com Anthony Salvagno

What do you mean by certification? If by that you mean journals are targeted ie people who are interested in physics read Physical Review Letters, etc then I feel like search takes care of that. You just type your interests into Google and away you go.

As far as metrics go, people will just have to decide what they value as far as scientific contribution goes. Do you rate peer reviewed publications with medium citations high? Or do you prefer that someone posted a youtube video about kinesin motility and that has 1 million views. Jason et al are working pretty hard to make some alternative metric tools which may make some decisions easier, but I feel like this is like the NBA Draft. Sometimes guys like Jeremy Lin just don’t get noticed by scouts, go undrafted and turn out to be pretty good (sorry to ride the Linsanity, but I’m a big Knicks fan).

Are article citations good? Sure. What about page views? What about comments? What about tweets, retweets, facebook shares and likes, +1’s, etc? Journals don’t even really measure that stuff except citations. So I don’t know if people are going to really miss that aspect of the journal.
We should teach everyone how to add web analytics to their sites. Once they see the stats roll in, I bet they’d be too excited to care about those paper bound relics!

http://www.carlboettiger.info Carl Boettiger

Of course I agree that knowing the number of publications someone has and what journals isn’t a good predictor of scientific quality, and probably not as good a predictor of funding & hiring as some suggest.

I think people looking at CV publication lists aren’t being guided impact factor as much as by personal experience. No one cares that PLoS ONE has a higher impact factor than most of the specialty journals in my field. They think having a Nature or Science paper is a reliable signal of success not so much for journal impact factor, but because personal experience tells them how exclusive it is.

Jason says once alt-metrics are there, people won’t be able to ignore them (in the words of Chekov). But I think it will be harder than that: First, like you say about web-analytics, everyone needs to know their own personal metrics, as a benchmark of what’s impressive and what’s not. If you’re out-competing me in a metric I’m not competing in, it’s easier to dismiss. Second I think the alt-metrics will have to largely reflect existing preconceptions about importance — identifying those who are best funded, best-published, most sought-after for talks, with only a few surprises thrown in. When/If it supports our intuition of “good science” better than publication length does (which shouldn’t be hard, since almost everyone is frustrated with that system), then I think we’re free.

http://www.iheartanthony.com Anthony Salvagno

You’re last statement really hit the nail on the head. Right now no one cares about alt-metrics (sorry guys) because most scientists don’t value social media or web analytics as any kind of statistical tool. Maybe they are right, but I think the numbers are at the very least fascinating. Plus it’s other numbers than just citations and pages of your publication, which has to be good.

Right now I’m training a class to not be afraid to explore the internet for answers to their questions. Get them familiar with the multitude of resources out there. And I think once more people start embracing these new tools, students will embrace them and maybe we’ll hit the tipping point when it comes to tearing down traditional science.

Srrven

You couldn’t have said it better. The sad part is that more than 50 percent of scientists still don’t use social networking tools, case in point the empty social networks for scientists. And besides over information in any field is an issue, which is bound to happen with open notebook. The best we can do currently embrace the tools that make other journals complete, like figshare, Journal of Errology and datadryad. Open Science will happen, but I guess not in our generation.

http://www.iheartanthony.com Anthony Salvagno

Don’t fret! Open science is happening now a lot more than we know. Sure we’re not at the point where most people are doing it or where most people will be doing it soon, but this is the internet age. Don’t discount how fast information moves and gains momentum. It will happen in our generation and to me it actually feels like we’re almost at the tipping point. And I think training young and future scientists in this regard will bring that change faster, as will laws that are enacted to force open access to publicly funded research.

http://www.benchfly.com/ Alan Marnett

I agree with Anthony. We recently polled scientists and asked them how long, if ever, they thought it would take before journals were no longer the primary mechanism of publication (The impending death of scientific journals). Nearly 40% said it would happen in under 10 years and 70% said within the next 20.

Obviously, this isn’t proof of anything, however I think it does reflect the continuing shift among researchers in their view of how publication should work. For many of us who have grown up with the current system, the knee-jerk reaction to removing peer review and journals from the process is to say “it can’t happen, no way”. While there may be pros and cons to the current peer-review system, many of which were articulated above, I think the idea is to first convince people that there *may* be another way of doing things. Maybe an alternate solution is better, maybe it’s worse, but we can debate that as we go along. However, if people don’t believe there’s any other way to publish our work, then discussing the details of the new approach is wasted breath.

Younger generations of scientists (who, unlike me, will never know the pain of looking up papers in the old 50-lb chemical abstracts books in the library) are keen to the idea that the Internet provides the opportunity for things to be different. As a result, they may be the researchers, like Anthony, who experiment with various systems of “alternate publication”. Again, some things may work in their initial systems and some may not, but it will evolve.

In my opinion, this is not about demonizing peer review, journals, or business. We all understand that publication carries with it significant costs for all of the functions mentioned above (publishing, marketing, archiving, etc.). Rather, it’s about finding out if there’s a *better* way to do it. Can we create a system that gives more people the ability to share their work at lower cost (eg, self-publication on their lab website), while still including some sort of vetting process (review, alt-metrics, etc.) in order to help those who are not experts in the field to understand the quality/importance of the work?

I think we can, and will. This is not to say there won’t be bumps along the road. It’s also not to say that the solution won’t be created by journals themselves. Who knows- maybe they evolve with the times. But I find myself among the 70% of researchers who think the next 20 years will see the emergence of a new primary mechanism of publication – and it could be much faster than that. It may only take a few big names (or a well-respected department) to announce they’re fully embracing an alternate system to tip the balance.

http://www.iheartanthony.com Anthony Salvagno

I seem to think it may take less than a few big names. I mean this is the internet age right?! People didn’t start using youtube because Justin Bieber used it, in fact it was the other way around! I think once we can get a larger base of “unknowns” to publish in alternative ways others will follow suit. I don’t know why but that seems like less effort than convincing the big wigs to change their thinking. Maybe because the people I want to target are undergrads and grads.

Just to peer review your blog post, I think there is a big flaw of reasoning in there, where you compare apples to oranges. It may not invalidate the greater cause, but it’s a shaky example to build on.

You wrote:
“After implementing those changes and adding some of my own, I had a complete grant for submission. I’m now awaiting a decision.I had just had an experience with peer review, it was anonymous peer review, but it was peer review nonetheless.”

No, you did not have an experience with peer review yet. Waiting for the grant decision, THAT is the peer review in this particular process. Not writing the application together with a bunch of other people. That is merely the collaborative process. Nobody has yet decided whether or not you get the grant.

There may well be alternatives for the way research is currently published, and there are good points to be made about the value of putting data out there for anyone to see, use, and comment on, but this example is not making a case for that. This is merely showing that you can have people help you write a grant that is part of the traditional (peer reviewed!!) grant application process.

http://www.iheartanthony.com Anthony Salvagno

Sure the grant review is the peer part of my example. But replace grant with the word publication and the story is exactly the same. And that is what I was doing when I told the story. I had imagined that what if instead of writing a grant I was writing a paper to be submitted to Nature or PLoS One or something. I would do the same thing and I’m literally about to do this. I would write it up either in my notebook or on Google Docs (or both) and tweet it for all my scientist friends to read, rate, and comment and correct. Then I would have a peer reviewed article.

And to be clear, it wasn’t collaboratively written. No one had access to edit the document other than to leave comments. So others could offer suggestion as to how to change my writing, but no one could actually write it. And it is up to me to input the edits and exactly how to do that. Isn’t that the nature of peer review? IE “you won’t get into XX Journal if you don’t make changes that reflect these suggestions.”

http://twitter.com/easternblot Eva

You make it sound like the only function of peer review is a filter to decide what gets published in a particular journal. But looking at it from a much broader perspective, the journal publications are the indicators of how a paper fared within the peer review process. The people who do the actual reviewing are still going to be the same peers – whether they have been asked to review your paper for Nature, or for a small niche journal, or whether they leave comments on something you put online. And the comments they give will be exactly the same. If you ask Big Person In Your Field how he thinks your work is, he’s going to tell you just that. That’s the peer review.
So if the journals were entirely factored out, you’d still get the same feedback from the same people – true. But who is going to ask them to comment on your work? Are you going to ask someone who might be overly critical, or rather someone you know will be friendly and supportive? And what if they don’t have time, or refuse to participate in the peer review process – who’s going to hold them accountable? That’s what journal editors do. It results, by convention, in published papers in particular journals, but who is to say what the output will be in a few years. It could be smaller fragments of output, requiring only small comments along the line of “where is your control?” rather than a discussion about whether or not the work contributes to the field. But there should still be people who make sure that it all happens fairly, and there should be a way to distinguish between someone getting 20 positive comments on one unanalyzed dataset or someone getting 5 positive comments from competitors about a well-worked out theory changing the field. Currently, that distinction is roughly based on whether or not something resulted in a published paper, and in what journal, and how many people cited it (i.e. found the work worth building upon).

I don’t yet see how merely putting things online for general critique is going to offer a similar kind of system where you can see at a glance who is merely productive, and who is actually making a difference. (Quantity vs quality). And, as I said, someone’s going to have to hold people accountable for taking part in the process. Finding reviewers for papers is time consuming, and it’s not going to be any less time consuming to ensure peer review is done fairly in any other format.

http://twitter.com/dalloliogm dalloliogm

Hello Anthony,

last year I launched an initiative to write an article of the PLoS
Computational Biology “Ten Simple Rules” series as a collaborative
article. Together with some colleagues, we wrote a draft of the
manuscript, we uploaded it to a public wiki, and we invited people on
twitter and on many mailing list to contribute. It has been a fun
experience, and the quality of the article increased a lot.

I agree that your story is exciting, as receiving feedback from other persons online is a nice experience. However, I think that it is too early to change how peer review works: there are just too many researchers who would not know how to use online tools such as twitter, and they would remain cut out from the scientific community if everything was made online.

http://www.iheartanthony.com Anthony Salvagno

I completely agree with that! It is too early to tear it all down, but little changes here and there aren’t unreasonable and PLoS to me is a pretty good step forward. My advisor Steve Koch likes to call them a top down approach. I’m working from the bottom up with open notebook science and that is another good step forward. The only way change is going to happen, is the same way it always does… education. And with that said I’m trying to take the next step forward by teaching students how to use online tools and social media for scientific purposes. Check out the efforts here: http://2012juniorlab.iheartanthony.com