How far we have come in communicating and discussing science

Responses to published articles (when they happened) took a year or more. Mostly people just talked about problems they saw (or thought they saw) with people in neighboring offices.

Today:

A paper is published

Within hours a member of our group posts it to the “Papers of the week” page on our wiki [1]

A few hours later another member of our group posts a comment saying “Hey, there’s something about this that doesn’t make sense to me” and then a day later posts again to say that they are pretty convinced that something doesn’t add up [2]

A few days/weeks [3] later I’m traveling and on the train ride from the airport to my hotel I take out my phone, open up twitter, and see a tweet, that is pointing to a blog post, that purports to show proof that the paper is wrong

Crazy.

Now, I still haven’t read the paper, and so I have no idea who’s right or what’s wrong [4]. And of course this sort of thing has been discussed extensively in relationship to #arseniclife. But there was something about the fact that it was happening in ecology, in an area of research that I was professionally interested in, that really brought home to me how far we’ve come in last decade [5].

————————————————————————————————————————————————————————

[1] I think – “I really should read that”

[2] I put it on my To Do list to read the paper once I get back from an upcoming trip and, since I’m apparently already out of touch at the tender age of 34, think – “If there’s really something there we’ll have to think about righting a response” (It never occurs to me to do anything other than write up something formal and send it to the journal)

[3] Who can keep track really

[4] Which is why I haven’t actually linked any of this to the paper/posts/tweets in question. This post is really about the process, not the science.

[5] And then just to drive the point fully home, after I dropped off my stuff I picked up my phone, hit the Nearby button on Yelp, found great Chinese food a quarter of a mile away, had the GPS walk me there, came back, enjoyed a video call with my daughter, and then fired up WordPress to post about my day. Before bed I’m going to go for a quick trip in my flying car.

8 Comments on “How far we have come in communicating and discussing science”

In the spirit of this post, I just “Stumbled It!”. I don’t even know what that means or what it does, but it was a link sitting there at the bottom of this RSS feed, and seemed like it must be cutting edge.

Don’t you think the kind of substantive post-publication review the arsenic life paper received is still very much the exception, though? After all, the vast majority of papers receive no comments or other post-publication review. Is the sort of process you’ve described just an accelerated version of the old way of doing things? Or is it some kind of more substantive shift in how we evaluate science?

I think it’s a more substantive shift in that we’re moving the old “water cooler” conversations into a place where everyone can participate in them. This includes things like Research Blogging style posts and “Hey, you should read this” or “I liked this” tweets in addition to the more substantive and critical treatment that was seen with #arseniclife. Most papers won’t motivate that kind of response, but that doesn’t mean that post-publication review isn’t operating, it just means that folks aren’t freaking out about the papers.

Ok Ethan, I don’t disagree. But I suppose I should just ask the question behind my other questions: can (and will, and should) post-publication review ever *replace* pre-publication review? I don’t claim that’s a straightforward question, especially as it’s badly framed (there are intermediate possibilities, such as post-publication review of arXiv-type online preprints followed by pre-publication review when those preprints are submitted to journals). But it is a question that worries me. Few papers attract any post-publication review at all, but as an old fogey I find it reassuring that they’ve gone through some pre-publication quality control. It’s hard enough these days lining up pre-publication reviewers, but we can still do it because a sufficient number of people still see it as their professional duty to do pre-publication reviews. But does anybody see it as their duty to do post-publication reviews? The strongest advocates of post-publication review *as a replacement for* pre-publication review seem to basically take the attitude of “just throw everything on the web and let ‘the crowd’ sort it out”. I’m yet to be convinced that that kind of system would effectively perform, *for all or even most papers*, the functions that pre-publication review performs (and I do think we want those functions performed) Indeed, even in the case of the arsenic life paper, would post-publication review have been as vigorous (or have even occurred at all) if the paper had not first been published in Science following pre-publication review? Surely what attracted all the post-publication attention was the prominence of the venue (and part of what makes the venue prominent is the rigor and selectivity of its pre-publication review process), and the fact that the paper had already been through quality control, which generated pushback by people who didn’t think the quality control was sufficient.

It’s a great question (and one that a full explication of which is beyond the scope of a Saturday afternoon comment). I would say that in general I have the same reaction as you to the folks who want to replace the pre-publication review system. I consider this aspect of quality control to be valuable and if for no other reasons than the regard with which pre-pub-peer-review is held by the higher administrations in academia and by non-scientists (courts, politicians, etc.) I don’t think that pre-PPR is going away anytime soon. That said, as the reputation economy of academia increasingly moves online I suspect that the role of pre-PPR will shift towards something that is of primary importance to the authors of papers, so that they avoid embarrassing mistakes and can figure out how to best cast their research to the broadest audience (in which case good friendly reviews are just a valuable as formal journal reviews). There are a lot of challenges to moving towards a more distributed review and publication system for science, but given the rate of change in publication in other areas I would be pretty surprised if the system doesn’t start to move away from traditional journals in the next decade. Given the proliferation of what I consider to be lite-review journals, I’m not sure that pre-PPR per se is going to mean much in 10 years anyway.