Pages

14 September 2009

I didn't really mean to present an object lesson in why peer review is a good thing. But, having done so, it seems a good time to use it to illustrate what the process looks like.

First step is, somebody has to put something forward for consideration. In this case, my note on field relevance last week. One important aspect of this is, the 'something' has to be said concretely enough that people can point to the mistakes you've made.

The second step is that the comments (reviews) have to point to specific things that are wrong. Ranting about leftists (happened elsewhere) doesn't count. Saying that I grossly understated the relevance of biologists because -- and give reasons for that 'because' -- does.

The third step is for the original author to revise the article in response to the reviewer comments. That doesn't necessarily mean 'do what every reviewer wants', not least because the reviewers (c.f. gmcrews and John Mashey) may disagree. But there should be at least some response, if only to add some explanation in the article that addresses why you're not doing (what reviewer X wanted). I'll be doing that later, but am waiting for words from the biology folks about how the field applies to deciding whether and how much of recent climate change is due to human activity.

To summarize the comments some here (do read the originals if you haven't already):

Many fields are missing

Many fields are placed too high or too low

(mostly too low)

I conflated two different questions -- whether and how much warming there has been, with whether and how much of it has been from human activity

(Some irony there, as one of the things I did say was that the picture changes depending on what exactly the question is.)

Irrespective of whether the previous points were addressed, the approach itself is not useful

Each of this is a common general sort of comment to see in a peer review. To rephrase it more generally:

Incomplete

Inaccurate

Question is not specific enough

Question is not interesting, approach is not useful

In terms of my rewriting process, the first two are pretty easy to deal with. Many people made many good comments. Those can be incorporated fairly straightforwardly, along with the fields that the comments prompted me to remember even if they weren't directly mentioned.

The second two, however, aren't quite so obvious. The third is taken care of if I go clearly to making the question addressed "How much of the recent warming is due to human activity?" And that is what the graphic actually tried to address (though still with some issues with respect to the first two sorts of comment).

But, is it useful to address that question in this way? My thought was that for non-experts, it could be a useful guide when encountering, say, a 'conference' whose speakers were almost entirely from the lower ranges. On the other hand those antiscientific conferences are seldom so specific about what they're addressing. Either the figure is focussed on too narrow a question, or many separate such figures would be needed. Experts, or at least folks at, say, K6 and above in Mashey's scale, should just go read the original materials to decide.

I haven't decided which way to go on this. Comments, as always, welcome. I also realized that it's a long time since I wrote up my comment policy, and link policy, so they are now linked to from the upper right (in the 'welcome' section).

In the mean time, I'm taking down my version of the figure and asking those who have copied it to remove it as well.

But, to come back to peer review: All this illustrates why it is you want to read peer-reviewed sources for your science. Nobody knows everything, so papers can otherwise be incomplete, inaccurate, etc.. People can also think that something is obvious, but have forgotten about things that they themselves do know (like my temporary brain death about biology as a field for knowing that climate is changing). Or they know certain things so well themselves that they don't write it up well for the more general audience. (Even in a professional journal, most of the readers aren't in your particular sub-sub-sub-field. 'more general' may only mean make it accessible in the sub-sub-field instead, but that can still be a challenge.) In a productive peer review process, these questions are all addressed.

6 comments:

Anonymous
said...

How can biologists help answer the question are humans responsible?

It's a good question. One answer that springs to mind is assistance in isolating what are and aren't effects of climate change (I'm thinking along the lines of range or abundance change and the like not necessarily being climate driven, rather due to changes in land use - something an ecologist would be better able to point out than other disciplines), but that's not really answering the question (there may of course be knock-on climate effects caused by changes in land-use but the attribution of these are probably beyond the field of ecology).

I'm struggling to find an answer, I'll be interested to see if someone else comes up with one...

In 1977, there was a conference called "Language Design for Reliable Software", about languages like ADA that had all sorts of features to help this (and some do). This was a very hot topic right then.

On a bit of a lark, a colleague (Brian Kernighan) and I wrote a paper that took somewhat of a counter view. I.e., that it helped more to build tools & components that could be easily found and reused, as the most reliable software was that which you didn't have to write fresh.

We got back referee's reports:1)Yes, great.2)Yes, great.3)No, nothing new.4)OK, but irrelevant to this conference.so:reject.

We laughed a bit.

Someone attended the conference, and oddly, exactly this exact topic was raised and discussed fairly often.

A bit later, a friend of Brian's was looking for an article for Software-Practice & Experience, so Brian offered him this one, published in 1979.

Then, a bit later, someone from IEEE Computer wanted an updated version for there, which got published in 1981 as "The UNIX Programming Environment" which was referenced moderately often for decades (although Google Scholar gets confused about it).

John: I certainly have my own stories about peer-review failures as well. Most of the time, though, papers are better after being shown around -- whether in a formal peer review or passing out the next to final draft to colleagues. Actually, I have to limit the improvement to the content and parts of presentation. The writing style usually gets crippled into a very dead, passive, form.

Oops, I didn't mean this as an example of peer-review failure:we *didn't* write the sort of paper they were looking for (and we knew it. I said it was sort of a lark. :-)

I just meant that one can sometimes get back confusing/contradictory advice, and of course, as it happened, the paper ended up with way wider exposure than if it had been accepted at the conference.

To me, real failures are those that let absolute junk through too easily. Missing a good paper can happen for a lot of reasons, and usually it gets out there sooner or later, which was the other point. I.e., it's assymetric. Acceptance doesn't mean it's right, but repeated non-acceptance is useful data.

naught101: The folks likely to think badly of me, will do so regardless of whether that graphic remains up or not. The graphic itself, given responses here, doesn't communicate my message well. Indeed, it prevents my message from being communicated at all. A graphic that is doing active harm, I'll pull down. In science, we're supposed to learn from what happens in the real world. I learned that my graphic was of negative use.

Those who might complain about me 'hiding' something, would complain about something else with it up. So they're not interesting. People who encounter the old post as it now stands, I hope, will get something of the sense of what I mean. The graphic itself is irrelevant to the main point -- for any given question, there are groups more likely, and less likely, to have studied the topic. They are, then, more likely, or less likely, to be good sources to turn to for more information about the topic. It'll depend, very sensitively, on what, exactly, the question you have in mind is.

Welcome

I'll be trying what seems to be an unusual approach in blogs -- writing to be inclusive of students in middle school and jr. high*, as well as teachers and parents (whether for their own information or to help their children). To that end, comments will have to pass a stricter standard than I'd apply for an all-comers site. It shouldn't be onerous, just keep to the topic and use clean language.

I expect it to be fun for all, however, as you really can get quite far in understanding the world, even climate, by understanding this sort of fundamental. If I get too much less fundamental, let me know where I went astray.

* Ok, I concede that not many middle school students will get everything. Even a fair number of adults will find some parts hard to follow. Still, some middle school kids will have fun. And almost everyone will follow a number of posts just fine.

Please see the comment policy for details. And the link policy for details about that. The latter is more open than you might expect.

About Me

In my day job I work on the oceanography, meteorology, climatology, glaciology end of my science interests, but I'm interested in everything, science or not. So I've also been on stage in a production of Comedy of Errors, run an ultramarathon, and been to Epidaurus, Greece, to see a production of Euripides' Iphigenia among the Taurians
Prior to starting the current job, I was a post-doc in oceanography in the UCAR ocean modelling program, and earned my doctorate from the Department of the Geophysical Sciences at the University of Chicago (1989). My undergraduate degree involved Applied Math, Engineering, Astrophysics, and Glaciology.
Of course I don't speak for my employer, whoever that may be.