29 March 2016

I recently discovered the TV show Skin Wars on Hulu. It’s one of those, “Make some cool really fast” competition shows like FaceOff, Top Chef, or Project Runway. In this case, the topic is body painting. The show’s had two seasons, with a third starting up next month. But this was the last winning entry that clinched season two for Lana!

23 March 2016

As a professor, you write recommendations for student regularly. One came in this morning, so I started filling it out. First questions were pretty standard, asking you to rate students compared to others you’ce interacted with.

One struck me as oddly worded: something like, “Student will do the right thing when no one is looking.” Um... if this student does these things when no one is looking, how can anyone know?!

But then I got to the question (click to enlarge):

And you can pick “All that apply”! “I’ll be Secretary of State and run a pro sports team!” “I’ll be Fortune 500 CEO and an astronaut, becoming the first entrepreneur in space!” These are the kind of career ambitions you might have when you’re eight. But for evaluating university students? No. These achievements are so rare and capricious that it makes no sense to ask someone to say that a student is likely to achieve these.

I thought, “This is the craziest, least realistic question I have ever had to do on a scholarship recommendation.”

Then I saw the next question, and I had to throw out my previous record holder for craziest, least realistic question I’ve had to answer for a scholarship recommendation.

There are some people in that list I would use if I wanted to diss a student. Some of the politicians come to mind.

The way this recommendation was going, I expected the next question to be, “What is this student’s mutant superpower?”

These questions are dumb. I get that a scholarship organization wants students to aim high and achieve and all, but geez. These are not realistic career expectations. And it’s not a helpful way for professors to try to characterize what they think the student’s good and bad qualities are.

Similarly, a New York Times article on pre-prints that appeared several weeks later focused on the Nobel laureates. I admit I got annoyed by tweets and articles about Nobel winners and Ivy League professors and HHMI labs and established professors at major research universities using pre-prints. I wasn’t the only one:

I wish this article didn’t erase the biologists who have been posting to arXiv for years.

If pre-prints are going to become the norm in biology, they can’t just work for the established superstars. Pre-prints have to have benefits for the rank and file out there. It can’t just be “more work.”

For example, I think one of the reasons PLOS ONE was a success was that it provided benefits, not just for superstars, but for regular scientists doing solid but incremental work: it provided a venue that didn’t screen for importance. That was a huge change. In contrast, new journals that try to cater to science superstars by publishing “high impact” science (PLOS Biology or eLife and Science Advances), while not failures, have not taken off in the same was that PLOS ONE did.

I decided I would try an experiment.

I don’t do the most glamorous scientific research, but I do have a higher than average social media profile for a scientist. (I have more Twitter followers than my university does.) So I thought, “Let’s put up a pre-print on biorXiv and see if anyone comments.”

I spent the better part of a morning (Thursday, 25 February 2016) uploading the pre-print. Since I had seen people whinging about “put your figures in the main body of the text, not at the end of the paper,” I had to spend time reformatting the manuscript so it looked kind of nice. I also made sure my Twitter handle was on the front page, to make it easy for people to let me know they’d seen my paper.

I was a little annoyed that I had to go through one of those clunky manuscript submission systems that I do for journals. I had to take a few stabs at converting the document into a PDF. biorXiv has a built-in PDF conversion built into it, but the results were unsatisfactory. There were several image conversion problems. One picture looked like it came out of a printer running low on ink. Lines on some of the graphs looked like they had been dipped in chocolate. Converting the file to PDF on my desktop looked much better. I uploaded that, only to find that even that had to go through a PDF conversion process that chewed up some more time.

biorXiv preprints are vetted by actual people, so I waited a few hours (three hours and thirty-nine minutes) to get back a confirmation email. It was up on biorXiv within a couple of hours. All in all, pretty quick.

I updated the “Non-peer reviewed papers” section of my home page. I put a little “New!” icon next to the link and everything. But I didn’t go out and promote it. I deliberately didn’t check it on biorXiv to ensure that my own views wouldn’t get counted. Because the point was to see whether anyone would notice without active promotion.

I waited. I wasn’t sure how long to wait.

After a day, my article had an Altmetric score of 1. biorXivpreprints and three other accounts that looked like bots tweeted the paper, apparently because they trawl and tweet every biorXiv paper. (By the way, “Bat_papers” Twitter account? There are no bats in my paper.) The four Twitter accounts combined had fewer followers than me. Looking at the Altmetrics page did remind me, however, that I need to make the title of my paper more Twitter friendly. It was way longer than 140 characters.

Four days later (29 February 2016), I got a Google Scholar alert in my inbox alerting me to the presence of my pre-print. Again, this was an automated response. That was another way people could have found my paper.

Three weeks has gone by now. And that’s all the action I’ve seen on the pre-print. Even with a New York Times article brought attention to pre-prints and biorXiv, nobody noticed mine. Instead, the attention is focused on the “established labs,” as Arturo Casadevall calls them. The cool kids.

I learned that for rank and file biologists, posting work on pre-prints is probably just another task to do whose tangible rewards compared to a journal article are “few to none.” Like Kim Kardashian posting a selfie, pre-prints will probably only get attention if a person who is already famous does it.

Update, 18 March 2016: This post has been getting quite a bit of interest (thank you!), and I think as a result, the Altmetric score on the article reference herein has jumped from 1 to 11 (though mostly due to being included in this blog post).

08 March 2016

The National Science Foundation just announced changes to its Graduate Research Fellowship Program (GRFP) that limits the number of applications for grad students to one. The NSF lists several reasons for this:

1) result in a higher success rate for GRFP applicants

You know how else you could increase success rate? Give more awards. But the political reality is that the NSF budget is stagnant, so the best they can do is increase in success rate at the cost of limiting chances to apply.

2) increase the diversity of the total pool of individuals and of institutions from which applications may come through an increase in the number of individuals applying before they are admitted into graduate programs,

You know what else could increase the diversity of applicants? Instead of changing the applicants, change the reviewers. Change the criterion those new reviewers are judging the awards by. Let’s not forget that in the past, GRFPs have been criticized for giving awards to doctoral students in a limited number of institutions. If the review process is biased, tweaks to the applicant pool won’t create more diversity, because they’ll be going through the same filter.

And just because I run a master’s program, let me call out that in the past, only 3.5% of awards went to students in master’s programs (page 3, footnote 3 in this report). Why do you hate master’s students, NSF?!

Shifting the emphasis of @NSF GRFPs to undergrads will increase representation because it will recruit people. If you give a GRFP to someone already in grad school, odds are they would be successful anyway. GRFPs can get undergrads into good labs.

This is a good point, but if undergraduates are still competing against graduates, and there’s still a bias towards the advanced doctoral students as the “safer bet,” the change in the applicant pool might not achieve the desired result in diversity. This might be an argument for splitting the program: one for undergraduates transitioning to graduate programs, one for graduate students in programs.

I am glad to see this listed as a reason, because it is honest. It’s a lot of hard work to review applications, and you have to keep it to a reasonable level. I think it might be even more honest if it was listed as reason #1 instead of reason #3.

PLOS ONE deserves credit for not dawdling on this. A concern was identified, and by the next day, they arrived at a decision.

PLOS ONE loses the points they just earned, however, for the lack of transparency in their retraction notice.

Our internal review and the advice we have received have confirmed the concerns about the article and revealed that the peer review process did not adequately evaluate several aspects of the work.

Which means... what, exactly? PLOS ONE claims it will publish any technically sound science. What’s the basis for retraction, then? What about the paper is not technically sound? You can’t find it in the retraction notice. I haven’t seen any criticisms about the paper’s scientific content beyond, “It’s not news; we know hands grab things.” But no comments like, “The wrong statistic was used,” or “The authors didn’t account for this uncontrolled variable.”

The retraction notice says the editorial process wasn’t very good, but that should not be the basis for a retraction. If an editor screwed up, a competent paper shouldn’t suffer. A good result shouldn’t be punished for a bad process.

PLOS staffer David Knutsen wrote more for this blog post, but we still only get vague references to:

(T)he quality of the paper in general, the rationale of the study and its presentation relative to existing literature(.)

It still doesn’t answer what technical flaw the paper had. PLOS ONE comes out looking like it might have swatted a flay with a sledgehammer, and facing accusations of racism.

This situation is not a bug in the PLOS ONE system: it’s a feature. Ph.D. Comics published this cartoon about PLOS ONEback in 2009, which takes on new relevance:

“Let the world decide what is important.” PLOS ONE was designed from the get go to:

Have a light editorial touch. “Is it technically competent? Then publish.” The high volume of papers (PLOS ONE is the biggest single scientific journal in the world) pretty much ensures you cannot have the same level of editing that happens at other journals.

Encourage post-publication peer review.

When you combine those two things, things like the “hand of god” paper are practically inevitable. The question becomes whether the benefits of light editing (authors having more direct and potentially faster communication of what they want) outweighs the potential downsides (disproportionate criticism for obvious things that could have been fixed).

02 March 2016

Back on January 5, PLOS ONE published an Intelligent Design paper containing numerous references to “the Creator.” Judging from the consistent use of capitals, this would mean the Judaeo-Christian God.

I have no idea how this went unnoticed by science social media. The paper didn’t bury this, but announced it right in the abstract:

The explicit functional link indicates that the biomechanical characteristic of tendinous connective architecture between muscles and articulations is the proper design by the Creator to perform a multitude of daily tasks in a comfortable way.

The Introduction marks it as an Intelligent Design paper:

Hand coordination should indicate the mystery of the Creator’s invention.

And again in the discussion:

In conclusion, our study can improve the understanding of the human hand and confirm that the mechanical architecture is the proper design by the Creator for dexterous performance of numerous functions following the evolutionary remodeling of the ancestral hand for millions of years.

Judging from the rest of the text,the paper probably has other credible information, which is probably why PLOS ONE published it. PLOS ONE reviews for technical competence, not importance.

From my point of view, these three sentences should have been removed as a condition of publication. Random references to a “Creator” do not advance the argument of the paper. It is not at all clear what predictions flow from the “Creator” hypothesis. The authors do not support the “Creator” hypothesis with relevant literature from the biomechanical field. In short, references to a “Creator” mark the paper as not competent science, and thus unfit for publication under PLOS ONE’s standards.

Update: Oh yeah, this story is now burning up my Twitter feed. PLOS ONEtweeted that they are looking into this paper.

Update: Grant pointed out that even one of the paper’s own authors doesn’t seem to buy into their references to “the Creator”:

Update: Terry McGlynn notes that the lab page of one of the authors doesn’t appear to have any overt religious overtones. Several people are suggesting that this is a mistranslation. I’m not convinced (nor are others). Even if it is “just” a mistranslation, the journal doesn’t come out looking much better, because it highlights weak editing.