Tuesday, April 17, 2012

Dysfunctional Science

Carl Zimmer, one of the top science writers in the world, has written an article for the New York Times with the following provocative title: A Sharp Rise in Retractions Prompts Calls for Reform. It's partly about the rise in the rate of retractions1 in scientific journals. This is a serious problem and it's hard to figure out the underlying cause, in spite of the fact that many of the people who comment think they know the answer.

In tomorrow’s New York Times, I’ve got a long story about a growing sense among scientists that science itself is getting dysfunctional. For them, the clearest sign of this dysfunction is the growing rate of retractions of scientific papers, either due to errors or due to misconduct. But retractions represent just the most obvious symptom of deep institutional problems with how science is done these days–how projects get funded, how scientists find jobs, and how they keep labs up and running.

As usual, Carl's got it right. There's something wrong with science, or perhaps I should say there's something wrong with the biological sciences since Sean Carroll doesn't see the same problem in physics [Is Physics Among the Dysfunctional Sciences?].

1. The rate is about 0.04%. Compare this to the rate of fraudulent creationist publications, which is close to 100%.

Robert Gentry's work with polonium halos is an example of legitimate science by a creationist, which Gentry (incorrectly) cited as evidence for the young Earth that fits his particular creationist cosmology. Whether that counts as a "creationist publication" is debatable, since it appeared in a legitimate scientific journal. But the experimental results reported were sound. On a slightly different vein, the American Scientific Affiliation is an organization of Christians with scientific degrees. It publishes serious papers by real scientists who investigate different claims with respect to origins. Some of the scientists are creationists, some are not, but most of them appear to be sincere, rather than frauds. I admit that these are but the tip of a largely rotten iceberg.

For additional examples of ‘dysfunctional science’ see these articles: “Why Most Published Research Findings Are False” (http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124) and “Lies, Damned Lies, and Medical Science” (http://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-and-medical-science/8269/).

However, the number of ‘retractions’ and the significance of ‘lies’ are not endangering the progress of science. What might do that is: lack of basic knowledge and interest in fundamental principles and concepts; persistence of misleading dogmas; not to mention, the culture of using science primarily as a career vehicle (just look at the CVs of our distinguished scientists, which are so full of ‘titles’, ‘positions’ and ‘prizes’ that they make the infamous former Soviet generals, with their chest full of medals, look modest).

Anyway, why would scientists be different than the rest of the people?

Do you see this as a problem right across the biological sciences or do some disciplines see a greater number of retractions than others? If the problem is largely confined to one or two disciplines then this is arguably less worrying (relatively speaking) and potentially easier to deal with than one which encapsulates the whole of the biological sciences.

I think that retractions and fraud are the symptom of a larger problem. As I mentioned on the New York Times site, some disciplines are in worse shape than others when it comes to dysfunctional science. Evolutionary psychology is a good example of one of the worst cases.

However, all biological sciences seem to be suffering. I haven't yet encountered a biological scientist who thinks their discipline is in good shape.

Thanks for the response Larry. I'm a student of biology but I have to admit to not having heard a great deal from supervisors and colleagues in this regard. It's something I'll certainly look into when I move institutions later this year.

The peer review process is the problem here- granted I am not the first person to say this, nor do I intend to propose any solution. With that stated, I do not think the problem with peer review is solely because of the increase in submissions/publications - it has more to do with the lack of attention given to the philosophy of science in the research process of biological scientists.

It depends what you mean. If you include the papers on the Three Domain Hypothesis then there's a lot of bad science. A lot of the stuff about the demise of the tree of life is also looking very much like dysfunctional science.

And I still don't think the debates over various types of cladism are always examples of good science.

There was a time not long ago when the debates over various methods of tree building got totally out of control and some very silly claims were made.

This almost needed two discussion. 1 for fraud (which most of us comments have focused on so far) and the second for bad science. I think all fields have their share of bad science, but that is harder to quantify than the retractions / fraud part. Many in various fields are guilty of, or are currently promoting the bad science part, so is much harder to objectively discuss.

But my little anecdote - I work in mitochondrial biology (now focusing on mammlian mito and mito disease, but used to do metazoan phylogenetics using whole-genome datasets). Since mitos keep popping up in ageing, and more and more common diseases, more people are moving into the field (a good thing!) but few of them seem to have read any papers published before 2000. Many labs seem to be continuously re-inventing the wheel, but in the process are making a lot of mistakes that were already dealt with in the 1980's.

I wonder how much of the current bad science can be attributed to the lack on knowledge of the past work. I know that Sandwalk has touched on that problem before...

I wouldn't say there's a lot of bad science in systematics. There is of course some. There always will be. But I see no sign that the field as a whole, or on average, is pathological. A few bad papers here and there don't cause a significant problem, nor do I see any increase in the proportion of bad papers from any time in the past. In fact, we are finally getting a good handle on phylogeny. What's to complain about?

Fights between lumpers and splitters are an odd thing to bring up, since the common position these days is that it's arbitrary. If you want a good fight on that subject, go back a hundred years.

My inner cynic is not at all surprised by this. It's no reflection on biology as a science, it's just a reflection on the motivations of a fraud.

The big motivation, saith my inner cynic, would be fraud for gain. Money from selling a snake oil medication, patents on processes which produce things which don't actually work, fame and authority (even if temporary) based on publications. Biology has a vulnerability in that a study establishing a conclusion may be too difficult to reproduce quickly to stop the fraud. If I publish a study showing that some random chemical, given to 5-year-olds, will make them smarter at the age of 20, that's 15 years before the study can be definitively discredited, and at least 5 before it can be challenged. That's a lot of time to manufacture and sell snake oil. Whereas with physics, most experiments can be duplicated in the same facility by different people fairly quickly.

And, adds the inner cynic, ideologues don't get hot and bothered about physics. There's nobody thumping their holy book and insisting that the gravitational constant is higher than that currently used, but there are all kinds of people who want to challenge biological science for religious or political reasons.

This may well be wrong (although my inner cynic compels me to add "but if it's wrong, the reason is probably even nastier") but I must confess that if I were going to commit scientific fraud, I would at least consider biology first.

But this paragraph really rings true right now. "But other forces are more pernicious. To survive professionally, scientists feel the need to publish as many papers as possible, and to get them into high-profile journals. And sometimes they cut corners or even commit misconduct to get there."

The "goal" of your own academic position is becoming harder to obtain. With the volume of papers published each day, the feeling that you will get caught may be smaller and smaller. If the return to risk ratio gets more tempting, more cheaters are inevitable.

While I think the issue of scientific fraud and the reasons it occurs needs to be addressed. I can't help but think 2 things when I first read this. One, how much of this increase is due to people actually getting caught (meaning more people have access to more publications, and are able to "fact check")? Two, The fact that these things are getting caught is a good thing. Science is self correcting (to some extent). There is too much competition out there to not be right, and there are plenty of scientists ready and willing to challenge your claims.

The last thing I must say is that there is so much new information being generated on a daily basis, I feel like it is easy for reviewers (of papers and grants) to give someone a free pass because they have published in high impact journals previously (thinking that If they put out solid work before, then this must also be good science). Is the answer to this a totally anonymous peer review? I don't know, but I think it would help.

In addition, I get the impression that the drive to support your own salary as an investigator and the salary of your post-docs and graduate students leads to the development of large labs and many projects. This does not allow the primary investigator to focus on good work (unless they are super-human). The focus becomes publish, publish, publish (get more money, money, money). I personally dislike the idea of very large labs. I get the feeling that this causes a separation of the PI from the actual work and in some cases a feeling of apathy about the quality as long as something is published (so your grant can be renewed).

-Science newb, optimistic graduate student (4th year in), feeling the pressure to publish (even though I have a few publications to my name, I still worry) BW

In response to "One, how much of this increase is due to people actually getting caught (meaning more people have access to more publications, and are able to "fact check")? "

If we are just catching more fraud, that is a good thing. But at the same time I really doubt it.

I am in that "senior postdoc looking to transition to independent" phase. Many of my peers seem very unhealthily obsessed with getting a Science/Nature/Cell paper as a minimum requirement to get the next stage. If that is your perceived "minimum level" on your CV, doesn't it seem a little desperate? Don't you think that they may be over-optimistically interpreting that gel that is the key observation that turns the story from a common one to a S/N/C "quality" one?

The incentives are too great. Look at the "Arsenic Life" flop. She even named the strains with her career goals in mind.

Second - with the ease of access has come the flood of papers that you need to read. How may papers a month do you read? I use pubmed to send me a keyword search of abstracts each month. Last month I got 188 abstracts, and I flagged 91 of them to read. In reality, I will read maybe 10-20 carefully enough to have any chance of noticing something. Then there are the papers I find by the other methods that I need to read...

“The peer review process is the problem here…” (Thummim)“Many in various fields are guilty of, or are currently promoting the bad science part, so is much harder to objectively discuss.” (Anonymous/The other Jim)“I think that retractions and fraud are the symptom of a larger problem.” (Laurence A. Moran)“I must confess that if I were going to commit scientific fraud, I would at least consider biology first” (The Vicar)

Many of these problems might have to do with the fact that many scientists, like other competitive professionals, are going to adapt to whatever the nebulous entity called “system,” (i.e. our professional environment) requires. And, apparently, in highly active fields of science, such as Biology, we are integrated in a “science pyramid scheme” described (i.e. disguised) as a “Ponzi Sci-fi” by Surelyyourjoking in a comment to article “Lies, Damned Lies, and Medical Science” (http://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-and-medical-science/8269/).

One more thing hit me during a chat with some friends. Look at the "faster than light neutrons" story. The CERN people approached the story with caution and skepticism, and and were very open to the chance that they were wrong. A well established principle in physics was being challenged, if they were correct.

Now imagine someone in biosciences who has some preliminary data that "the central dogma of molecular biology" is wrong? Will they behave like CERN, or more like this?http://sandwalk.blogspot.de/2012/01/mind-of-james-shapiro.html?showComment=1328742052040#c9077880841917658438

It depends on who they are, doesn't it? Anyway, there's a major difference. We had good reason to believe that faster than light neutrinos (not neutrons) were impossible. But it's perfectly conceivable that there might be some mechanism, somewhere, that might contradict the central dogma, even if it were correctly stated. I can't offhand think how such a mechanism would work, but it isn't physically impossible.

I think, Crick's quote seems to be very clear on what he meant by Central Dogma: “transfer of information from nucleic acid to nucleic acid, or from nucleic acid to protein may be possible, but transfer from protein to protein, or from protein to nucleic acid is impossible”(emphasis mine).

As shown also in Fig 1, Crick had no conceptual problem envisioning transfer of information among nucleic acids and from nucleic acids to protein, and that includes transfer from DNA to protein (which, yes, could be accomplished with some ingenious engineering of ribosomes; anyone wants to jump on that?, surely it will make gazillions of front pages!).

Crick’s problem was with transfer of information from protein to protein, or from protein to nucleic acid, which he considered taboo. And here is where the prion hypothesis comes into play. See, for example, these articles: “A scientific revolution? The prion anomaly may challenge the central dogma of molecular biology” (http://www.ncbi.nlm.nih.gov/pubmed/16065057) and “Prion diseases and the central dogma of molecular biology” (http://www.ncbi.nlm.nih.gov/pubmed/10390630)

It is important to mention here that in regard to the term ‘information’ used in the context of Central Dogma, it makes sense that Crick referred to ‘coding information,’ as in ‘hereditary information’ not as in ‘biological information’ which all proteins and nucleic acids use during their endless interactions.

Tachyons "explain" faster than light travel...of tachyons. If they explain anything. Neutrinos aren't tachyons. Sorry, but it's true that neutrinos traveling faster than light is theoretically impossible. That's what all the furor was about: a supposedly impossible result that, if confirmed, would have overturned physics.

I just read Larry’s article “The Central Dogma of Molecular Biology” (see ‘Essays and Articles’), which is one of the best pieces I read on the subject.

I think, Crick’s quote seems to be crystal clear on what he meant by Central Dogma: “transfer of information from nucleic acid to nucleic acid, or from nucleic acid to protein may be possible, but transfer from protein to protein, or from protein to nucleic acid is impossible”.

As shown also in Fig 1 of Larry’s article, Crick had no conceptual problem envisioning the transfer of information between DNA and RNA, and from nucleic acids to protein.

Recent Comments

Principles of Biochemistry 5th edition

Disclaimer

Some readers of this blog may be under the impression that my personal opinions represent the official position of Canada, the Province of Ontario, the City of Toronto, the University of Toronto, the Faculty of Medicine, or the Department of Biochemistry. All of these institutions, plus every single one of my colleagues, students, friends, and relatives, want you to know that I do not speak for them. You should also know that they don't speak for me.

Superstition

Quotations

The old argument of design in nature, as given by Paley, which formerlyseemed to me to be so conclusive, fails, now that the law of natural selection has been discovered. We can no longer argue that, for instance, the beautiful hinge of a bivalve shell must have been made by an intelligent being, like the hinge of a door by man. There seems to be no more design in the variability of organic beings and in the action of natural selection, than in the course which the wind blows.

Charles Darwin (c1880)Although I am fully convinced of the truth of the views given in this volume, I by no means expect to convince experienced naturalists whose minds are stocked with a multitude of facts all viewed, during a long course of years, from a point of view directly opposite to mine. It is so easy to hide our ignorance under such expressions as "plan of creation," "unity of design," etc., and to think that we give an explanation when we only restate a fact. Any one whose disposition leads him to attach more weight to unexplained difficulties than to the explanation of a certain number of facts will certainly reject the theory.

Charles Darwin (1859)Science reveals where religion conceals. Where religion purports to explain, it actually resorts to tautology. To assert that "God did it" is no more than an admission of ignorance dressed deceitfully as an explanation...

Quotations

I have championed contingency, and will continue to do so, because its large realm and legitimate claims have been so poorly attended by evolutionary scientists who cannot discern the beat of this different drummer while their brains and ears remain tuned to only the sounds of general theory.

The essence of Darwinism lies in its claim that natural selection creates the fit. Variation is ubiquitous and random in direction. It supplies raw material only. Natural selection directs the course of evolutionary change.

Rudyard Kipling asked how the leopard got its spots, the rhino its wrinkled skin. He called his answers "just-so stories." When evolutionists try to explain form and behavior, they also tell just-so stories—and the agent is natural selection. Virtuosity in invention replaces testability as the criterion for acceptance.

The first commandment for all versions of NOMA might be summarized by stating: "Thou shalt not mix the magisteria by claiming that God directly ordains important events in the history of nature by special interference knowable only through revelation and not accessible to science." In common parlance, we refer to such special interference as "miracle"—operationally defined as a unique and temporary suspension of natural law to reorder the facts of nature by divine fiat.

Quotations

My own view is that conclusions about the evolution of human behavior should be based on research at least as rigorous as that used in studying nonhuman animals. And if you read the animal behavior journals, you'll see that this requirement sets the bar pretty high, so that many assertions about evolutionary psychology sink without a trace.

Jerry Coyne
Why Evolution Is TrueI once made the remark that two things disappeared in 1990: one was communism, the other was biochemistry and that only one of them should be allowed to come back.

Sydney Brenner
TIBS Dec. 2000
It is naïve to think that if a species' environment changes the species must adapt or else become extinct.... Just as a changed environment need not set in motion selection for new adaptations, new adaptations may evolve in an unchanging environment if new mutations arise that are superior to any pre-existing variations

Douglas Futuyma
One of the most frightening things in the Western world, and in this country in particular, is the number of people who believe in things that are scientifically false. If someone tells me that the earth is less than 10,000 years old, in my opinion he should see a psychiatrist.

Francis Crick
There will be no difficulty in computers being adapted to biology. There will be luddites. But they will be buried.

Sydney Brenner
An atheist before Darwin could have said, following Hume: 'I have no explanation for complex biological design. All I know is that God isn't a good explanation, so we must wait and hope that somebody comes up with a better one.' I can't help feeling that such a position, though logically sound, would have left one feeling pretty unsatisfied, and that although atheism might have been logically tenable before Darwin, Darwin made it possible to be an intellectually fulfilled atheist

Richard Dawkins
Another curious aspect of the theory of evolution is that everybody thinks he understand it. I mean philosophers, social scientists, and so on. While in fact very few people understand it, actually as it stands, even as it stood when Darwin expressed it, and even less as we now may be able to understand it in biology.

Jacques Monod
The false view of evolution as a process of global optimizing has been applied literally by engineers who, taken in by a mistaken metaphor, have attempted to find globally optimal solutions to design problems by writing programs that model evolution by natural selection.