Friday, September 18, 2009

How Bad Papers Get Published in Good Journals

Donald Williamson used to be a marine biologist. He has some strange ideas about evolution. He thinks, for example, that the reason why butterflies have distinct larval and adult stages is because they arose from the fusion of two separate species—a larva-like species and a butterfly-like species.

Lots of us have crazy ideas but it's a real challenge to get them published in the peer-reviewed literature, and that's how it should be. The reason why science is so successful as a way of knowing is, in part, because it's dominated by skepticism and a requirement for evidence-based rational thought. The system sometimes impedes the acceptance of real innovative ideas but not for long. What is does do successfully, however, is weed out the kooks. But even that doesn't work all the time.

The Proceedings of the National Academy of Sciences (PNAS) is a prestigious journal that's run by the National Academy of Sciences. Once elected to the academy, members have some special privileges when it comes to publishing in the journal. They can "contribute" one of their own papers, in which case they can have a great deal of influence on choosing reviewers, or they can "communicate" the paper of a friend or colleague, in which case they choose the reviewers and send the reviews to the editor of the journal.

It's easy to see the potential for abuse but the remarkable thing is that the process actually works quite well. The quality of papers "contributed" or "communicated" is, in general, no worse than that of papers published in other front-line journals. Two of my own papers were "contributed by" my former Ph.D. supervisor and the process was a rigorous as any other.

But when the system fails, it fails spectacularly.

Lynn Margulis is a member of the National Academy. She "communicated" a paper by Donald Williamson on his strange idea about butterfly evolution (Williamson 2009). That's when the excrement hit the fan.

The editor of PNAS, Randy Schekman, has announced that the "communicated by" option for members will end in July 2010 [PNAS Nixes Special Privileges for (Most) Papers] The Science article reporting on this change in policy leaves little doubt about what prompted it.

An example of alleged gamesmanship popped up online 28 August in PNAS. Lynn Margulis, the noted biologist at the University of Massachusetts, Amherst, communicated a paper by Donald Williamson, a retired marine biologist in the United Kingdom. In it, Williamson promoted his longheld, intriguing—and, say most other biologists, almost certainly misguided—theory about the origins of caterpillars and butterflies. Current biological theory argues that they were always a single species and that each stage evolved via natural selection. Williamson argues instead that two distinct species (one caterpillar-like, one butterfly-like) somehow fused into a hybrid way back when. One species' sperm must have fertilized the other's eggs, transferring genes laterally across species in a non-Mendelian fashion.

Margulis was unavailable for comment, but Williamson says, "Lynn Margulis is prepared to put her name and reputation on the line" to prove that "genome mergers" occur in evolution, a position his paper supports. He also says he knows that Margulis sent his paper to a half-dozen academy reviewers. Williamson says that he thinks they were all positive reviews, but Margulis told Scientific American last week that she canvassed six or seven reviewers to find the two positive reviews necessary to push the paper through.

Shame on you, Lynn Margulis, You've made some outstanding contributions to biology over the years—endosymbioisis being the best example—but it's time to hang up your hat and retire gracefully. Your latest ideas are totally wacky and your inability to distinguish between science and fantasy—as evidenced in your promotion of the Williamson paper—is an embarrassment to those of us who, for several decades, have been holding you up as an example of a successful and creative scientist.

Of course when ultradarwinian hogwash is published in PNAS (which is often), no one says a peep... Larry just screams when everyone else does.I still have to read the williamson paper...then I'll have an opinion.

Margulis is famous as a Maverick Who Turned Out To Be Right. Unfortunately, she seems to be trying to continue playing the same schtick (on her own account, or by proxy), apparently unaware that, in fact, most mavericks are wrong. Reality can be such a bitch.

Q. Okay. Now you stated on Monday that Darwin's Black Box was also peer reviewed, right?A. That's correct.Q. You would agree that peer review for a book published in the Trade Press is not as rigorous as the peer review process for the leading scientific journals, would you?A. No, I would not agree with that. The review process that the book went through is analogous to peer review in the literature, because the manuscript was sent out to scientists for their careful reading.Furthermore, the book was sent out to more scientists than typically review a manuscript. In the typical case, a manuscript that's going to -- that is submitted for a publication in a scientific journal is reviewed just by two reviewers. My book was sent out to five reviewers.

but who are you to tell her she should stop doing research for good because she supports one wacky idea?

I don't know if I'd go as far as telling her to retire, but it all boils down to this: either you perform evidence-based, ultimately hypothesis driven science, or you don't. There's no room for ambiguity here. You can't disregard evidence when it's no convenient - that's scientific fraud.

The Williamson paper is awful, it ignores everything we know about speciation and developmental genetics (I blogged about it here). If a graduate student came up with this idea, and their supervisor knew anything about the subject matter, they'd end up getting a lecture about what science is. It's perfectly reasonable to call such a view being promoted by senior scientists utterly shameful.

The article is fun, but I am not convinced. I think it is scientific enough, in the sense that clear predictions are made: 1) you shuld be able to hybridize onychophorans and insects 2) You should be able to track down an onychophoran genome besides the insect genome.

The lack of any new data to support 1 or 2 is what is disappointing (I wondered about those 2 just from the title of the paper) , and the kind of thing a normal person like me would need to publish in PNAS. But I'm all for fun. You usually read fun things like this in the smaller journals only, though.

It is well-documented that hybrids crosses between different species can produce new species with hybrid genomes (as is the case with allopolyploidy). This frequent phenomenon certainly does not fit speciation through typically "darwinian' gradualism, but is extensively documented to have occurred naturally (and experimentally) in many known species of plants and animals.

The question is to the limits of phylogenetic distance for species hybridization, and experimentation is the way to prove it. I think the most impressive I've heard are crosses at genus family and even among snakes, separated by tens of millions of years, but never anything like among phyla.

Williamson himself argued he had gotten an urchin larva from fertilizing a tunicate egg with urchine sperm, but many controls were missing.

http://8e.devbio.com/article.php?ch=23&id=228

Since he takes time to talk about many thing except this experiment, I guess there was some artifact involved? contaminated seawater?

I don't know if anyone has ever conclusively explained what happened in Williamson's tunicate X echinoid experiment (after all, to do that you'd probably have to have been there at the time). I believe the most popular explanation is that he was simply incorrect about the sea urchins being solely male, which Williamson of course denies vehemently.

If I may reproduce Williamson's (2001) own summary of the results:

Eggs of the ascidian Ascidia mentula have been fertilised with sperm of the sea-urchin Echinus esculentus to give plutei, i.e. wholly paternal larvae. The majority of the fully developed plutei did not develop Echinus rudiments, but retracted their larval arms to produce spheroids, each with an adhesive disk. There is no corresponding stage in the development of Echinus x Echinus. Some spheroids lived up to 90 days from hatching, but none developed further. A minority of the plutei developed Echinus rudiments and metamorphosed into urchins in 37-50 days. Four years later, the three surviving urchins all produced eggs, which were fertilised with wild Echinus sperm to give apparently normal pluteus larvae. Nucleotide sequences were investigated for a ribosomal and a mitochondrial gene extracted from tube-feet of the urchins. These sequences were wholly paternal, i.e. resembling Echinus.

In other words, Williamson claims that the resulting sea urchins must have been hybrids, because otherwise where did the eggs come from? Whereas other researchers have simply replied that if it quacks like a duck...

Margulis should know better than using her name to further something that's that far out of tested scientific thinking... endosymbiosis was out there at the time, but plausible. This would require strong documentation of another selective process, and lacking that is an assertion floating in thin air.

Bayesian> Behe got that part right, but what he ignored was the fundamental difference between reviewers for his book and reviewers for peer reviewed journals: you don't get to choose your reviewers based on how likely they are to agree with your paper.

you don't get to choose your reviewers based on how likely they are to agree with your paper

Of course you do: you send the MS to an editor friend of yours or to a buddy (or a former supervisor) to "contribute". And, just in case, you disqualify several people who you suspect are not likely to agree. All of this is practically a default and everyone is doing it precisely because it decreases the likelihood of negative review.

I like Lynn Margulis, this world is too full of unoriginal, narrow-minded neodarwinians, she doesn't buy it and she pokes their eye all the fuckin time... I love it! So, she decided to use her PNAS powers to introduce a rather speculative paper? That happens on PNAS all the time...all this fuss is just because its Lynn Margulis, who excells at pissing off the (still abundant) neodarwinians.

Of course, I'll take Lynn margulis over 100 Larry Morans, any day. I think by and large she is absolutely right: Symbiosis is a huge, underestimated factor of evolution. The cases demosntrating this are spectacular and ever increasing

So, she decided to use her PNAS powers to introduce a rather speculative paper?

There's no point on 'speculating' about things that we know are incorrect. As Jerry Coyne pointed out in his blog post about this paper, this hypothesis makes easily testable predictions that we already know are false. The genomes of insects should have large complements of their genes with obviously different evolutionary signatures (i.e., the trees generated from 'larval' genes should not agree with those generated from the other genes - assuming we attempt to draw a deep-enough time phylogeny). Furthermore, we know quite a bit about speciation genetics and this ignores all of it (there was a successful mating between 2 phyla?).

You can be as underdog-ish and contrarian as you like, but supporting crap science just because it's different isn't helping anyone.

"Furthermore, we know quite a bit about speciation genetics and this ignores all of it"

Aha...so, what is your opinion about the origin of new species by species hybridization? Species with combined genomes exist. Allopolyploidy. Do you think species hybridization is a very darwinian way of speciating?

I agree that the phylogenetic distance is large and seems very improbable. But some still pretty large phylogenetic distances can be breached in hybrid crosses (I understand snakes that diverged in the cretaceous can hybridize, as well as cetaceans of different genus)

Since you know so much about the genetics of speciation, please enlighten us on how is it that you establish just how much phylogenetic distance is "too much".

Aha...so, what is your opinion about the origin of new species by species hybridization? Species with combined genomes exist.

I will refer you back to the excellent treatise on speciation genetics by Jerry Coyne and Allen Orr called Speciation (Sunderland; 2004).

Yes, speciation by hybridogenesis is quite common in plants, but not so much in animals. I believe that certain closely related species of toads are thought to have arisen by hybridization, but I'm not up on the literature.

In terms of reasonable divergence times, I think that we can make educated guesses based on the numerous studies that have been conducted in insects. Reproductive isolation is already firmly established in species that have been diverged by about a million years in Drosophila (e.g., crossing D. simulans with D. sechellia). Males produced by the crosses are sterile, females are fertile, but can show abnormalities. By 2.5 million years, (e.g., D. melanogaster and D. simulans) all offspring are sterile and show massive developmental defects.

In these examples, we're talking about closely related species of the same genus. In the Williamson paper, we're supposed to be considering the mating between an Onychophoran, which modern phylogenetics clusters near Crustaceans at the base of the arthropod lineage, and an insect. These two species had to be at least diverged enough such that they had radically different morphologies and developmental body plans. So yes, I think that it's reasonable to ask for a mechanism of how this could have occurred.

Furthermore, why Butterflies specifically? Wouldn't it be more parsimonious to assume that all holometabolous insects acquired their larval stage in one hybridization (with subsequent losses in particular lineages)? It's difficult enough to imagine how the hybrid genome suddenly 'knew' to express some genes at one time and others at a different time and somehow had all of the mechanisms in place required to make that transition. Are we supposed to accept that this has happened repeatedly?

I don't buy Williamson's hypothesis, but I'd make you a bet that hybrids are possible over much larger timescales than you think. To begin with, "divergence time" does not equal "cannot hybridize by allopoplyploidy". Allopoliploidy occurs between clearly distinct species, with different sets of chromosomes and all.

I'm already on bibliographic hunt...so watch out...

Oh an you may want to check reptiles and butterflies for more animal examples of speciation by hybridization.

Buzz: ...but who are you to tell her she should stop doing research for good because she supports one wacky idea?

He already explained why he disapproves. It is not the "one wacky idea," it is the subversion of the peer-review process. And Margulis has thrown in with more than "one wacky idea." Marc has already mentioned "the Gaia crap." It is also worth mentioning her statements on HIV and AIDS.

I an not competent to make any judgments on the Williamson paper but it has been clear for some time that Prof. Margulis has turned into a whackjob in her advancing years, much like Linus Pauling, William Shockley, and J. Allen Hynek did. Among other things, she hob nobs with Holocaust deniers, rejects the theory that HIV causes AIDS, and is a 9/11 truther. But of course, that's good company for clowns like Vargas and Wagner.

There are more to fault than Lynn Margulis on this one. The major problem that should have prevented the paper being published was not its hypothesis but the fact that this particular hypothesis is easily tested using genomic sequencing of the sort that is routine nowadays. For a journal of the impact of PNAS one would expect the editors to at least ask the basic in sequence verification before they would allow such a wild claim to be made in their journal. Ideas that are against the grain should always have a chance of publication but they should be more than speculative, especially when a simple obvious experiment is all that is needed to eithere dismiss them or to give them at least some basis of plausibility.

Blaise Pascal said "Isn't there something disingenuous, improper, or at least reputation-killing to shop a paper around 6 or 7 reviewers to get 2 good reviews for publication?"The opposite case sometimes occurs as well. Our group had a paper reviewd by 'Science' a couple of years back when all the reviews were pretty favorable but asked for fairly minor revisions before acceptance. We did all these, sent it back and one of the original reviewers had an illness and so opted out of the rechecking of the reviews. The editors, rather than accepting the other reviewers verdict (both positive) decided to send the paper out to a new reviewer who was negative and thus the paper was eventually not accepted. Unfortunately our group is not such a big name and cannot kick up a fuss over this as big name groups can do and we were left with a rather bitter experience of big journal editorial practices.As in the case of PNAS a lot of the blame is not to be placed on the individual reviewers but on the editor who gives the go-ahead (or not) for publication.

What sigmund says is interesting too...sound sliek a very unfair and frustrating experience. What I can say is that, long run, where you publish is not relevant. Bad papers published in great journals are forgotten. Good papers are well-cited, no matter how small the journal that published them. It's all about content.The unfair thing is that those with the contacts to publish in the "great" journals can get funding more easily, and of course, not always deservedly so.

PNAS continues to publish "junk science", or "poorly substantiated" claims if one wants to be charitable. Sometimes reading articles feels like reading a marketing promo where the authors are using carefully crafted language to present their poor results in an exciting way. A recent article (past couple of years) claimed to have wonderful de novo results. Not only it did not stand up to scrutiny once you carefully read their supplementary data, in light of the supplementary data the main article was presenting a method that was not useful at all! (I am intentionally being oblique in order to avoid pointing fingers at any one person or group).

Communicated by was certainly part of the problem. I am glad they are nixing the "communicated by". In my opinion, this option led to a lot of "gaming the system" and "inside baseball promotion" - not the kind of thing you want to see in science.

I'd really like someone to do a survey of the articles published in the PNAS in the last 10 years to see how many of the "breakthrough" exciting technologies have held up to the test of time.

Lynn Margulis was a nut-job. She believed AIDS was really syphilis. For a microbiologist to make this claim is absolutely ridiculous. She should know better. Syphilis can be easily detected, and its symptoms do not match those of AIDS. As has been shown before with AIDS denial (Duesberg, Mullis), this position stems from homophobia. So no, this isn't Margulis being a "maverick." To be a maverick, you have to be *right*.

Neo-Darwinism has more evidence behind it than most scientific theories - even those in physics. (Still waiting on that theory of gravity.) Endosymbiosis may be a force of evolution, but hybridogenesis in animals is a fictitious exaggeration. These claims are easily testable - Williamson's claims have been debunked within the very parameters he set himself. Margulis could never show, genetically, that flagella originated from spirochete-to-archaea fusion, yet she insisted this was the case until the end.

Was she a great scientist? Early on, yes. But she took that ball and ran with it, and went into some very strange (and wrong) places.

And please stop comparing the sexism Margulis experienced to that of McClintock. McClintock supported her hypotheses with heavy, solid evidence. Margulis stopped short after her success with mitochondria/chloroplasts.

Laurence A. Moran

Larry Moran is a Professor in the Department of Biochemistry at the University of Toronto. You can contact him by looking up his email address on the University of Toronto website.

Sandwalk

The Sandwalk is the path behind the home of Charles Darwin where he used to walk every day, thinking about science. You can see the path in the woods in the upper left-hand corner of this image.

Disclaimer

Some readers of this blog may be under the impression that my personal opinions represent the official position of Canada, the Province of Ontario, the City of Toronto, the University of Toronto, the Faculty of Medicine, or the Department of Biochemistry. All of these institutions, plus every single one of my colleagues, students, friends, and relatives, want you to know that I do not speak for them. You should also know that they don't speak for me.

Subscribe to Sandwalk

Quotations

The old argument of design in nature, as given by Paley, which formerly seemed to me to be so conclusive, fails, now that the law of natural selection has been discovered. We can no longer argue that, for instance, the beautiful hinge of a bivalve shell must have been made by an intelligent being, like the hinge of a door by man. There seems to be no more design in the variability of organic beings and in the action of natural selection, than in the course which the wind blows.Charles Darwin (c1880)Although I am fully convinced of the truth of the views given in this volume, I by no means expect to convince experienced naturalists whose minds are stocked with a multitude of facts all viewed, during a long course of years, from a point of view directly opposite to mine. It is so easy to hide our ignorance under such expressions as "plan of creation," "unity of design," etc., and to think that we give an explanation when we only restate a fact. Any one whose disposition leads him to attach more weight to unexplained difficulties than to the explanation of a certain number of facts will certainly reject the theory.

Charles Darwin (1859)Science reveals where religion conceals. Where religion purports to explain, it actually resorts to tautology. To assert that "God did it" is no more than an admission of ignorance dressed deceitfully as an explanation...

Quotations

The world is not inhabited exclusively by fools, and when a subject arouses intense interest, as this one has, something other than semantics is usually at stake.
Stephen Jay Gould (1982)
I have championed contingency, and will continue to do so, because its large realm and legitimate claims have been so poorly attended by evolutionary scientists who cannot discern the beat of this different drummer while their brains and ears remain tuned to only the sounds of general theory.
Stephen Jay Gould (2002) p.1339
The essence of Darwinism lies in its claim that natural selection creates the fit. Variation is ubiquitous and random in direction. It supplies raw material only. Natural selection directs the course of evolutionary change.
Stephen Jay Gould (1977)
Rudyard Kipling asked how the leopard got its spots, the rhino its wrinkled skin. He called his answers "just-so stories." When evolutionists try to explain form and behavior, they also tell just-so stories—and the agent is natural selection. Virtuosity in invention replaces testability as the criterion for acceptance.
Stephen Jay Gould (1980)
Since 'change of gene frequencies in populations' is the 'official' definition of evolution, randomness has transgressed Darwin's border and asserted itself as an agent of evolutionary change.
Stephen Jay Gould (1983) p.335
The first commandment for all versions of NOMA might be summarized by stating: "Thou shalt not mix the magisteria by claiming that God directly ordains important events in the history of nature by special interference knowable only through revelation and not accessible to science." In common parlance, we refer to such special interference as "miracle"—operationally defined as a unique and temporary suspension of natural law to reorder the facts of nature by divine fiat.
Stephen Jay Gould (1999) p.84

Quotations

My own view is that conclusions about the evolution of human behavior should be based on research at least as rigorous as that used in studying nonhuman animals. And if you read the animal behavior journals, you'll see that this requirement sets the bar pretty high, so that many assertions about evolutionary psychology sink without a trace.

Jerry Coyne
Why Evolution Is TrueI once made the remark that two things disappeared in 1990: one was communism, the other was biochemistry and that only one of them should be allowed to come back.

Sydney Brenner
TIBS Dec. 2000
It is naïve to think that if a species' environment changes the species must adapt or else become extinct.... Just as a changed environment need not set in motion selection for new adaptations, new adaptations may evolve in an unchanging environment if new mutations arise that are superior to any pre-existing variations

Douglas Futuyma
One of the most frightening things in the Western world, and in this country in particular, is the number of people who believe in things that are scientifically false. If someone tells me that the earth is less than 10,000 years old, in my opinion he should see a psychiatrist.

Francis Crick
There will be no difficulty in computers being adapted to biology. There will be luddites. But they will be buried.

Sydney Brenner
An atheist before Darwin could have said, following Hume: 'I have no explanation for complex biological design. All I know is that God isn't a good explanation, so we must wait and hope that somebody comes up with a better one.' I can't help feeling that such a position, though logically sound, would have left one feeling pretty unsatisfied, and that although atheism might have been logically tenable before Darwin, Darwin made it possible to be an intellectually fulfilled atheist

Richard Dawkins
Another curious aspect of the theory of evolution is that everybody thinks he understand it. I mean philosophers, social scientists, and so on. While in fact very few people understand it, actually as it stands, even as it stood when Darwin expressed it, and even less as we now may be able to understand it in biology.

Jacques Monod
The false view of evolution as a process of global optimizing has been applied literally by engineers who, taken in by a mistaken metaphor, have attempted to find globally optimal solutions to design problems by writing programs that model evolution by natural selection.