Post navigation

Was Einstein Wrong About Insanity?

“Insanity”, said Einstein, “is doing the same thing over and over again and expecting different results.”

But in science, how sane is it to expect experiments to give precisely the same results?

How big a problem is irreproducibility in science?

Experiments on that question have given variable results. Haha.

But seriously, why is it that so much of what is published in science is so difficult to reproduce? While Einstein (pictured above in an image from Wikipedia) may have had less trouble with irreproducibility in Physics, the biomedical sciences–particularly in pre-clinical and clinical studies–is highly variable and often data are not reproducible.

Most biomedical scientists agree that this is a vexing issue and in the last week even NIH was reported by Nature to be considering taking action on the issue, possibly via new requirements for grantees with awards that are clinical or pre-clinical in nature.

Some biological scientists might flip Einstein’s quote around:

Insanity in Science is Doing the Same Thing Over & Over Again and Expecting the Same Results

Most scientists have had the experience of experimental irreproducibility.

On a microlevelit occurs with our own experiments at times. On Monday we do qPCR. On Wednesday we repeat the exact same qPCR to the best of our knowledge doing it precisely the same, but we get different results.

Why? I’m not sure we can know, but often it seems to be the result of how complicated biology can be and variables that we just don’t perceive.

On a macro scale, we may try to repeat a published experiment and we get very different results.

Why?

Sometimes the cause cannot be pinned down. Other times I think it is due to scientists publishing their data too fast, egged on by the quest to be first and get into high impact journals so they simply do not repeat their experiments enough times before publishing.

Reproducibility is central to all science because our goal is to test hypotheses and try to come to solid conclusions based on data. Solid conclusions should be ones that other scientists can confirm.

How big is this irreproducibility problem? It seems to be huge as we look at another section of the Nature article:

In a 2011 internal survey, pharmaceutical firm Bayer HealthCare of Leverkusen, Germany, was unable to validate the relevant preclinical research for almost two-thirds of 67 in-house projects. Then, in 2012, scientists at Amgen, a drug company based in Thousand Oaks, California, reported their failure to replicate 89% of the findings from 53 landmark cancer papers. And in a study published in May, more than half of the respondents to a survey at the MD Anderson Cancer Center in Houston, Texas, reported failing at least once in attempts at reproducing published data (see ‘Make believe‘).

Those are big numbers.

Another part of the problem surely is that the emphasis amongst scientists themselves, editors, funding agencies, and others is on doing new stuff, not reproducing published results.

Do not even propose trying to repeat or publish things that others have done or your grant proposal or paper will be rejected. One of the big criterion for NIH grants is Innovation after all.

Is repeating Ted’s experiment published last year going to be high on a study section’s priority list? No! You must say you are doing something very different and better than what Ted did.

This obsession with perceived novelty results in many published studies standing alone without ever having been recapitulated.

Of course if you are deeply interested in the same topic as another scientist, say Ann, then you may choose to try to repeat her experiment anyway and the experience of science over the years predicts that often you will get very different data than Ann.

What does that mean?

Most of the time it doesn’t mean anything bad about you or Ann.

Rather it is the nature of science that there is a lot of noise in most systems. Molecular machinery and cells themselves in biological systems in particular are often inherently variable with major stochastic elements.

Sometimes the key conclusions of papers are correct and the results in a general sense are reproducible.

Still I believe that science needs to also reward validation studies, not just what some view as sexy new stuff. Science also needs to reward studies that are themselves rock solid technically speaking and have done a rigorous job on replicating their own studies and used robust statistical evaluation.

At the same time we need to be mature and realistic about the fact that not every paper or project will produce binary, black and white results. Too often reviewers expect all data to march lock step together to produce one uniform conclusion and they attack a manuscript that includes subtleties, what I call the “grays” of science.

It may not be flat out insane to expect perfectly reproducible results and our goal should be reproducibility, but on the other hand let’s be mature and accepting of the fact that science and biology in particular in the real world often produce variable results.

We should do our best to minimize that variability by doing the most rigorous science that we can and keep expectations high, but at the same time a certain realism is healthy on an individual and field-wide basis. It should also motivate us to use extra caution before jumping into clinical studies where there is likely to be even more variability due to each person being so different.

7 thoughts on “Was Einstein Wrong About Insanity?”

Your complaint about a bias towards innovation as opposed to confirmation is well founded in the scientific disciplines with which I am familliar. I guess it’s human nature. Other problems include not publishing enough of the detail to ensure that others can replicate the work without making implicit assumptions.

Two general comments:
(1) The purpose of an experiments (or observations) is not replication of other data but confirmation of theory. Thus, I don’t entirely agree with your ultimate conclusion. When experiments don’t replicate, nonlinear dynamicists become interested.
(2) Following from (1), it is important to realize that Einstein belonged to an era before deterministic chaos was properly appreciated by scientists. Einstein should be forgiven if his statement only applies within what we would now regard as a limiting set of constraints.

This is a good article, and I hope no one thinks less of it because it is based on a misattributed quote.

From Wikiquote: “The definition of insanity is doing the same thing over and over and expecting different results.
Variously misattributed to figures also including Benjamin Franklin and Mark Twain. The earliest known occurrence, and probable origin, is from a 1981 text from Narcotics Anonymous.”

Disclaimer.
The material on this blog is intended for educational purposes only and absolutely should not be considered medical or financial advice. The opinions here are only those of the authors and do not reflect those of the University of California, Davis.

Comment Policy.
Comments are encouraged. Those submitted to this blog will be posted at the sole discretion of the editor and may be edited for content. Comments cannot include person attacks, promotional material for stem cell clinics, or other inappropriate content. Commenters are encouraged to use their real names, but anonymous comments are accepted.