Lies, Damned Lies, and Medical Science

His model predicted, in different fields of medical research, rates of wrongness roughly corresponding to the observed rates at which findings were later convincingly refuted: 80 percent of non-randomized studies (by far the most common type) turn out to be wrong, as do 25 percent of supposedly gold-standard randomized trials, and as much as 10 percent of the platinum-standard large randomized trials.

These were challenging facts for me, as a deep believer in science and rigorous research. It’s worth considering the meaning of these findings in the health field for the new focus on evidence-based funding in the social sector and in foundations. The author argues, based on the above, that when science becomes big business, when one’s livelihood depends on finding positive effects, the scientific ideal (the search for truth) regularly falls under the wheels. Are social interventions any easier to study than biological systems — and therefore any more likely to produce accurate results under the best of research conditions? Does the new pressure for evidence-based funding create high stakes that will generate bias and inaccuracy in purportedly scientific studies of programs? The arts aren’t under the microscope of evidence-based funding today, but this shift in the thinking around us is already starting to inform the thinking of the program officers and donors upon which we rely. What can we do to ensure a reliable system of study in our field if evidence-based funding is going to become a foundation of philanthropy?

About the author

Jim Rosenberg

Jim Rosenberg is National Arts Strategies’ Senior Advisor. He is interested in customer-centered strategy, the innovation process and small and large group collaboration. Jim spends most of his time these days chasing his daughter and trying to keep the house in one piece, but still finds a few moments to draw, try to learn guitar and puzzle over the ways that new information and ideas do and don’t get integrated into people’s lives.