Methods proposed to improve how observational studies are conducted

Statisticians point out that medical and other observational studies often produce results that are later shown to be incorrect. They now suggest ways to fix the system.

Share:

Total shares:

FULL STORY

S. Stanley Young, assistant director for bioinformatics at the National Institute of Statistical Sciences (NISS), and Alan Karr, director at NISS, have published a non-technical article in the September issue of Significance magazine pointing out that medical and other observational studies often produce results that are later shown to be incorrect, and -- invoking a quality control perspective -- suggest ways to fix the system.

Their central point is that the current system of publication in peer-reviewed journals relies on post-production inspection to ensure quality, a practice that has disappeared from modern industry in favor of controlling the process instead: quality control is now process control, not product control. They cite W. Edwards Deming, considered by many the most innovative thinker ever about quality, arguing not only for process control, but also that the problem lies with the managers -- funders and journals -- rather with than the workers -- individual researchers who respond rationally to the current set of incentives.

Young and Karr describe both their and others' studies of the extent to which observational studies do not replicate. Published claims such as "coffee causes pancreatic cancer," or "women eating breakfast cereal are more likely to have boy babies," have been refuted by subsequent studies and analyses. When these studies reach the popular media and influence individual consumers, the burden falls not just on science but also on society. And even if there were no impact on the public, scarce research resources, both money and personnel, have been squandered.

The paper describes several technical difficulties with observational studies, among them multiple testing (if enough questions are asked, some will yield false positive answers), bias (systematic error) and multiple modeling (searching among mathematical models until one is found that "fits the data"). Publication bias is another issue: papers reporting positive scientific results (for example, an association between Type A personalities and heart attacks) are more likely to be published than those reporting negative results, even though the latter may be as important scientifically.

Young and Karr recommend that when a study is submitted for publication, the data be split into two sets, a modeling data set and a holdout data set. Journals would then accept or reject papers based on the analysis of the modeling data set without knowing the results of applying the methods to the holdout set. But then the journal would also publish an addendum to the paper giving the results of the analysis of the holdout set.

Significance magazine is published by the Royal Statistical Society of the UK and the American Statistical Society.

National Institute of Statistical Sciences. "Methods proposed to improve how observational studies are conducted." ScienceDaily. ScienceDaily, 25 August 2011. <www.sciencedaily.com/releases/2011/08/110825123832.htm>.

National Institute of Statistical Sciences. (2011, August 25). Methods proposed to improve how observational studies are conducted. ScienceDaily. Retrieved August 2, 2015 from www.sciencedaily.com/releases/2011/08/110825123832.htm

National Institute of Statistical Sciences. "Methods proposed to improve how observational studies are conducted." ScienceDaily. www.sciencedaily.com/releases/2011/08/110825123832.htm (accessed August 2, 2015).

July 30, 2015  It is possible to predict the timing and intensity of influenza outbreaks in subtropical climates like Hong Kong where flu seasons can occur at different times and more than once during a year, ... read more

July 30, 2015  A new automated data mining system could lead to a dramatic increase in the detection of potentially illegal online sales of elephant ivory through eBay. Law enforcement agencies and conservationist ... read more

July 28, 2015  Researchers explain how the new paradigm of a digital healthcare system, as it matures, is putting the picture of the doctor-patient relationship in an entirely new frame and not always in a positive ... read more

July 24, 2014  Shift work is linked to a heightened risk of developing type 2 diabetes, with the risk seemingly greatest among men and those working rotating shift patterns, indicates an analysis of the available ... read more

May 3, 2011  Systematic reviews that attempt to assess the risk of harms (adverse effects) associated with specific therapies should consider a broad range of study designs, including both systematic reviews and ... read more