Subjective Ranking of Quality of Research by Subject Matter Area

Last updated on
Mar 27, 2017
4 min read

While being engaged in biomedical research for a few decades and watching
reproducibility of research as a whole, I’ve developed my own ranking of
reliability/quality/usefulness of research across several subject matter
areas. This list is far from complete. Let’s start with a subjective
list of what I perceive as the areas in which published research is
least likely to be both true and useful. The following list is ordered
in ascending order of quality, with the most problematic area listed
first. You’ll notice that there is a vast number of areas not listed for
which I have minimal experience.
Some excellent research is done in all subject
areas. This list is based on my perception of the proportion of
publications in the indicated area that are rigorously scientific,
reproducible, and useful.

Subject Areas With Least Reliable/Reproducible/Useful Research

any area where there is no pre-specified statistical analysis plan
and the analysis can change on the fly when initial results are
disappointing

behavioral psychology

studies of corporations to find characteristics of “winners”;
regression to the mean kicks in making predictions useless for
changing your company

animal experiments on fewer than 30 animals

discovery genetics not making use of biology while doing
large-scale variant/gene screening

nutritional epidemiology

electronic health record research reaching clinical conclusions
without understanding confounding by indication and other
limitations of data

pre-post studies with no randomization

non-nutritional epidemiology not having a fully pre-specified
statistical analysis plan [few epidemiology
papers use state-of-the-art statistical methods and have a
sensitivity analysis related to unmeasured
confounders]

prediction studies based on dirty and inadequate data

personalized medicine

biomarkers

observational treatment comparisons that do not qualify for the
second list (below)

small adaptive dose-finding cancer trials (3+3 etc.)

Subject Areas With Most Reliable/Reproducible/Useful Research

The most reliable and useful research areas are listed first. All of
the following are assumed to (1) have a prospective pre-specified
statistical analysis plan and (2) purposeful prospective
quality-controlled data acquisition (yes this applies to high-quality
non-randomized observational research).

observational research [however only a very small minority of
observational research projects have a prospective analysis plan and
high enough data quality to qualify for this
list]

Some Suggested Remedies

Peer review of research grants and manuscripts is done primarily by
experts in the subject matter area under study. Most journal editors
and grant reviewers are not expert in biostatistics. Every grant
application and submitted manuscript should undergo rigorous
methodologic peer review by methodologic experts such as
biostatisticians and epidemiologists. All data analyses should be
driven by a prospective statistical analysis plan, and the entire
self-contained data manipulation and analysis code should be submitted
to journals so that potential reproducibility and adherence to the
statistical analysis plan can be confirmed. Readers should have access
to the data in most cases and should be able to reproduce all study
findings using the authors’ code, plus run their own analyses on the
authors’ data to check robustness of findings. Medical journals are reluctant to (1) publish critical letters to the editor and (2) retract papers. This has to change.

In academia, too much credit is still given to
the quantity of publications and not to their quality and
reproducibility. This too must change. The pharmaceutical industry has
FDA to validate their research. The NIH does not serve this role for
academia.

Rochelle Tractenberg, Chair of the American
Statistical Association Committee on Professional Ethics and a
biostatistician at Georgetown University said in a 2017-02-22 interview
with The Australian that many questionable studies would not have been
published had formal statistical reviews been done. When she reviews a
paper she starts with the premise that the statistical analysis was
incorrectly executed. She stated that “Bad statistics is bad
science.”