Bias Arising From the Use of Patient-Reported Outcome Measures

Objective The objective was to assess the bias in outcomes effects associated with the use of patient-reported outcome (PRO) measures of varying psychometric quality in peer-reviewed clinical studies.

Design A literature search was conducted using PubMed (January 1, 2011, to December 31, 2016) to search for randomized and observational studies (cohort, case-control studies) published in the top 5 orthopedic journals (as ranked by their Impact Factors), in humans that used PRO measures, for rotator cuff disease.The study design, sample size, standard aspects of risk of bias (high or low using Cochrane Methods, such as randomization methods, allocation concealment, and blinding) for each study type, types of PRO outcome measures used, data related to the PRO measure results in all groups, measures of effect (odds ratio, relative risk, incidence risk ratio, absolute risk reduction, mean change, etc), and confidence intervals or other measures of variance were extracted. Continuous measures of effect were transformed by dividing the effect estimate by the standard deviation. PRO measures were given numerical ratings of quality based on a systematic review of evidence for their psychometric properties. Linear regression analyses were performed to determine whether PRO measure quality was associated with the magnitude of effects and the influence of a variety of covariates on this relationship.

Results Overall, 162 articles were included for this study across 5 high-impact orthopedic journals. Of the studies, 80% were observational and 20% randomized. Studies included 1 to 7 PRO measures. Greater than 75% of the included studies did not justify the use of PRO and greater than 50% did not describe the PRO details. Linear regression revealed that lower-quality PROs had larger estimates of effect, and by contrast, higher-quality PROs had smaller estimates of effect (n = 123; β = −0.21, 95% CI, −0.43 to −0.02; P = .03). In univariable regression analyses, we also found that a longer follow-up period (range, 0.1 to 96 months) predicted slightly increased effect estimates (n = 96; β = 0.05; 95% CI, 0.02 to 0.09; P = .002).

Conclusions PRO measures with poor or unknown psychometric properties bias (ie, inflate) the estimates of treatment effect in clinical research of rotator cuff disease. To our knowledge, this is the first empirical evidence, to date, that variations in the quality of PRO measures bias treatment effect estimates. Researchers and clinicians using data from PROs must be cautious to explore the quality of that measure so as to not mislead decision making from biased outcomes.