Effect size

The effect size statistic is a method of reporting the magnitude of improvement by converting raw score change to standardized change scores by dividing the raw score change by the standard deviation of the outcome measure. In recent years most psychology and psychotherapy journals have required that results include all of the information necessary to calculate Cohen's d effect size statistic. An effect size of 1 means that the patient improved the equivalent of one standard deviation on the outcome measure.

Meta-analysis utilizes the effect size statistic to combine results from multiple studies using different outcome measures.

Various criteria have been used describe the magnitude of effect sizes though must accept the opinion of Cohen (1988, 1992).

Cohen's Effect Size Interpretation

.2=small

.5=medium

.8=large

Note: The term effect size is also used when evaluating the magnitude of differences based on statistics other than pre-post change, such as differences between two groups, differences in Pearson r correlations, Chi-square goodness of fit, one way analysis of variance, as well as multiple and partial correlations. In each instance, the formulas for calculating effect size are quite different, as are the interpretation of the magnitude of the effect size. The attached table from Cohen's 1992 classic article "A Power Primer" provides the formulas and guidelines for interpreting the various effect size indexes: Cohen (1992); Table 1

Outcome measures and effect size

Minami et al (2007a) note that different types of outcome measures produce different magnitude effect size. Clinician assessments of improvement produce larger effect sizes than patient self report. Among patient self report questionnaires, focused on symptoms of a specific disease (i.e. The Beck Depression Inventory)tend to produce large effect sizes than more global measures of patient distress such has the SCL-90 or the OQ-45. The ACORN questionnaires are of this later type.

Minami et al (2007b) report the results of a benchmarking study comparing effect sizes for treatment as usual in a managed care environment, as measured by the OQ-30 (a global distress measure), to the effect size estimate obtained from a meta-analysis if published clinical trials for psychotherapy treatment of depression that utilized similar global distress measures. The benchmark from clinical trails was .79 effect size (large by Cohen's criteria). The effect size for treatment as usual was comparable, with the lowest effect size being for patients with co-morbid conditions receiving psychotherapy alone (.72). Since the difference between .72 and .83 is less than .2 effect size (Cohen's threshold for small effect size), Minami et al argue that .72 be considered "good enough" as the difference between the the two groups was to small to be clinical meaningful at the individual patient level.

In light of this evidence, it appears that Cohen's general rule of thumb to consider a point .5 effect as moderate and .8 as large is appropriate when evaluating effect sizes for behavioral health treatment, as measured using patient self report measures of global distress. These threshold are employed as part of the the ACORN Criteria for Effectiveness.