Statistical Significance

Usually, mental health research examines the difference between two
groups to determine if it is statistically significant. The difference
between two groups is statistically significant if it can not be explained
by chance alone.

Usually, statistical significance is determined by
calculating the probability of error (p value) by the t
ratio.

The difference between two groups (such as an experiment vs. control
group) is judged to be statistically significant when p = 0.05
or less.

At p= 0.05, the differences between
the two groups have only a 5% probability of occurring by chance
alone.

At p= 0.01, the differences between
the two group have only a 1% probability of occurring by chance
alone.

Clinical Significance

The problem with the p value approach is that two groups could differ
significantly (with p= 0.05 or less), yet the actual difference
between the two groups could be so small that it is not clinically
significant. This problem usually occurs when:

The the group sizes are very large.

Scores within the groups are very similar (i.e.,
the groups have small standard deviations).

Statistical And Clinical Significance

EFFECT SIZE

There are ways of determining if the difference between two groups is
both statistically and clinicially significant. The most commonly
used method of determining the clinical significance of the difference
between two groups is to calculate the effect size (ES).

Effect Size (ES), as proposed by Cohen, is estimated by the ratio of
the mean difference between the two groups divided by the standard deviation
of the control group (or the pooled variance of the treatment groups if
there is no control group).

The control group is normally used as an estimate
of the variance because it has not been contaminated by the treatment
effect.

Effect Size (ES) = (experimental group
mean - control group mean) divided by the standard deviation of the control
group:

An effect size (ES) value of 0.2 represents
a small statistical and clinical difference between two groups.

An effect size (ES) value of 0.5 represents
a moderate statistical and clinical difference between two groups.

An effect size (ES) value of 0.8 represents
a large statistical and clinical difference between two groups.

PERCENT IMPROVEMENT

A second way to assess clinical significance is to calculate the percent
improvement. In mental health, the convention is that a 12% improvement or greater
represents a clinically significant difference. (This 12% is selected because antidepressant medications are, on average, 12% better than placebo in the treatment of moderate depression.)

A third way to assess clinical significance is to calculate the Number
Needed To Treat (NNT). The number needed to treat (NNT) is simply
the inverse of the percent improvement.

Number needed to treat
(NNT) = 1 divided by percent improvement.

The number needed to treat (NNT) is the number of
patients who need to be treated in order to produce one additional good
outcome.

In mental health, a number needed to treat (NNT) value
of 8 or less represents a clinically significant finding.

To explain this concept, consider the following. Let's say a drug caused
1% of the people on it to improve. On average, how many people would you
have to treat with this drug before one person improved? The answer is
100 people.

Now consider this second example. Let's say a drug improved 20% of the
people on it. On average, how many people would you have to treat with
this drug before one person improved? The answer is 5 people.

Now consider this third example. Let's say the percent improvement on
drug A was 20% more than on drug B. On average, how many people would
you have to treat with drug A before one person improved more than
would have improved on drug B? The answer is 5 people.

(0.20 greater chance of improvement on drug A per
individual x 5 individuals = 1 additional improved individual on drug
A more than would have improved on drug B)

In these three examples, you calculated the number needed to treat (NNT).

Mathematically, a number needed to treat (NNT) of
5 is equal to a 20% improvement. A number needed to treat (NNT) of 2 is
equal to a 50% improvement.

When calculating the percent improvement or number
needed to treat (NNT), the researcher also has to calculate the probability
of error (p value) to determine if the research findings could
have been due to chance alone.