Research

When Is A Difference Between Two Groups Significant?


Statistical Significance

Usually, mental health research examines the difference between two groups to determine if it is statistically significant. The difference between two groups is statistically significant if it can not be explained by chance alone.

Clinical Significance

The problem with the p value approach is that two groups could differ significantly (with p= 0.05 or less), yet the actual difference between the two groups could be so small that it is not clinically significant. This problem usually occurs when:

Statistical And Clinical Significance

EFFECT SIZE

There are ways of determining if the difference between two groups is both statistically and clinicially significant. The most commonly used method of determining the clinical significance of the difference between two groups is to calculate the effect size (ES).

PERCENT IMPROVEMENT

A second way to assess clinical significance is to calculate the percent improvement. In mental health, the convention is that a 12% improvement or greater represents a clinically significant difference. (This 12% is selected because antidepressant medications are, on average, 12% better than placebo in the treatment of moderate depression.)

NUMBER NEEDED TO TREAT

A third way to assess clinical significance is to calculate the Number Needed To Treat (NNT). The number needed to treat (NNT) is simply the inverse of the percent improvement.

To explain this concept, consider the following. Let's say a drug caused 1% of the people on it to improve. On average, how many people would you have to treat with this drug before one person improved? The answer is 100 people.

Now consider this second example. Let's say a drug improved 20% of the people on it. On average, how many people would you have to treat with this drug before one person improved? The answer is 5 people.

Now consider this third example. Let's say the percent improvement on drug A was 20% more than on drug B. On average, how many people would you have to treat with drug A before one person improved more than would have improved on drug B? The answer is 5 people.

In these three examples, you calculated the number needed to treat (NNT).


Internet Mental Health (http://www.mentalhealth.com/) copyright 1995-2011 by Phillip W. Long, M.D.