The four possible outcomes
| H₀ is true (no effect) | H₀ is false (real effect) | |
|---|---|---|
| Reject H₀ | Type I error (false positive), probability α | Correct detection, probability 1 − β (power) |
| Fail to reject H₀ | Correct, probability 1 − α | Type II error (false negative), probability β |
Examples
- Clinical trial: concluding a new drug lowers blood pressure when it doesn't (type I) vs dismissing a drug that works (type II).
- Screening test: a healthy person flagged as ill (type I) vs a disease missed (type II).
- Education study: reporting that a tutoring programme raised scores when the difference was chance (type I) vs a small pilot of 15 students per group that misses a genuine d = 0.4 improvement (type II; its power is only about 18%).
Why thresholds mislead intuition
The critical value is a cut-off, and cut-offs are where intuition fails: p = .049 and p = .051 are practically identical evidence, yet land on opposite sides of the line. People misread cut-offs everywhere, and a classic case is income tax, where many believe earning a dollar past a bracket raises the tax on all their pay; ahaboo's narrated walkthrough of US tax brackets untangles that one.
For research, the practical lesson is to report the exact p-value, the effect size and its confidence interval, and to plan power in advance so a “non-significant” result is informative rather than a type II error waiting to happen.
Multiple testing inflates type I errors
Run 20 independent tests at α = .05 on pure noise and the chance of at least one false positive is 1 − 0.95²⁰ ≈ 64%. Corrections such as Bonferroni (α/m) or Holm keep the family-wise rate at 5%; the false discovery rate approach of Benjamini and Hochberg (1995) is less conservative for large numbers of tests.
Frequently asked questions
What is a type I error?
Rejecting a null hypothesis that is actually true: a false positive. Its probability is α, the significance level you choose, usually 5%.
What is a type II error?
Failing to reject a null hypothesis that is actually false: a false negative, missing a real effect. Its probability is β; power is 1 − β.
Which error is worse?
It depends on the consequences. Approving an ineffective drug (type I) and abandoning an effective one (type II) carry different costs. Fields set α and target power to reflect that balance; the conventional 5% vs 20% treats a false positive as four times as costly.
How do I reduce both errors at once?
Increase the sample size, reduce measurement noise, or study a larger effect. With a fixed sample, lowering α always raises β.