How the test works
Under H₀ both groups share one true rate, estimated by pooling: p̄ = (x₁ + x₂)/(n₁ + n₂). The standard error of the difference under that assumption is √(p̄(1 − p̄)(1/n₁ + 1/n₂)), and z is the observed difference divided by it. The confidence interval uses the unpooled standard error, which is appropriate when estimating rather than testing.
Why small differences need huge samples
Required sample size grows with the inverse square of the effect: halve the difference you want to detect and you need about four times as many people. That is why a minimum detectable effect should be set by what matters to the decision, not by what the sample happens to allow.
For more than two groups or outcomes, use the chi-square calculator.
Frequently asked questions
How do I know if the difference between two percentages is significant?
Run a two-proportion z-test: z = (p̂₁ − p̂₂) / √(p̄(1 − p̄)(1/n₁ + 1/n₂)), where p̄ is the pooled proportion. If the two-tailed p-value is below α (usually .05), the difference is statistically significant.
How many visitors do I need for an A/B test?
Use the “Plan a test” tab. For a baseline of 5% and a minimum detectable effect of 1 percentage point (to 6%) with 80% power at α = .05, you need about 8,160 per variant.
Can I stop the test as soon as it becomes significant?
No. Repeatedly checking and stopping at the first p < .05 inflates the false-positive rate well above 5%. Fix the sample size in advance, or use a sequential method designed for peeking.
Is this the same as a chi-square test?
For a 2 × 2 table, yes: the two-proportion z-test squared equals the Pearson chi-square statistic, and the two-tailed p-values are identical.