Test whether two sample proportions are significantly different — ideal for A/B tests and clinical trials.
The two-proportions z-test tests whether the proportion of successes in two independent groups is statistically different. The null hypothesis is H₀: p₁ = p₂ — the proportions are the same in both groups.
Example: you run an A/B test where Version A converted 45/100 visitors and Version B converted 35/100. Is this 10-percentage-point difference statistically significant, or could it be due to random chance?
A "success" is just the event you're counting — whatever you want to compare between the two groups:
The 95% CI for (p₁ − p₂) gives you a range of plausible values for the true difference in proportions. If this interval does not contain 0, the difference is statistically significant at α = 0.05.
For example, a CI of [0.02, 0.18] means the true difference is likely between 2 and 18 percentage points, with Group 1 being higher — a meaningful and significant result.
Cohen's h is the effect size measure for proportions. Unlike the raw difference p₁ − p₂, Cohen's h accounts for the non-linearity of proportions near 0 and 1.
The formula is: h = 2·arcsin(√p₁) − 2·arcsin(√p₂)
Cohen's benchmarks: small (0.2), medium (0.5), large (0.8). A statistically significant result with a small Cohen's h may not be practically meaningful — you need both a significant p-value and a meaningful effect size before acting on the result.
For the normal approximation to be valid, each group should satisfy: np ≥ 5 and n(1−p) ≥ 5. With very small proportions (like 1-2% conversion rates), you may need hundreds or thousands of observations per group to detect meaningful differences.
Use the Sample Size Calculator to find the required n for a given power and minimum detectable difference.