A/B Test Significance Examples & P-Values
You ran the test, one variant won — but was the lift real or just noise? Calling a winner too early on thin data wastes budget and poisons future decisions. These worked examples show how to read p-values and confidence intervals so you ship the variant that actually converted better.
Bottom Line
A/B test significance determines if observed differences between variants are likely due to the change made or merely random chance, guiding data-driven decisions.
A/B Test Significance Calculator
Check if your A/B test results are statistically significant and estimate sample size for reliable conclusions.
On This Page
Worked Examples
See the inputs and outcome together
Each scenario keeps the starting point, the outcome, and the actual lesson in one place so the page reads like a decision notebook, not a data dump.
- 1
Baseline case
Compare a control and variant that each drew 5,000 visitors, with B converting 285 against A's 250.
Rate A is 5.0% and rate B is 5.7%, a 14% relative lift. But the p-value is 0.12 and the verdict is Not Significant: you would need about 16,224 visitors per arm to trust it.
Visitors A
5,000
Conversions A
250
Visitors B
5,000
Conversions B
285
A 14% lift looks promising yet fails the test at this sample size. Resist shipping on the headline number; the math says keep the test running until traffic catches up.
- 2
Stronger variant
Keep the 5,000 visitors per arm but let B convert 320, widening its edge over the control.
Rate B rises to 6.4% versus 5.0% for A, a 28% relative lift. The p-value drops to 0.0025 and the verdict flips to Significant.
Visitors A
5,000
Conversions A
250
Visitors B
5,000
Conversions B
320
Doubling the effect size from 14% to 28% pushed the same traffic past the bar. Bigger wins reach significance faster, which is why a clearly better variant settles quickly.
- 3
Same lift, more traffic
Reproduce the baseline 14% lift but quadruple the sample: 20,000 visitors per arm, 1,000 against 1,140 conversions.
Rates are still 5.0% and 5.7% with the same 14% lift, but the p-value now falls to 0.0019 and the verdict is Significant.
Visitors A
20,000
Conversions A
1,000
Visitors B
20,000
Conversions B
1,140
The effect size is identical to the baseline, yet four times the traffic turned a coin-flip into a confident win. Sample size, not just the lift, decides whether a result holds.
- 4
Marginal variant
Hold 5,000 visitors per arm and shrink B's edge to 260 conversions, a barely-there improvement.
Rate B is 5.2% against 5.0%, a 4% lift with a p-value of 0.65 and a Not Significant verdict. Detecting it reliably would take roughly 189,992 visitors per arm.
Visitors A
5,000
Conversions A
250
Visitors B
5,000
Conversions B
260
A 4% lift is statistical noise at any realistic traffic level. The required sample of nearly 190,000 per arm is a signal to stop the test and chase a bolder change instead.
Patterns
Try These Tools
Run the numbers next
Net Promoter Score (NPS) Calculator
Calculate NPS from promoter, passive, and detractor counts with benchmark context and action guidance.
Open →Churn & Retention Calculator
Estimate recovered customers and revenue lift from retention improvements.
Open →FAQ
Questions people ask next
The short answers readers usually want after the first pass.
Sources & References
Related Content
Keep the topic connected
How to Use A/B Test Significance Calculator
Validate your A/B test results to make data-driven decisions. Learn how to use this calculator to determine if observed differences in conversion rates.
What Is Churn Rate? Simply Explained
Understand Churn Rate, a vital SaaS metric, with a clear definition, formula, real-world examples, and strategies to improve customer retention and revenue.