15 A/B Testing Statistics

At Microsoft, only about one-third of well-designed experiments improved the metric they were built to move.

Most ideas do not win. This finding is the central argument for testing rather than shipping on intuition, and it held up across thousands of experiments.

Microsoft reported that roughly one-third of ideas were positive and significant, one-third were flat, and one-third were negative and significant.

A sizable share of changes actively hurt the metric. Without a controlled test, those losers ship silently and erode results.

In well-optimized products such as search, the share of experiments that fail to improve the target metric runs 80 to 90 percent.

The more mature a product, the harder a real win is to find. High failure rates are a sign of a hard problem, not a broken testing program.

Microsoft ran more than 1,000 experiment treatments per month on Bing alone.

High volume is how a low win rate still produces gains. When most tests fail, you need many tests to find the few that work.

A common A/B testing setup uses a 5 percent significance level and 80 percent statistical power.

These two thresholds drive the required sample size. Choosing them before the test, not after, is what keeps the result honest.

Under one worked example in the methodology review, detecting a given effect at 5 percent significance and 80 percent power required roughly 456 users per variant.

Sample size depends on the effect you want to detect. Smaller expected lifts demand far larger samples, which is why many tests are underpowered.

Checking a test repeatedly and stopping as soon as the p-value crosses significance, known as peeking, seriously inflates the false-positive rate.

Stopping early when the result looks good is the most common way to fool yourself. The nominal 5 percent error rate no longer holds once you peek.

Sequential testing methods allow repeated significance checks without inflating the false-positive rate, unlike a fixed-horizon t-test.

If you need to monitor a test continuously, use a method built for it. Sequential designs are the correct fix for the peeking problem.

A 5 percent significance level means that, with no real effect, about 1 in 20 tests will still show a significant result by chance.

False positives are built into the threshold itself. Running many tests guarantees some chance winners, which is why replication matters.

Across Microsoft, Netflix, and others, published reviews report that only 10 to 33 percent of new features positively move their desired metric.

The win rate is similar across large experimentation programs. It is not a Microsoft quirk; it reflects how hard it is to improve a product.

Microsoft ran more than 100 experiment treatments per month across Office, OneNote, Xbox, Cortana, Skype, and Exchange.

Experimentation scaled well beyond search at Microsoft. The same low-win-rate, high-volume pattern recurs across very different products.

A result that is statistically significant can still be too small to matter; the methodology review stresses separating statistical significance from practical significance.

A tiny but significant lift may not justify the cost of shipping. Deciding the minimum worthwhile effect before testing prevents chasing noise.

Many promising ideas that teams were confident would work failed to move the metric once tested in a controlled experiment at Microsoft.

Expert intuition is a weak predictor of which changes win. That gap between conviction and outcome is the strongest case for running the test.

Statistical power of 80 percent means a real effect of the assumed size will be detected 80 percent of the time, and missed 20 percent.

An underpowered test can miss a true winner one time in five even at the standard threshold. Low power is a silent cause of false negatives.

When Microsoft first shared that only about a third of experiments succeed, many staff dismissed it, but repeated experiments confirmed the figure.

The low win rate surprises even practitioners. Setting that expectation up front keeps a testing program from being judged as a failure.