The Most Common Way Teams Fool Themselves With A/B Test Results
Ask most people what "95% statistically significant" means and you'll get some version of: "we're 95% sure B is better than A." That's a natural reading. It's also wrong, and the gap between what significance actually means and what people assume it means is where a lot of bad CRO decisions come from.
What 95% significance actually says
Convert's explainer on statistical significance puts it precisely: a result significant at 95% confidence means that if there were truly no difference between A and B, there'd only be a 5% chance of seeing a difference this large purely by random luck. It's a statement about how surprising your data would be under the assumption of no effect. It is not a direct statement about the probability that your variant is actually better.
That distinction sounds academic until you see what it does in practice.
Peeking is the most common self-inflicted wound
The single most common mistake in A/B testing isn't a math error, it's impatience. Teams check results daily, and the moment a test crosses the significance threshold, they call it and ship the winner. The problem is that significance naturally fluctuates above and below the threshold throughout a test's run, especially early on with a small sample. Stop the instant it crosses 95% and you're systematically more likely to have caught a random fluctuation, not a real effect.
The fix is boring but effective: decide your sample size and minimum detectable effect before the test starts, and don't call it until you hit that number, regardless of what the significance indicator says on day three.
Small sample sizes make this worse
If your test only gets a few hundred visitors a week per variant, hitting genuine statistical significance in a reasonable timeframe is difficult, and the temptation to call it early on a promising-looking early swing is strongest exactly when the underlying data is least reliable. Low-traffic pages are where peeking does the most damage, because the noise is loudest right where teams are most impatient for a result.
The other quiet failure mode: data quality
Even a textbook-correct significance calculation is only as good as the data feeding it. Broken tracking, bot traffic, or an experiment that accidentally excludes a chunk of real users will produce a confident, clean-looking, completely wrong answer. Before trusting a result, it's worth checking that the numbers going in make sense, not just that the math coming out looks tidy.
None of this means testing is unreliable. It means testing is a discipline, not a dashboard you glance at until it turns green. If you want your experimentation program audited for exactly this kind of issue, that's a normal part of what we cover in a free consultation.