The peeking problem: why stopping an A/B test early breaks your stats

Q: Is it OK to check my A/B test results before it finishes?

Looking at the dashboard is fine. The problem is acting on it: stopping the test the moment it crosses p<0.05. Each extra look is another chance to cross the line by chance, so early stopping is what inflates your false-positive rate, not viewing the numbers.

Q: How much does peeking inflate the false-positive rate?

It depends on how often you look, but the effect is large. A test with a nominal 5% false-positive rate can reach 15% to 25% or higher if you check it daily over several weeks and stop at the first significant reading. The more looks you take, the worse it gets.

Q: Does ABTestly let me stop a test early when it looks significant?

ABTestly uses a fixed-horizon approach and will not declare a winner before the test reaches significance at the sample size you planned for. Until then it shows Still collecting. Sequential testing, which is designed for continuous monitoring, is on the roadmap and will be labelled as such when it ships.

A methodology guide for CRO teams. Published 7 July 2026.

Peeking is the habit of watching a running A/B test and stopping it the moment the result reads significant. It feels responsible. You are paying attention, you spotted a winner, you shipped it fast. In reality it is the single most common way teams ship false winners, because the classic significance test was never designed to be checked over and over.

What peeking actually is

A standard fixed-horizon test works like this: you decide the sample size in advance, run until you reach it, then look once and read the result. The 5% significance threshold (p<0.05) is the promise that if there is truly no difference between A and B, you will be fooled into calling a winner only 5% of the time.

Peeking breaks the promise. Instead of looking once at the planned finish line, you look every day and stop as soon as p drops below 0.05. That single change quietly rewrites the maths, and not in your favour.

Why repeated looks inflate false positives

Think of each look as a separate coin toss for a false alarm. One look at a true null gives you a 5% chance of a spurious significant result. But the p-value of a live test wanders up and down as data trickles in, driven by ordinary random noise. Every time you look, you give that wandering line another chance to dip below the threshold, even when nothing real is happening.

This is the multiple-comparisons problem wearing a disguise. You are not comparing many variants, you are testing the same variant many times, and each test carries its own 5% risk. Stack enough looks together and the chance that at least one of them trips the wire climbs steeply.

A concrete intuition: if you check a genuinely null test once a day for a few weeks and stop at the first significant reading, your real false-positive rate is not 5%. Depending on how often you look, it can climb to 15%, 25%, or higher. You have quietly converted a strict test into a loose one, and you did it by being diligent.

Why the false winners feel so convincing

Early stopping does not just inflate the count of false positives, it also biases the size of the effect you report. A test that crosses significance early tends to do so on a lucky upward swing, so the measured lift is exaggerated. You ship the variant, the swing regresses toward the true (often zero) effect, and the win you celebrated never shows up in revenue. The test looked productive. The decision was noise.

What to do instead

There are two honest ways out, and they are not mutually exclusive.

Fix the horizon in advance. Decide the sample size before you launch, based on your baseline conversion rate, the minimum detectable effect you care about, 95% significance, and 80% power. Then run to that number and read the result once. Our guide on how long to run an A/B test turns that sample size into a calendar date, and the A/B test calculator does the arithmetic for you.

Use a method built for continuous monitoring. Sequential testing and always-valid inference are designed so that you can look as often as you like without inflating error, by adjusting the threshold to account for the repeated looks. These methods trade a little raw power for the freedom to peek safely.

ABTestly currently takes the fixed-horizon route. It shows you the sample size you need and a 95% confidence interval on every result, and it will not declare a winner before the test reaches significance. Until then the result reads Still collecting, which is the honest state for a test that has not gathered enough evidence yet. Sequential testing is on our roadmap, and we will label it clearly as an always-valid method when it lands so you know which rules apply.

If you want the details of how we count conversions and compute significance, the results methodology docs are public, and the SRM docs explain the chi-square check that flags a broken split before you trust any result.

The short version

Peeking feels like diligence and behaves like cheating. Looking at your dashboard is fine. Stopping the moment it turns green is where the damage happens. Decide your sample size before you launch, run to it, and read the result once, or adopt a sequential method that is honest about repeated looks. Either way, let the plan end the test, not your enthusiasm.

FAQ

Is it OK to check my A/B test results before it finishes?