A test gets called a winner on day four because variant B is up 18%, the team ships it, and the lift quietly evaporates over the following month. This isn't bad luck — it's the predictable result of calling a test before it had enough data to mean anything.
The sample size nobody calculates first
Before a test starts, it's possible to calculate roughly how many conversions each variant needs to detect a real difference at a given confidence level. Most tests never get this calculation done, which means there's no actual answer to "how long should this run" beyond a guess.
Peeking daily inflates false positives
Checking results every day and stopping the moment one variant looks ahead is one of the most common ways a test produces a false winner. Statistical significance calculated on a partial, self-selected stopping point isn't the significance the test was designed to measure — it's closer to picking the best of several coin flips and calling it a pattern.
What to do when traffic can't support a real test
Below a certain traffic level, a proper A/B test on a single page simply won't reach significance in a useful timeframe. The options at that point are: run the test across a longer window and accept it, test a bigger, more obviously different change so any real effect is larger and easier to detect, or drop split testing for that page and rely on qualitative signal — session recordings, on-page surveys, and direct comparison against a documented baseline — instead.
Where to start
- Calculate the required sample size before launching, not after
- Pick a stopping date in advance and hold to it, rather than stopping on a good day
- If traffic can't support a real test, say so and pick a different method instead of running a test that can't reach a real answer
- Re-check any "winning" test again a month later against a fresh baseline
The discipline isn't exotic. It's mostly deciding, before the test starts, what would actually count as evidence — and then not moving that line once the results start coming in.