A/B testing guides

Short, honest guides for people who actually run experiments. They cover the statistics that decide whether a test result is real: how big a test needs to be, how long to run it, and the mistakes that quietly turn noise into a false winner.

• The peeking problem

  Why checking an A/B test every day and stopping the moment it hits significance inflates your false-positive rate far above 5%, and what to do instead.

• Minimum detectable effect (MDE), explained

  What MDE means, how it drives sample size, how to pick a sensible one, and a worked example: detecting a 10% lift on a 4% baseline needs about 39,475 visitors per variant.

• How long should you run an A/B test?

  Duration is the sample size you need divided by your traffic. Run whole weeks to capture seasonality, hold a floor of one to two business cycles, and never stop early.

Want the numbers rather than the theory? The free A/B test calculator sizes a test from your baseline, MDE, significance, and power. For how we count conversions and compute results, the results methodology docs are public.