A/B test calculator: significance, sample size and revenue per visitor
Free, no signup, no email wall. Runs entirely in your browser. Same statistics as ABTestly's results page, methodology published.
Four tools in one A/B testing calculator: check whether a finished (or running) test is actually significant, work out how many visitors a future test needs, see what lift your existing traffic could detect, and compare revenue per visitor between two variants. All four do the math the statistically honest way, with no winner below 95% confidence, every lift shown with its confidence interval and sample size, and an automatic sample-ratio-mismatch check so a broken traffic split can't masquerade as a result.
Nothing here asks for an email address and nothing here needs an account. Use it as an A/B test confidence calculator for a test that has finished, as a sample size calculator for an A/B test you have not started yet, or put it on your own site with the iframe snippet at the bottom of this page.
Is my result significant?
Counts should be unique visitors and unique converters (each person once), or the independence assumption behind the test breaks (why that matters).
A/B test sample size calculator: how many visitors do I need?
4 means 4%.
10 means a 10% relative lift (5.0% → 5.5%), not 10 percentage points (5.0% → 15.0%).
Significance is fixed at 95% (α = 0.05, two-sided). We don't offer 90% as an alternative. Multiple thresholds invite reading whichever number tells the story you want.
What lift can my traffic detect?
Already know your traffic but not your MDE? Enter your baseline rate and available visitors, and this card tells you the smallest relative lift that sample can reliably detect.
This is the inverse of the sample size calculation: given a fixed audience, what is the smallest effect you could catch? If the answer is larger than the lift you expect, you need more traffic or a higher baseline.
How to calculate revenue per visitor
Revenue per visitor is total revenue divided by everyone who entered the test, not by the people who bought. It is the number that says whether a change made you money, and it is the one a conversion rate cannot answer on its own: a variant can win more orders at a lower order value and still come out behind.
Paste the individual order values rather than a total. How wide a revenue interval comes out depends on how far apart your order values are, not only on how many orders there were, so a total and an order count cannot produce one honestly. Export the order value column from your checkout or your analytics and paste it in.
1299.00, not 1,299.00.
Revenue has no p-value here and no confidence percentage, because it has neither in the product. A mean is not a proportion: the frequentist signal is whether the 95% interval on the difference between the two revenue-per-visitor figures excludes zero. Below 100 visitors or 5 orders on either variant, the card reports your counts and withholds the interval instead of printing a number that wide.
Embed this calculator on your site
The calculator is free to use and free to put on your own site. Paste the snippet below into any page and it loads as an iframe with no header, no navigation, no chat bubble and nothing to sign up for. The embedded copy carries a visible link back to this page, and that link is the only thing we ask for it.
The embedded copy runs the same four tools and the same statistics as this page. allow="clipboard-write" is what lets its copy buttons reach the clipboard from inside your frame; drop it and everything else still works. The height is a starting point, so adjust it to your layout.
How this calculator works
Significance is a two-sided, two-proportion z-test with pooled standard error, the same test behind ABTestly's Confidence column. The lift is relative, and its 95% confidence interval uses the unpooled standard error exactly as published in how we count results. The SRM check is a chi-square goodness-of-fit test flagged below p = 0.001, and it stays quiet until 500 total visitors so it can't false-alarm during warm-up. Large platforms all sit in this conservative range rather than at the usual 0.05: Microsoft's experimentation platform uses the same chi-square test at p < 0.0005, stricter than ours. The KDD 2019 taxonomy of SRM causes is the reference for what a flag actually tells you to go and look at. Sample size uses the standard two-proportion power calculation. We refuse to render a verdict when any expected cell of the implied 2×2 falls below 5 visitors, because the normal approximation breaks down before then.
Revenue per visitor is a mean, so it takes a different route. Each variant gets a 95% interval of RPV ± 1.959964 × √(variance / visitors), with the variance read off the order values as Σx²/n − (Σx/n)², and the two are compared with the unpooled standard error √(SE² + SE²). That is the same composition the product's results route and results page use. There is deliberately no p-value and no confidence percentage attached to it, in the product or here, because the frequentist signal for a mean is simply whether the interval on the difference excludes zero.
This is a fixed-horizon test, so plan once and look once. Repeatedly checking interim results and stopping when one looks significant inflates the false positive rate, and how far depends on how often you look. Our own simulation puts a nominal 5% at 12.9% for a team checking weekly, 22.5% every other day, and 28.0% daily. Independently, Evan Miller puts the ceiling at 26.1% for checking after every single observation, and shows that ten looks turn a reported 1% into a real 5%. Looking is not what breaks it, though. Acting on what you see is. That is the "peeking problem." Decide your sample size up front (using the card above), run to it, then judge once. More on this from the docs.
Built by ABTestly. Every function on this page is verified against the production implementation by an automated drift-guard suite that imports the worker's source directly. If the product and this page ever disagreed, our test suite would fail before the page shipped. Nothing you type leaves your browser.
Why this is stricter than most online significance calculators
Significance calculators on the internet vary more than the underlying math should permit. Most are flexible about how they let you read a result; this one isn't, and that's deliberate. A calculator's job is to refuse to tell you what you want to hear when the data doesn't support it.
| Most online calculators | This one | |
|---|---|---|
| Significance threshold | Often 90% / 95% / 99% selectable, or three rows of decisions | 95% only. Multiple thresholds invite reading whichever number tells the story you want. |
| A 94%-confident result | Often reported as "significant at 90%" by relaxing the threshold | "Below 95%, no verdict." We won't let you talk yourself into a winner. |
| Peeking warning | Rarely mentioned; many implicitly encourage continuous checking | One-line caveat on every "no verdict" result, linking the methodology |
| Broken traffic splits (SRM) | Not detected, so a 5,000 vs 4,200 test would still report a winner | Auto-checked (chi-square, p < 0.001, the conservative range large platforms use) and flagged before any verdict |
| Tiny sample sizes | Will render a confident-looking verdict on 8 visitors | Refuses below 5 expected conversions or non-conversions per arm |
| Difference shown as | Percentage points (+0.12 pp), which undersells real business impact | Relative lift (+25%) with its 95% CI and sample size, always together |
| Math source | Often hand-rolled, or transcribed from a spreadsheet | Test-pinned to the same code that powers ABTestly's results page |
Where we don't (yet) match other calculators: we only handle two variants. ABTestly the product is A/B-only today, so adding multivariate (MVT) support here for a product that doesn't do it would mislead either direction. When MVT lands in the product, it'll show up here too, with proper Tukey HSD math, not a normal-curve approximation. Sequential monitoring is not a gap: ABTestly the product offers a sequential, anytime valid method on Pro for teams who want to peek safely while a test runs. This calculator implements the fixed-horizon frequentist method on purpose.
Quick answers
What does statistically significant mean in an A/B test?
A result is statistically significant when it would be unlikely to appear by chance if the variants truly performed the same. This calculator uses a two-tailed, two-proportion z-test and calls a result significant when confidence reaches 95% (p < 0.05), the standard CRO convention. Below that bar the honest verdict is "still collecting", not "almost a winner".
Why is the threshold fixed at 95%?
Because offering 90% or 99% side by side encourages reading whichever number tells the story you want. ABTestly's frequentist engine applies the same fixed 95% bar and shows "Still collecting" until a test clears it; the calculator behaves identically.
What is sample ratio mismatch (SRM)?
SRM means the traffic split you observed differs from the split you configured by more than chance allows (chi-square test, flagged below p = 0.001). It usually means something is systematically dropping one arm (a broken redirect, a variant erroring on one browser), and the test's numbers can't be trusted until it's fixed. This calculator checks it automatically from your visitor counts. Full guide.
How many visitors does an A/B test need?
It depends on your baseline conversion rate and the smallest relative lift worth detecting. A worked example: detecting a 10% relative lift on a 4% baseline at 95% confidence and 80% power needs 39,475 visitors per variant. The sample-size calculator above computes it for your numbers.
What lift can my available traffic detect?
The inverse MDE calculator answers this. Enter your baseline conversion rate and available visitors per variant, and it computes the smallest relative lift that sample can reliably detect at 95% confidence and your chosen power (80% or 90%). If the detectable lift is larger than the effect you expect, you need more traffic.
How do you calculate revenue per visitor?
Revenue per visitor is total revenue divided by the visitors who entered the test, not by the visitors who bought. Paste each variant's order values into the revenue card above and it reports both figures with a 95% confidence interval on each and on the difference between them. It asks for the individual order values rather than a total because the width of the interval depends on how far apart those values are, which a total cannot tell it.
Can I use this as an A/B test confidence calculator?
Yes. The significance card is an A/B test confidence calculator: give it visitors and conversions for both variants and it reports the confidence, the p-value, the relative lift with its 95% interval, and a verdict that stays at no verdict until confidence reaches 95%. The revenue card is deliberately different, because revenue per visitor carries no confidence percentage in this product, only an interval that either excludes zero or does not.
Can I embed this A/B testing calculator on my own site?
Yes, and the snippet is on this page under Embed this calculator on your site. It loads an iframe with no header, no navigation and no signup, and it carries a visible link back here. Readers can use it as an A/B testing sample size calculator before they start, or as a significance check on a test they have already run.
Found a number you disagree with? Email [email protected] with your inputs. If the calculator is wrong we'll fix it publicly. The ABTestly team.
Related guides
The statistics behind the numbers above, explained in plain English:
- Minimum detectable effect (MDE), explained: how the smallest lift worth detecting sets your sample size.
- How long should you run an A/B test?: turning that sample size into a real end date.
- The peeking problem: why stopping the moment you see 95% inflates false positives.
- Optimizely sample size calculator: the same calculation with the formula written out, and where it differs from Optimizely's own tool.