Test for revenue, not just conversion rate.

The four numbers on a revenue goal

Each one answers a different question, and they do not move together. A variant can lift conversion rate and lower revenue per visitor at the same time, by winning more small orders than it loses large ones. That is the reason all four are shown rather than a single headline number.

Revenue per visitor

Net revenue divided by every visitor bucketed into the variant, not by the visitors who bought. Visitors who never converted stay in the denominator, which is what makes revenue per visitor the number that tells you whether the page got more valuable overall. It is the money metric ABTestly runs statistics on.

Average order value

Net revenue divided by the number of orders. The denominator is orders, not visitors and not buyers. Average order value answers whether the orders got bigger. It is reported without an interval, so read it as a description of what happened rather than as a result you can act on alone.

Orders per visitor

Orders divided by visitors. It separates order frequency from order size. If revenue per visitor moved but average order value did not, orders per visitor is where the change lives. Descriptive, like average order value.

What counts as an order

One firing of a revenue goal that was not suppressed as a duplicate. Two consequences worth knowing before you read a report. A firing with a zero amount still counts as an order, and so does a firing with no order id attached. And because conversion rate counts distinct visitors while orders count events, the two diverge as soon as one visitor buys twice: that visitor is one conversion and two orders.

Which numbers are tested, and which are not

ABTestly does not put a verdict on a number it cannot defend. The split is deliberate and it is visible in the interface.

• Conversion rate. A full verdict: confidence, a p value, and significance at 95 and 99.
• Revenue per visitor. A verdict, presented differently. Significance is read off the lift: the result turns when the 95% interval on the relative lift clears zero. An absolute interval on revenue per visitor is shown alongside it.
• Average order value and orders per visitor. No interval, no p value, no significance. They are point values and lifts, and they are labelled that way.

Confidence appears only once both arms have at least 100 visitors and 5 conversions. Below that the numbers are still shown and the verdict is withheld. The revenue interval is withheld in two further cases: when one goal has recorded more than one currency, and when the variance needed to compute it is not available.

How a purchase reaches ABTestly

You send it. Nothing watches your checkout, parses your order confirmation, or reads your analytics for you. A revenue goal fires only when your own code calls the browser API, which is the same explicit model as the rest of the browser runtime.

1// Fire once, when the order is confirmed.
2window.__abtestly.trackGoal('purchase', {
3  value: 49.99,
4  orderId: 'A-1001',
5  currency: 'GBP'
6});

• value. The amount in major units, as a number. A bare number is also accepted as a shorthand second argument.
• orderId. One to 100 characters. This is what deduplication keys on. Without it, revenue is counted at full value and cannot be deduplicated.
• currency. Three uppercase letters. This is a shape check, not a lookup against a register of real currency codes, so a typo that is still three letters will pass.

An invalid order id or currency does not lose the sale. The field is dropped, a warning is written to the console, and the revenue still counts, because a cosmetic typo should not cost you a conversion.

There is one partial shortcut. A revenue goal can name a CSS selector to read the amount from the page when no value was passed in code. It supplies the number only. It cannot supply an order id or a currency, and it never guesses them, because an order id scraped from page text would silently corrupt deduplication. A selector only setup therefore produces revenue that cannot be deduplicated, and the goal editor flags it as fragile on sight.

Duplicate orders

A confirmation page gets refreshed, a customer hits back, a retry fires the same event twice. Every purchase is keyed on the experiment, the goal, and the order id. The first arrival wins and any later arrival with the same key is flagged.

A flagged duplicate is recorded and its amount subtracted, not thrown away. That is a deliberate choice: gross revenue and net revenue are both visible, so you can see how much duplication your integration is producing rather than wondering. Order counts exclude duplicates. Keys are retained for about 120 days, and the check runs per experiment and goal, so one real order can legitimately count in several experiments running at once.

If the deduplication check itself is unavailable, the event is counted rather than dropped. Losing a real sale is the worse failure.

Refunds

A refund is the same call with a negative amount, the original order id, and a refund flag. The amount must be negative. Nothing flips the sign of a positive number for you, and a refund without an order id is ignored in the browser and never sent.

Refunds net against revenue, and therefore against revenue per visitor and average order value. They never touch the conversion rate, the converter count, or the order count, because a refunded sale still happened. One limitation stated plainly: refund variance is not modelled in the revenue per visitor interval, so a goal with heavy refunding has a slightly optimistic interval.

Outlier control

One enterprise order in a consumer test can move revenue per visitor more than the variant did. This is a property of revenue data rather than a quirk of ours: monetization metrics are heavy tailed, and in one published analysis of ranking experiments the top 0.01% of users dominate the variance. Treatments for it differ. Some platforms winsorize the extreme tail down to a percentile; ABTestly excludes the order outright, and the next paragraph says exactly what that costs you. Results accept a maximum order value, applied when the report is read rather than baked into the stored data.

It excludes rather than trims. An order above the cap contributes no revenue and drops out of the order count entirely, and the number of excluded orders is reported so the exclusion is disclosed rather than hidden. It is off unless you set it, there is no default multiple, and you supply an absolute amount in your own currency. Because it is a reading of the same stored data, two people can look at the same experiment with different caps and both see honest numbers.

Two limits on where it reaches. The cap applies to the headline figures and the series over time. It does not apply to the segment breakdown, and it is not available on experiments pinned to the exact experiment ledger, where results are read back as pre summed aggregates that a single order cannot be retroactively removed from.

More than one currency in one goal

ABTestly does not convert between currencies, and it will not pretend that summing them is meaningful. When a single goal has recorded more than one currency code, the revenue interval and the revenue significance verdict are withheld and a warning appears on the goal.

The totals themselves are still shown. That is intentional. Hiding the number would leave you unable to see the mix that caused the problem, so you get the figures, the warning, and no verdict.

The revenue trust score

Revenue is only as good as the integration sending it, so every revenue goal carries a score out of 100 that grades the data rather than the result. It answers a question no statistical test can: did the events arrive complete?

• Missing order id, up to 40 points. The heaviest deduction, because without an order id there is no deduplication.
• Missing currency, up to 20 points.
• Missing value, up to 20 points. A revenue event that carries no amount.
• Suspect traffic, up to 20 points. Events flagged as likely automated.

The duplicate rate is reported beside the score but does not deduct from it, because catching duplicates is the system working. With no events at all the score is empty rather than zero: an unwired goal and a broken goal should not look the same.

What this measures, and what it does not

What the three engines do with a money metric

Revenue is not a coin flip, so the conversion rate machinery does not apply to it. Each engine runs a continuous method on revenue per visitor, chosen per experiment. The peeking guide covers why the choice matters.

• Frequentist. Not a proportion test. It compares the two revenue per visitor means with a normal test on their difference and reports a p value and an interval. There is no revenue series over time on this engine.
• Sequential. An asymptotic confidence sequence on the revenue per visitor difference, following Waudby-Smith and Ramdas. It is anytime valid, so the interval stays correct however often you look, which is the point of choosing it.
• Bayesian. A closed form normal posterior on each arm's mean. It reports chance to win, expected loss, and a credible interval, and no p value.

One caveat we would rather state than have you find. The second moment behind the revenue interval is computed per event rather than per visitor. It is exact when a visitor places at most one order inside the window, and slightly narrow for repeat buyers. If your customers routinely order twice in a single test, treat the revenue interval as marginally optimistic.

Where this leaves you

If your revenue lives behind a checkout you control, and a developer can add one call to the confirmation step, you get money metrics with better handling of duplicates, refunds and outliers than most tools in this category expose at all. If you need a store integration to do the wiring, or basket level detail, this is not the right tool and we would rather you knew that now. The same tradeoff runs through the whole product, and it is the same reason there is no visual editor.

Quick answers

Behavior on this page reflects the current ABTestly revenue pipeline and its public documentation. Revenue capture is opt in: nothing is recorded until your own code calls trackGoal. See the revenue metrics documentation for field level detail, and the pricing page for current plans and limits.