A/B test a Shopify discount: 10% off vs 20% off vs none

Most stores pick a discount depth by habit. 20% because last year was 20%, 15% because the margin says so, 30% because the competitor did it. Almost nobody measures whether the extra ten points bought any extra orders, and the stores that do usually measure it the wrong way: this week against last week, Black Friday against the Tuesday before, one collection against another. Those comparisons move for a hundred reasons that have nothing to do with the discount.

The only clean answer is to show different discounts to different visitors at the same time, on the same products, and compare. That is what the A/B test in SafeSale does, and since it runs inside the sale you already set up, it costs you a checkbox and a bit of patience. This post explains how it works, what it measures, how to read the result and when to use Shopify’s own Rollouts experiment instead.

What you can test

An A/B test in SafeSale is attached to one checkout-discount sale. The sale’s own discount rule becomes variant A; you add between one and four more variants, each with its own rule or with no discount at all, and a share of visitors. Typical set-ups:

• Does the sale pay for itself? A: 20% off (50%), B: no discount (50%). The oldest question in retail, finally with an answer for your store.
• How deep does it need to be? A: 20% off, B: 10% off, C: 30% off, a third each. Often 10% converts almost as well as 20% and makes far more money per visitor.
• Percentage or price point? A: 25% off, B: fixed price ending in .99. Same average depth, different psychology.

Everything else about the sale stays the same for all variants: the products, the dates, the hard cap, the typed confirmation above your threshold. Per-variant overrides (this SKU 30%, that one 10%) still apply inside variant A; the other variants use their own single rule.

How a visitor ends up in a variant

Shopify applies an automatic discount per cart, and a cart has no idea who is looking at it. So SafeSale adds two small pieces to make the split possible, both installed and removed automatically:

1. An app embed in your theme (invisible, enabled once from SafeSale’s Settings). When a sale with a test is live, it reads the test definition from a shop metafield, rolls a weighted die once per browser, and stores the result in a first-party cookie for the length of the sale. It then copies the same value into the cart as a hidden attribute, and re-copies it if the cart is emptied and rebuilt. No visitor data leaves the browser in this step.
2. The discount function (the same Shopify Function that already applies your SafeSale discounts) reads that cart attribute and applies the matching variant’s rule. A visitor with no attribute, or with one from a different sale, simply gets variant A, the sale’s normal rule. A no-discount variant gets nothing: regular prices in cart and checkout, and the SafeSale badge and countdown blocks hide themselves.

Because the split happens in the cart rather than on a product page, it works on every page, on accelerated checkouts and on carts built from a shared link. The one thing it needs is the embed: if a theme does not load it (a headless storefront, or Canvas-edited themes for now), everyone falls back to variant A and the test collects nothing, which SafeSale tells you on the sale page.

What gets measured, and by what

Counting is done by a Shopify web pixel extension, the sandboxed kind that Shopify loads on the storefront and in checkout and that honours the customer privacy settings. It records exactly two things:

• Visitors: one per browser session that loaded a page while assigned to a variant.
• Orders: each completed checkout with the variant that was on the cart, plus the order subtotal and currency.

It sends the shop domain, the sale id, the variant letter, an order id and an amount. No customer names, emails or addresses, no product details, no IP. The SafeSale sale page then shows, per variant: visitors, orders, conversion rate, revenue and average order value, and one sentence that tells you where you stand.

Reading the result

That sentence comes in three flavours:

• Collecting data. At least one variant has fewer than 100 visitors or fewer than 5 orders. Nothing is compared yet; early numbers are noise.
• No clear winner yet. Every variant has enough data but the best and second-best conversion rates are not far enough apart. The page shows the current confidence so you can see it climbing (or not).
• Variant X is winning. The lead on conversion rate passes 95% confidence (a two-proportion z-test; the same arithmetic every A/B tool uses).

One more line matters: if a different variant makes the most revenue per visitor, SafeSale says so. A deeper discount converting slightly better is not automatically the winner; 10% off at a 2.1% conversion usually beats 20% off at 2.3%. You decide which metric your margin cares about.

When you are happy, press Apply to everyone on the variant you want. The test ends, the counters stay on the page, and the live discount is updated so every visitor gets that rule for the rest of the sale. Choosing the no-discount variant ends the sale. There is no second deploy and no new discount: the same automatic discount keeps running with a new configuration.

How long it takes

Statistics are unkind to small stores. To detect a difference between a 2.0% and a 2.5% conversion rate at 95% confidence you need roughly 6,000 visitors per variant; between 2.0% and 3.0%, about 1,800. Three variants triple that. Our rules of thumb:

• Test two variants unless you have the traffic for more.
• Run for at least one full week so weekdays and weekends both count.
• Test before the big event, on a normal week, then run the event with the winner. A test during the event itself is tempting but often ends when the event does.
• Do not peek and stop early on a good day; wait for the verdict or for the planned end date.

Where Shopify Rollouts fits

Shopify’s Rollouts can also run an experiment that includes a discount: it shows a discount to a share of buyers and compares them to a control, and it can bundle a theme or checkout change into the same treatment. Use it when the question is about the launch as a whole (new homepage plus a discount, yes or no). Use SafeSale’s test when the question is about the discount itself (how deep, which products, percentage or price point) and you want the answer next to the sale with a one-click way to act on it. The two do not interfere: SafeSale keeps its discount out of any rollout, and the sale page warns you if a rollout happens to include it anyway.

Honest limits

• Checkout-discount sales only. A price rewrite changes the stored price for everyone; it cannot be split.
• Assignment is per browser. The same person on a phone and a laptop can land in two variants, like with any A/B tool without login.
• Where consent is required, visitors and orders are counted only for people who accepted analytics cookies. The split itself applies to everyone, so the report is a representative sample, not a full count.
• The test needs the Pro plan; the sale itself works on both plans.

If you have been arguing about 15% versus 20% for years, this is the cheapest way to end the argument. Set it up in the sale form, turn on the embed once, and let your own customers vote.