SafeSale · Blog

Applying an A/B test winner to a Shopify sale: what changes

What happens the moment you press Apply to everyone on a SafeSale A/B test: the discount, the storefront labels, the sale rules, the numbers and the audit log.

The earlier post on testing discount depth stopped at the moment the report says We have a winner. That is where most A/B testing guides stop too, as if reading the result were the end of the job. It is not. Somebody has to press the button, and the button in SafeSale is a real action against your store: it changes the discount every shopper gets for the rest of the sale. This post is about that one click. What exactly changes, in what order, what stays, and the two cases that surprise people.

Where the button is and when it exists

Open a sale that has an A/B test and the A/B test section lists each variant in a table: letter, name, what it does, split, visitors, orders, conversion, revenue, average order value. At the end of each row there is a tertiary button. For a variant with a discount it reads Apply to everyone; for the control, the variant with no discount, it reads End sale (no discount). Press either and the row asks you to confirm with Yes, apply to everyone or Yes, end the sale, next to a Keep testing escape. Nothing happens until the confirmation.

The buttons are only there while the sale is active. A scheduled sale has no traffic to report on yet, and an ended sale has nothing left to apply a winner to, so in both cases the table shows the numbers and no buttons. The server enforces the same rule: a conclude request for a sale that is not live is refused, so a stale tab cannot change a sale that has already ended.

One thing the button does not require is a statistical winner. The verdict banner above the table says Collecting data, We have a winner or Undecided, based on at least 100 visitors and 5 orders in every bucket and 95 percent confidence, but the Apply button is live on every row regardless. The app tells you what the numbers support; it does not stop you from acting early, because sometimes the calendar matters more than the confidence interval. The Friday before a campaign ends is a fine reason to pick the variant that is ahead and move on.

What happens when you apply a discounted variant

The confirmation posts one request to the sale, naming the winning letter, and the server does the following in order.

  1. The experiment is unpublished. The storefront reads its bucket assignment from a shop metafield that tells it which sale is testing and how the traffic is split. That metafield is cleared first, so from the next page load no new visitor is assigned a bucket and the storefront stops writing the test cookie. Browsers that still carry the cookie simply find no test to belong to.
  2. The automatic discount is rewritten. A SafeSale A/B test runs on a checkout discount, and that discount carries the per-bucket rules in its configuration. The winning rule becomes the default rule, the bucket list is removed, and the hard cap you set as a safety limit stays in place. From that moment every cart gets the winning discount, whether or not the browser was ever in the test.
  3. The storefront labels are republished. The sale badges and price labels that the theme block shows are driven by the same data, so they are rewritten to describe the winning variant only. During the test a shopper in bucket B saw the B label; now everyone sees the same one.
  4. The sale rules are updated. The sale record in SafeSale keeps a rule per bucket while a test runs. On conclusion the winning rule is written as the default and the others are dropped, so the sale now looks like an ordinary single-rule sale on the sale page, in the preview and in any later edits.
  5. The result is stamped on the test. The experiment config records the winning key and the date of conclusion. That is what turns the section heading from A/B test into A/B test result and puts the green banner at the top: the variant name, its letter, and the date it was applied to everyone.
  6. The change log gets a line. The sale audit log, covered in the audit log post, records which variant won and that it now applies to everyone, with the staff member who pressed the button. Reading the log later tells you exactly when the switch happened, which matters when you compare revenue before and after.

None of these steps touches product prices. A test runs on a checkout discount precisely because changing prices per visitor is impossible, so the price fields in your admin are the same before and after the click. If the sale also has compare-at prices showing, they stay as they were, because they belong to the sale, not the test.

What happens when the control wins

This is the case people do not plan for. If the variant with no discount has the best revenue per visitor, the honest conclusion is that the sale is costing you money, and the button on that row says so in plain words: End sale (no discount). Choosing it writes a change log line that the no-discount variant won and the sale is ending for everyone, then hands over to the normal end-of-sale process. The discount is removed, labels come off, and anything the sale changed is restored the same way a sale that reached its end date would be. The test numbers stay on the sale page as evidence.

It is worth sitting with the decision for a moment rather than flinching from it. A control win on revenue per visitor with similar conversion means shoppers were buying anyway and the discount only trimmed the margin. A control win driven by a lower conversion rate in the discounted buckets is stranger and usually means something else was going on, for example the discounted price landing on an awkward number or a label that read as suspicious. Either way, the numbers are kept and the end-of-sale restore is the same safe path every SafeSale sale takes.

What stays and what you can still change

  • The numbers. Visitors, orders, conversion, revenue and average order value per variant remain on the page under A/B test result, frozen at the moment of conclusion. The sales pixel stops counting because there is no longer a test to attribute to.
  • The sale itself. It is still a sale, with its end date, its targets and now a single rule. You can edit it, extend it or end it early like any other sale. The one thing you cannot do is reopen the test.
  • The safety limit. The hard cap on the automatic discount is carried over unchanged, so a 30 percent winner is still bounded by whatever ceiling you set when the sale was created.
  • The audience. Everyone gets the winner, including browsers that were in a losing bucket a minute ago. A shopper who had a 10 percent cart in one tab and reloads sees the 20 percent winner. That is the point, but it is also why you should not conclude in the middle of an email send that quotes a specific percentage.

Timing the click

Three practical rules. First, conclude when the report says winner and both buckets have cleared the minimums, unless the calendar forces your hand; the thresholds exist so that a lucky Tuesday does not pick your discount for the month. Second, look at revenue per visitor and not just conversion, because the deeper discount nearly always converts better and the question is whether it paid for itself. Third, read the average order value column before applying a winner that is only ahead on conversion: if the order size dropped, the winner may be selling the same units for less.

After the click, open the storefront in a private window and add a product to the cart. You should see the winning discount and the winning label with no test cookie set. Then check the change log line and note the time. When you later compare the sale against a previous one in the Shopify reports, that timestamp is the boundary between the testing period and the applied period, and mixing the two is the most common way to misread a sale.

An A/B test that is never concluded is a test that quietly keeps giving a third of your shoppers the worse deal. The button is small and it is there so that the test has an ending. Press it, read the banner, and let SafeSale get on with the rest of the sale.

Frequently asked questions

Can I apply a winning variant while the Shopify sale is still running?

Yes, that is the only time you can. The Apply to everyone and End sale (no discount) buttons appear while the sale is active. Pressing one ends the test and gives every visitor the chosen variant for the rest of the sale. Once the sale has ended the buttons are gone, because there is nothing left to apply the result to.

Do I lose the test numbers when I apply the winner?

No. The visitors, orders, conversion rate, revenue and average order value per variant stay on the sale page under the heading A/B test result, with a banner naming the variant that was applied and the date. The split percentages and the sales pixel stop being used, but the counts collected up to that moment are kept.

What if the control with no discount wins?

Choosing a variant that has no discount ends the whole sale. SafeSale records in the change log that the no-discount variant won and is ending the sale for everyone, then runs the normal end-of-sale restore: prices or the automatic discount go back, compare-at values are cleared and the storefront labels are removed.

Can I undo applying a winner and go back to testing?

No. Concluding a test is a one-way step. The experiment is unpublished from the storefront and the sale continues as an ordinary sale with the chosen rule. If you want to test again, end this sale and create a new one with a new A/B test.