SafeSale · Blog

What the SafeSale A/B test pixel records (and skips)

How a Shopify web pixel counts visitors and orders per sale variant: the cookie, the cart attribute, the 2 KB event, what gets deduplicated and what is never sent.

An A/B test report is only as honest as the thing that feeds it. When SafeSale shows that variant B made 14% more revenue per visitor than variant A, two numbers sit under that claim: how many browsers saw each variant, and how much each group bought. Both come from a small Shopify web pixel. This post explains what that pixel does on each page, what it sends, what the server throws away, and where the counts can be lower than you expect.

If you have not run an experiment yet, the guide to testing discount depth covers setting one up. This post assumes a sale is running with two or more variants and you are looking at the report.

Two pieces, two jobs

SafeSale splits the work between a theme app embed and a web pixel, because Shopify lets each of them see different things.

The embed runs in your theme on every storefront page. It assigns the shopper to a variant, remembers that choice, shows the right label and badge, and tells the cart which variant applies. It can touch the page and the cart, but it cannot see checkout.

The pixel runs in the sandbox Shopify provides for web pixels. It cannot touch the page at all, but it does receive the standard customer events, including the one that fires when a checkout completes. Its only job is to report two kinds of events to SafeSale: a visit and an order.

Assignment: a cookie and a cart attribute

When a page loads during a running experiment, the embed looks for a cookie named _safesale_ab. Its value is the sale ID and the variant key, for example 42:B. If the cookie names the current sale and a variant that still exists, the shopper keeps it. If not, the embed picks a variant with a weighted random draw using the splits you configured and writes the cookie for 30 days, scoped to the whole site, with SameSite=Lax and the Secure flag on HTTPS.

The embed then puts the variant key on the <html> element as a data attribute, which is what your theme can use to style each variant, and writes the same sale-and-variant value into a cart attribute, also named _safesale_ab. The cart attribute is the handoff to pricing: the discount Function reads it to decide which variant depth to apply. If the cart is updated by something else and the attribute goes missing, the embed writes it again on the next cart update event.

Two cases are skipped on purpose. The embed does nothing inside the theme editor, so you can preview a theme without being assigned, and it does nothing once the sale end time has passed.

The visit event

On every page_viewed event the pixel reads the cookie. If there is none it waits a moment and looks twice more, 1.5 seconds apart, because the embed may set the cookie slightly after the page starts loading. With a cookie in hand, the pixel checks a session flag keyed to that sale and variant. If the flag is already there the pixel stops: the browser was counted for this variant earlier in this tab session. If not, it sets the flag and sends one visit event.

So a visitor in the report is one browser session on one variant of one sale. Ten page views in the same session count once. Closing the tab and coming back tomorrow with the same cookie counts again, on the same variant, which is what you want: the shopper saw the price again and had another chance to buy.

The order event

On checkout_completed the pixel looks for the variant in two places, in order. First the cart attribute, because that is what the discount Function used to compute the price. Second the cookie, as a fallback. It then sends the order ID, falling back to the checkout token if the order ID is not yet available, the subtotal (falling back to the total), and the currency code.

That is the whole payload: store domain, sale ID, variant key, event type, order ID, amount, currency. There is no field for the customer, the line items or the address, and the pixel does not read them.

What the server refuses

The endpoint that receives pixel events is public. It has to be, because the pixel sandbox cannot sign requests with a secret. SafeSale compensates by being strict about what it accepts and lazy about what it tells the sender.

  • Any request body over 2,000 bytes is dropped before parsing.
  • The shop must look like a .myshopify.com domain, the event type must be exactly visit or order, an order amount must be a plain decimal with at most two places, and the currency must be three capital letters. Anything else is discarded.
  • The shop must still have SafeSale installed, the sale must be active or ending, the experiment must be running, and the variant key must exist on that experiment. Events for a paused experiment or a sale that already ended are ignored.
  • Every response is an empty 204, whether the event was recorded or not. A script poking at the endpoint learns nothing from it.

Orders have one more guard. Each recorded order is stored under its order ID with a uniqueness constraint, so the same order reported twice counts once. A shopper refreshing the thank-you page cannot double a variant.

Why visitors can run low

The number people question most is visitors, and the usual reason is consent. Shopify starts web pixels only after the shopper has agreed to analytics in regions where your Customer Privacy settings require it. The embed still runs, so the shopper is still assigned and still sees the right price, but no visit event is sent until the banner is answered. If a large share of your traffic is in those regions and few people accept, visitors will undercount while orders, which tend to come from shoppers who stayed long enough to answer, undercount less. Conversion rate then looks better than it is, by a similar amount on every variant. The comparison between variants still holds; the absolute rate does not.

The second reason is permissions. The pixel needs two scopes, write_pixels and read_customer_events, that SafeSale added after its first release. A store that installed before that sees a banner in the app asking to reopen it and grant the new permissions. Until that happens the pixel is not registered, and the experiment runs with no counts at all. The report labels explain this in a sentence: visitors are browsers that loaded a page with the embed, orders are checkouts reported by the pixel.

Third, ad blockers and privacy browsers sometimes stop web pixels entirely. There is nothing an app can do about that, and it affects both variants equally.

Reading the report with this in mind

The report table shows, per variant, visitors, orders, conversion and revenue, and it points out when one variant makes the most revenue per visitor even though another converts best. Knowing where the numbers come from suggests a few habits.

  • Compare ratios, not totals. Visitors are undercounted by consent in the same proportion on every variant, so conversion and revenue per visitor stay comparable even when the raw counts look small.
  • Do not change splits mid-test. Shoppers with a cookie stay on their variant, so a split change only affects new browsers, and the report cannot tell the two groups apart.
  • Give it sessions, not days. A visitor is a session, so a quiet store needs longer to collect enough of them for a difference to mean anything.

The short version: the pixel counts one session per variant and one order per order ID, sends nothing personal, and only counts at all once the shopper has said yes to analytics. Everything in the report follows from those three facts.

Frequently asked questions

Why does my SafeSale experiment show orders but very few visitors?

Visitors are counted by a web pixel, and Shopify only starts web pixels after the shopper has given analytics consent in regions where the Customer Privacy banner applies. Orders also come from the pixel, but a shopper who buys has usually been on the site long enough to answer the banner. If the gap is large, check that SafeSale has the analytics permissions (the app shows a banner asking you to reopen it if not) and that your consent banner is actually being shown.

Does the pixel send customer names, emails or addresses?

No. An event carries the store domain, the sale ID, the variant letter, the event type, and for orders the order ID, the order total and the currency code. There is no field for anything personal, and the server drops any request over 2,000 bytes or with a field that does not match its expected shape.

What happens if the same order is reported twice?

It is counted once. Each order row is keyed by the Shopify order ID, and a second event with the same ID hits a uniqueness constraint and is ignored. Reloading the thank-you page, or the pixel firing twice, does not inflate a variant.

Which variant does an order count towards if the shopper switched buckets mid-visit?

The one the discount was actually computed with. The embed writes the assigned bucket into a cart attribute, and the discount Function reads that attribute. At checkout the pixel prefers the cart attribute over the cookie, so revenue lands on the variant that produced the price the shopper paid.