Incrementality Tests Without a Data Team
Holdouts a Shopify store can actually run.
The first incrementality test I ran cost nothing and took an afternoon to set up. I excluded three Swedish regions from retargeting for three weeks. Those regions bought exactly as much as before. The retargeting campaign had a reported ROAS of 9. Its incremental ROAS was, within the noise, zero.
The one-sentence version
An incrementality test switches a channel off for some customers and measures what changes. No model, no vendor, no PhD. The only requirement is enough orders for the difference to be bigger than random noise.
The setup
Plan it
Try it: plan a holdout
Is your store big enough to read the answer?
Orders per week (whole store)
300
Share of customers in the holdout regions
20%
Test length
3 wk
Effect you expect to see (sales drop when ads pause)
15%
Effect you expect
15%
Noise you must beat
14.9%
180 holdout orders
Readable. Run it.
You would give up roughly 27 orders in the holdout to learn whether this channel is incremental. Cheap, compared with a year of budgeting on a guess.
Turn it off. See what happens. Then decide what it is worth.
How to run it
Five steps, one afternoon
1. Pick the channel
Highest reported ROAS, lowest new-customer share. Retargeting or brand search.
2. Pick the regions
15 to 30% of orders, similar to the rest, nothing else changing there. Write the list down.
3. Record the baseline
Orders per week in the holdout regions vs the rest, for the previous six weeks. Verified attribution gives you this per region.
4. Exclude and wait
Exclude those regions from the campaign. Three weeks minimum. Touch nothing else.
5. Compare
Did the holdout regions drop relative to the rest, by more than the noise? That drop is the incremental effect. Divide by the spend you saved for the incremental ROAS.
Why it matters
Attribution says who touched it; a test says who caused it
Even perfect attribution credits a touch that happened. It cannot know whether the sale would have happened without it. For channels that mostly reach existing customers, that is the only question that matters. A holdout answers it in three weeks.
Retargeting stayed off in those three regions for a year. Nobody noticed. Then it stayed off everywhere.
The TrueROAS version
The baseline is already there
Verified orders by region, by week, per campaign are in the dashboard. That is the baseline and the readout. Larger accounts can add True Horizon for always-on geo tests with proper statistics; most stores get the answer they need from the afternoon version.
Honest footnote
When not to bother
Prospecting campaigns to cold audiences with 90% new customers are incremental almost by definition. Test the harvesters, not the planters.
Questions people actually ask
Straight answers
Which channel should I test first?
The one with the highest reported ROAS and the lowest new-customer share. Brand search and retargeting. They are the most likely to be non-incremental and the cheapest to pause.
How do I pick the holdout regions?
Regions that together hold 15 to 30% of orders and behave like the rest of the country. Not your home city, not a region with a store or a promo running. Two or three similar regions beat one.
What do I compare?
Holdout regions vs the rest, this period vs the same weeks before the test. If the holdout regions drop by more than the noise, the channel was incremental. Order-verified attribution gives you the baseline by region.
Is this as good as a proper geo-lift platform?
No. It is 80% as good for 0% of the price, and it answers the question most stores actually have. When you spend $100k a month, buy the proper version.
See your own numbers, verified against your orders.
Free under $10k/month in revenue. 14-day trial above that. Nothing changes in your ad accounts. You will know within a week which platform has been telling stories.
"I built this because I was making $50k-a-month decisions on numbers I knew were wrong. If TrueROAS does not show you something Meta or Google got wrong in the first two weeks, tell me and I will help you cancel."
Rasmus Arvidsson
Founder, TrueROAS. Answers support himself.
Keep going