Run a geo-holdout pilot before you sign a measurement vendor. Hold media in a few low-volume markets, keep it running in matched ones, and measure the gap. This planner picks the right incrementality test for your setup, then sizes the pilot: how many markets, how long, the smallest lift you can actually detect, the revenue you put at risk, and whether a result would be sharp enough to move a budget decision.
Answer five questions. The planner routes you to geo-holdout, platform conversion lift, ghost ads, or a media mix model, and tells you when each one does not fit.
Example values are loaded so you can see how it works. Replace them with your own. This is a planning estimate, not a substitute for a full power analysis.
Read it with difference-in-differences or a synthetic control against the matched markets, not a raw before-and-after. If the confidence interval crosses zero, the honest answer is that you could not tell, not that the channel is dead.
A geo-holdout tells you whether a channel is incremental. It does not tell you why, or how to make the lift bigger. That part lives in the conversion path. The same spend produces a larger incremental lift when the offer is sharper, the creative matches the intent behind the click, the landing page answers the question the ad raised, and the steps between click and purchase are few. The test and the funnel are one job. You hold spend out to learn whether it is real, and you read the click, the landing page, and the drop-off to learn what is making it work. Measuring incrementality without understanding what makes users click just tells you to cut a channel you could have fixed.
| Test | Use it when | Main tradeoff | Skip it when |
|---|---|---|---|
| Geo-holdout | You can split spend by geography and have conversions by market. You want a channel or media-level causal read without user tracking. | Needs enough markets and volume to reach statistical power. Thin per-market volume kills it. | You cannot separate media by region, or per-market conversions are too few to detect a real effect. |
| Platform conversion lift | The question is one platform's incremental value and you will accept its managed holdout (Meta, Google). | The platform runs and reports the test it is being graded on. | You need a cross-platform or vendor-neutral read you can defend to a CFO. |
| Ghost ads / PSA holdout | You want a user-level control inside a platform and have user or device-level delivery data. | Setup is technical and specific to each platform. | You lack user-level data, or the platform does not support a clean control group. |
| Media mix model (MMM) | You cannot turn anything off and have two to three years of weekly data across channels. | Correlational with priors, not a clean experiment. It explains the past, it does not prove a counterfactual. | You have a single channel question you could answer with a direct test instead. |
A geo-holdout compares the held-out markets against a matched control built from the markets you keep running. The smallest lift you can detect scales with how noisy the markets are week to week and how many market-weeks of data you gather. More markets and more weeks lower the detectable effect. Fewer, noisier markets raise it. The planner also docks the effective sample when a few markets dominate, because a handful of large markets behaves like fewer independent markets, not more. It uses a standard two-sample approximation across market-weeks at your chosen confidence and power, then reports the smallest lift that design could catch, the revenue you expose by pausing the holdout markets, and whether that result would be sharp enough to move a real budget decision. Treat the output as a plan to pressure-test, not a guarantee.
Long enough to clear weekly noise and short enough to get approved. A common window is four to six weeks of test on top of a two-week pre-period freeze on promotions. Thin conversion volume needs more time or more markets to reach the power to detect a real effect.
Enough that the holdout and control groups each carry a stable baseline, and enough total conversions to detect the lift you care about. Few markets with low conversion counts is the most common reason a pilot proves nothing. Size it before you run it, not after.
Keep your biggest markets in the control group and hold out several mid-size ones. Holding out one giant market makes the result hostage to that single market and shrinks your effective sample. The planner accounts for this, and the more concentrated your volume, the larger the lift you will be able to detect.
No. The design and the readout can be done with open-source tools and a competent analyst. A vendor buys you execution and convenience, not the ability to run the test. Know what a sound test looks like first, so you are buying execution and not permission to believe a number.
Geo-holdout when you can separate spend by region and want a clean read without user tracking. Platform conversion lift when the question is one platform's value and you accept that the platform runs the test. MMM when you cannot turn anything off and have years of weekly data. The selector above walks you through it.
You can run this yourself. If you want someone to sanity-check the markets, the window, and the readout before you spend, that is worth a conversation. More at lindseysobrinski.com.