Attribution is not a counterfactual
A platform can report that advertising influenced a purchase without showing whether that purchase would have occurred anyway. Attribution assigns credit to observed interactions. Incrementality asks a different question: what additional outcome happened because the intervention occurred? This difference matters most when marketing reaches audiences already familiar with a brand, such as existing customers or visitors who recently abandoned a cart.
An incrementality test attempts to estimate a counterfactual: the outcome that would have happened without an ad, campaign or intervention. Because we cannot observe both realities for the same person at the same time, we compare reasonably similar groups. Random assignment can make that comparison more credible, provided implementation and measurement are sound.
A simple holdout example
Imagine an eligible audience is randomly split into two groups. The treatment group can receive an ad, while the control group is intentionally withheld from the campaign. After an appropriate observation window, 6% of treatment users purchase compared with 5% of control users. The observed absolute lift is one percentage point; the relative lift is 20% compared with the control conversion rate. That does not mean every purchase in the treatment group was caused by advertising.
Translate the difference into economic terms. If 10,000 treatment-eligible users experience a one-percentage-point lift, that corresponds to roughly 100 additional purchases under the example assumptions. Multiply by contribution per additional order, not merely gross revenue, and subtract campaign costs to evaluate economic benefit. Confidence intervals, randomization quality and spillover risks determine how much trust to place in the estimate.
Types of incrementality experiments
User-level randomized holdouts are often the clearest design when a platform can reliably assign exposure eligibility. Geo experiments compare regions with and without a treatment, typically requiring careful matching, adequate geographic diversity and adjustment for underlying trends. Time-based tests, such as turning campaigns off for a short period, can be informative but are particularly vulnerable to seasonality, promotions and demand shocks.
The right design depends on privacy restrictions, campaign reach, available data and the operational cost of withholding ads. For high-spend programs, accepting some temporary measurement sacrifice can be worthwhile if it prevents months of misallocated investment. Small businesses may need to test a narrowly defined question or aggregate data over longer periods.
The challenge of statistical power
Many tests fail because there is not enough data to detect a commercially meaningful effect. Before launch, estimate baseline conversion rate, minimum detectable lift, audience size and test duration. A tiny expected lift may require a very large sample. Teams should decide in advance what effect would justify increasing spend and what uncertainty level they will tolerate.
Do not peek at early results every day and end the experiment the moment one group looks ahead. Repeated informal checking increases the risk of misleading conclusions. Use a pre-specified evaluation plan, or a properly designed sequential-testing method if frequent checks are necessary.
What can invalidate the result?
A control group exposed through another campaign may no longer represent a clean no-ad condition. Cross-device behavior, delayed purchases, geographic spillovers and overlapping promotions can also distort results. A new product launch during one test week may overwhelm the advertising effect. Track these factors and avoid claims of perfect precision.
Another risk is selection bias. If the treatment audience contains high-value customers and the control audience contains mostly new visitors, the difference may reflect customer composition rather than the ad. Randomization and balance checks help, but the analysis still requires judgment. The outcome measure must be recorded consistently across groups.
Incremental ROAS and business value
Incremental ROAS divides incremental revenue attributed to the test intervention by relevant incremental advertising spend. Suppose an experiment estimates $30,000 of additional revenue from $10,000 of campaign spend. Its incremental ROAS is 3.0. Whether that is attractive depends on margin and other variable costs. If the test produces $30,000 in revenue but only $8,000 in contribution before advertising, the economics are negative.
Incrementality also changes budgeting discussions. Instead of asking which platform reports the highest ROAS, marketers can ask where an additional dollar is most likely to create additional profit. Experiments do not need to replace attribution dashboards; they can calibrate them. A channel’s weekly attribution data remains useful for operations, while periodic experiments establish how much of that reported effect is plausibly causal.
How to organize your first test
Choose one decision: for instance, whether retargeting spend adds enough new purchases to justify its cost. Identify the eligible audience and the outcome. Select a practical holdout design, establish a pre-test baseline, write down the hypothesis, and decide how long outcomes should be observed after exposure. Make sure sales and product teams know about the experiment so they do not introduce avoidable changes.
Run the test with stable settings and document deviations. At the end, report the estimated lift, uncertainty, spend, incremental contribution and limitations. If the result is inconclusive, say so. Inconclusive does not equal zero impact. It may mean the detectable effect was smaller than your sample could distinguish.
Moving beyond one experiment
Create an experimentation roadmap across acquisition, retargeting, brand and retention. Prioritize campaigns with high spend, disputed value or a material pending budget decision. Repeat important tests because consumer behavior and channel performance evolve. Maintain a central record of evidence so every new agency or marketer does not restart the same debate from zero.
A mature measurement system combines attribution, marketing-mix analysis where appropriate, customer cohorts and incrementality tests. No single method answers every question. The reward is not an elegant academic report; it is more confidence when deciding where to deploy scarce marketing resources.
Final takeaway
If the purpose of advertising is to create additional business, it is worth learning how to measure additional business. Incrementality testing does not eliminate uncertainty, but it directly confronts the limitation of attribution: observing a purchase after an ad is not the same as proving the ad caused it.
FAQ
Is a platform lift study enough? It can be useful if design and assumptions fit your question. Can small advertisers test incrementality? Yes, but power and cost often constrain the scope. Does a zero-significance result mean no effect? No; the confidence interval may still allow meaningful positive or negative effects. Is incremental ROAS profit? No; margins and costs remain essential.

