The ad platform says it worked. Prove it.

A conversion is not an explanation

Your campaign reports 210 purchases. The checkout works, the events are deduplicated, and the revenue matches the order records. Great. You have established that people bought something and the platform associated those purchases with an ad. You have not established how many purchases the ad caused.

Some customers were already going to buy. Some were reminded at a useful moment. Others might have found you through a different channel. Attribution assigns credit under a set of rules. Incrementality asks a harder question: what changed because you advertised?

That distinction matters when you are deciding whether to spend more. A campaign can look excellent at finding existing demand and be mediocre at creating additional demand. It can also produce useful effects that a narrow attribution window misses. Neither possibility can be settled by making the dashboard prettier.

The question we want to answer is specific: for this audience, offer, spend, and time window, how many additional customers did this advertising policy produce? The answer will be conditional. That is more useful than a universal claim that a channel works.

Compare assignments, not just people who saw an ad

Comparing exposed people with unexposed people sounds reasonable. It is often misleading. Delivery systems choose whom to show ads to, and people differ in how likely they are to buy before the ad ever appears. That selection can create an apparent advertising effect even when much of the difference came first.

A 2016 Facebook field-experiment white paper compared randomized results with observational estimates across 12 studies. The observational approaches often disagreed with the experimental results despite extensive controls. It is historical evidence about measurement bias, not a verdict on every campaign or today's delivery system. ^[1]

In a clean randomized design, eligible people are assigned to a group that can receive the tested advertising or a holdout that cannot receive it. You compare outcomes across the assigned groups. You do not quietly remove eligible people who were never served an impression: doing so reintroduces selection into the comparison.

This estimates the effect of making the advertising available under the tested delivery policy. It is not automatically the effect of one impression on one person. Keep that distinction in the report, especially if delivery reaches only a fraction of the eligible group.

Thirty extra buyers, with a large question mark

Consider a deliberately simplified experiment. Random assignment puts 10,000 eligible people in each arm. You measure whether each person becomes a buyer within the same fixed window, with complete outcome measurement in both groups. No repeat-order counting, uneven follow-up, or exposure-based exclusions.

The estimated difference is 0.30 percentage points, or 30 additional buyers per 10,000 people assigned. Relative to the holdout rate, that is about 16.7% lift. It does not mean all 210 buyers were incremental.

Now the part that tends to disappear from a slide: uncertainty. Using an unadjusted normal approximation for two independent binary outcomes, the estimated standard error is about 0.196 percentage points. The approximate 95% confidence interval for the difference runs from −0.08 to +0.68 percentage points.

The data are compatible with no effect and with a useful positive effect. Calling this a confirmed win overstates the result. Calling it proof that advertising does nothing also overstates it. This experiment has not separated those possibilities well enough.

The interval is an illustration, not a universal analysis recipe. Clustered assignment, incomplete measurement, unequal allocation, repeated observations, and a platform's particular estimator can change the calculation. Specify the analysis with the people running the study before the numbers arrive.

Test the decision you actually face

“Does advertising work?” is too large a brief. “Should we keep spending this amount on this audience?” is testable. “Should we increase it by 30%?” is a different experiment: the return on the additional spend may differ from the average return on everything already running.

Before launch, write down the decision and the smallest effect that would change it. If only a substantial improvement would justify the operating cost, design around detecting that improvement. Do not run an underpowered study and hope the result will become decisive through enthusiasm.

Meta's own training distinguishes A/B tests, creative tests, conversion lift, and brand lift. They answer different questions. A comparison between two ads is not automatically a comparison between advertising and no advertising. Check which study your account can actually run and what its control condition means. ^[2]

Our proposed test brief has six fields:

• Decision: the spending or campaign change the result will inform.
• Population: who qualifies, with exclusions fixed before assignment.
• Treatment: the exact advertising policy that differs between groups.
• Outcome: one primary business result, its denominator, and its observation window.
• Analysis: assignment unit, uncertainty method, sample requirement, and stopping rule.
• Decision rule: what would count as useful, inconclusive, or unacceptable evidence.

This is not paperwork for its own sake. It makes it harder to turn whichever metric moved into the metric you supposedly cared about all along.

The holdout has to survive contact with reality

A control group that receives the same offer through a duplicate campaign is not much of a control. Check overlapping campaigns, audience exclusions, tracking coverage, and whether people can cross between the groups. Other marketing can continue if the design accounts for it; the point is to know what difference the experiment identifies.

Avoid changing the landing page for one group halfway through, moving the outcome window after a weak first week, or stopping the moment a favorable number appears. If you want to inspect and stop early, use a design and analysis that explicitly support that behavior. Ordinary fixed-horizon uncertainty calculations do not become valid for repeated stopping just because the dashboard updates hourly.

Watch for business spillovers too. One person can influence another, a household can share a purchase decision, and an ad can redirect a sale between products. Randomizing individuals does not magically remove those effects. A different assignment unit may be appropriate, but it changes sample requirements and analysis.

If you cannot implement a credible holdout, be honest about the evidence you have. A carefully documented observational analysis can still inform a decision. Label the assumptions and plausible alternative explanations. Do not rebrand a before-and-after revenue chart as a randomized experiment.

Put the uncertainty back into the budget

Suppose an incremental buyer contributes $40 after the relevant variable costs, and the illustrative test costs $600. The point estimate of 30 additional buyers implies $1,200 of incremental contribution before advertising and $600 after it. That arithmetic is useful. The underlying estimate is still uncertain.

Do not multiply a noisy short-window result by twelve and present an annual forecast as if the uncertainty disappeared. Customer mix, repeat behavior, saturation, and seasonality can all change the economics. Our guide to ROAS and contribution explains the accounting side; an incrementality study addresses a separate causal question.

Keep the attributed results for day-to-day operations. Put the experimental estimate, its interval, its tested conditions, and its implications beside them. Record when the evidence would become stale enough to revisit.

The goal is not to win an argument with the platform. It is to find out what you are buying. A dashboard that gives your ads credit is useful. A study that helps you decide what to do with the next dollar is better.