Don’t crown a winner after seven purchases.

Seven purchases are very good at telling stories

One ad gets two purchases. Another gets five. The second is up 150%. Someone calls it the winner, scales the budget, and rewrites next month’s creative strategy around its opening line.

The arithmetic is correct. The confidence is not earned by the arithmetic alone. With so few outcomes, a handful of ordinary purchases can make a large relative difference. The test may contain a promising signal, but it also contains a lot of uncertainty about how much of that difference will persist.

Small businesses cannot always buy enormous experiments. Telling them to wait for thousands of conversions is often an elegant way of avoiding the actual decision. The useful question is what a small test can reasonably teach, what it cannot, and how to make the next decision without pretending the uncertainty disappeared.

That starts by separating a screening decision from a performance claim. “This is worth another test” can be justified with much less evidence than “This creative reliably improves purchase rate by 150%.” The first allocates attention. The second makes a statement about an effect.

Write down the decision before the dashboard moves

Define the thing you are changing and the people eligible for the comparison. Keep the assignment mechanism explicit. Two ads receiving different platform-optimized audiences are not automatically a randomized experiment, even if they sit in the same campaign.

Choose a primary outcome that matches the decision. For a purchase-focused test, that might be unique purchasers within a fixed window per eligible assigned visitor. Decide how returning visitors, delayed purchases, cancellations, and duplicate events will be handled. A moving denominator can produce a moving result before the creative does anything.

Set the budget, duration, and review rule in advance. A fixed-sample analysis should not be treated as valid under unrestricted checking and stopping whenever it looks exciting. Research on always-valid inference addresses this problem with procedures designed for continuous monitoring; ordinary fixed-sample intervals do not acquire that property just because a dashboard updates hourly. ^[1]

This does not mean you must keep spending through an obvious operational problem. A broken page, tracking failure, or unacceptable customer experience can justify stopping. Record that the test was interrupted for that reason rather than presenting the partial result as a clean performance finding.

Put a range beside the point estimate

Consider an invented, completed test with 250 independently assigned visitors in each arm. Assume one binary purchase outcome per visitor, a fixed observation window, no cross-arm contamination, and no outcome-based early stopping. Ad A produces two purchasers; ad B produces five.

The observed rates are 0.8% and 2.0%. B is 1.2 percentage points higher, or 150% higher relative to A’s observed rate. Those are descriptions of this sample.

Using a 95% Wilson interval for each underlying purchase rate gives approximately 0.22%–2.87% for A and 0.86%–4.60% for B. NIST documents this binomial interval method. ^[2] The ranges make the limited precision visible: the data are compatible with a broad set of purchase rates under the stated assumptions.

These are separate intervals for each rate, not an interval for the difference and not a complete two-arm hypothesis test. Overlap is not a formal rule for deciding whether an effect is significant. The point of showing them is to prevent two decimal places and a large relative lift from masquerading as certainty.

One additional purchase in A would move its observed rate from 0.8% to 1.2%, reducing B’s relative advantage from 150% to about 67%. That sensitivity does not prove B is worse or that the experiment is worthless. It tells you how much of the story currently rests on very few events.

Buy information about the biggest uncertainty

A small test is usually better at detecting a severe mismatch than measuring a subtle improvement. If visitors repeatedly misunderstand the offer, abandon a broken step, or arrive expecting a different product, fix that before funding a contest between two nearly identical headlines.

Use intermediate metrics as diagnostic evidence. A difference in qualified landing-page visits can suggest that the opening attracts a different audience. A checkout drop can point toward price or friction. Neither automatically establishes a purchase improvement. Do not promote a convenient intermediate metric to the primary outcome after the purchase result disappoints.

Qualitative review can be valuable here. Watch the page on a real phone, inspect the offer and checkout, and ask a few unfamiliar people what they believe is being sold. Their answers can reveal comprehension failures. They do not estimate population conversion rates, and they should not be dressed up as a representative survey.

Choose the next creative contrast to address a specific uncertainty. A product demonstration versus a vague lifestyle montage may answer whether people understand the product. Two slightly different background shades are unlikely to answer that question. Bigger conceptual differences do not guarantee bigger effects, but they can make the result easier to interpret.

Our creative-versus-offer analysis shows why a weak commercial proposition cannot always be repaired by another visual hook.

Make the action proportional to the evidence

After the small test, there are more options than “scale the winner” and “learned nothing.” You can retain the current version, run a larger confirmation, test the promising concept in a new context, or make a limited reversible allocation while explicitly treating the evidence as provisional.

The right action depends on the cost of being wrong. A modest shift between two reversible creative variants is different from rebuilding a product offer, signing a long production contract, or promising a client a particular return. Use stronger evidence for decisions that are harder to reverse.

If you select B because it looks promising, expect its observed advantage to be an optimistic estimate in some repeated tests. Selection favors results that happened to look good. A follow-up test is useful partly because it asks whether the promising pattern survives another sample rather than merely adding more words to the original explanation.

Keep economics visible. A higher purchase rate can accompany a lower average order contribution or a higher refund rate. Use the break-even ROAS calculator to understand the margin assumptions, then evaluate the test against those assumptions. A statistically clearer result is not automatically a profitable one.

Keep a record of what changed your mind

For every completed test, save the original question, assigned sample, outcome window, exclusions, spend, result, uncertainty method, and next decision. Note whether the test was stopped early or whether tracking changed. This is a learning record, not a trophy cabinet of winning screenshots.

Review a sequence of tests for recurring questions. Perhaps product demonstrations repeatedly earn qualified attention but the offer fails at checkout. Perhaps a new audience responds to a different use case. Those patterns can motivate a better experiment, but combining unlike tests casually does not create a clean pooled estimate.

Before running another small variation, ask what result would actually change the next decision. If no plausible outcome would alter the plan, the test may be serving as reassurance rather than research. Spend the budget on a question with consequences.

Seven purchases can earn a concept another look. They can expose a tracking problem, suggest a direction, or help decide where to spend the next small budget. What they should not do is turn a noisy afternoon into a universal theory of what customers want.