The short answer
A useful advertising experiment starts with a decision and a measurable hypothesis. It should distinguish the effect of the intended change from shifts in traffic, timing, or reporting. A higher observed ROAS alone does not establish a reliable winner.
Define the change and comparison
Specify whether the experiment tests creative, landing-page content, an offer, or a budget policy. Identify what stays comparable and how eligible traffic or units are assigned. Random assignment can help create comparable groups, but implementation, spillover between groups, and measurement quality still matter. Do not call an uncontrolled before-and-after comparison randomized.
Choose an economic outcome
Track revenue and spend, but also contribution after ads and the relevant order or customer count. If an offer changes product costs or discounts, include those changes in the model. A ratio can rise while total contribution falls.
Illustrate why the ratio is not enough
Illustrative observations, not evidence of a real test: group A spends $1,000 for $4,000 revenue; group B spends $1,000 for $4,500 revenue. Observed ROAS is 4× and 4.5×. If both have a 40% pre-ad contribution margin, contribution after ads is $600 and $800, respectively. The observed difference is $200.
Those totals do not show whether the difference is statistically reliable. Order-level variability, group sizes, assignment, and observation timing are missing. Do not derive a confidence claim from the two ratios alone.
Set the review rules in advance
Specify the primary outcome, acceptable downside, observation horizon, and method for assessing uncertainty. Account for conversion lag and refunds. Required sample size depends on the baseline behavior, variability, and effect worth detecting; there is no universal order count that makes every ROAS test decisive.
Interpret an inconclusive result honestly
A result may be consistent with both a meaningful benefit and a meaningful loss. Report that uncertainty rather than declaring a winner from the larger point estimate. Repeatedly checking and stopping at a favorable moment can distort conclusions unless the analysis method accounts for that review process.
After the test, retain the design, exclusions, dates, raw amounts, and analysis method. A successful small test also needs a separate assessment before being projected to a much larger budget or a different audience.
Write the test decision first
- Gather: Specify the tested change, allocation method, primary outcome, review horizon and acceptable downside.
- Produce: Preserve exclusions and use an analysis method appropriate to repeated checking and variability.
Decision boundary: Two aggregate ratios do not establish statistical reliability.
Next reading
Examples are illustrative, not client results or benchmarks. The calculations use the inputs shown; real-world decisions depend on your records and assumptions. Report a correction.