The A/B Test Interpreter: Turn Raw Experiment Results Into a Ship-or-Kill Decision
Why this prompt matters
Shipping a “winning” variant that was actually statistical noise wastes engineering effort maintaining a change that does nothing, and teams burned by a false positive often overcorrect by distrusting real wins later — the cost of misreading one test result compounds across every experimentation decision that follows it, not just the one bad rollout.
What we use it for
A product manager runs a two-week A/B test on a new checkout flow, sees the treatment converting at 5.7% versus 5.0% for control — a promising-looking 14% relative lift — and needs to know before Monday's roadmap review whether that result is real or noise before recommending a full rollout.
Prompt
Role: You are a senior product analyst who has run hundreds of A/B tests and knows exactly how teams misread "significant" results. Context: - Test name and hypothesis: [TEST NAME AND HYPOTHESIS] - Raw variant data: [PASTE DATA FOR EACH VARIANT: name, sample size, number of conversions/successes, and the metric definition] - Test duration: [START DATE - END DATE, OR NUMBER OF DAYS RUN] - Business context: [WHAT DECISION DEPENDS ON THIS TEST, e.g. "deciding whether to roll the new checkout flow out to 100% of users"] - Minimum meaningful effect, if known: [MDE, e.g. "we only care about lifts bigger than 2%"] Task: Analyze the results and produce: 1. A statistical significance calculation (p-value and/or confidence interval), with the method used stated explicitly 2. A plain-English verdict: Ship / Kill / Inconclusive — Need More Data 3. If inconclusive, the approximate additional sample size or time needed to reach a conclusive result at the current traffic rate and effect size 4. A check for common pitfalls: peeking bias (was this checked before the pre-registered end date?), sample ratio mismatch between variants, novelty effect risk, and multiple-comparisons risk if more than one metric was tracked 5. The practical business impact translated into revenue or user terms, not just a percentage lift Constraints: - Do not declare significance from p < 0.05 alone without first checking for sample ratio mismatch - If sample sizes are small for the observed conversion rate, flag that the result may be underpowered regardless of the p-value - Do not recommend shipping a "winning" variant if the confidence interval for the lift includes zero or a negative value - State any assumptions explicitly if required inputs are missing Output Format: ## Verdict [Ship / Kill / Inconclusive] ## Statistical Analysis [Method used, calculation shown, p-value or CI] ## Pitfall Check [Bulleted: peeking, sample ratio mismatch, novelty effect, multiple comparisons] ## Business Impact [Translated into revenue/user terms] ## If Inconclusive: What's Needed [Additional sample size or time required, with the math shown]
Result
Example: checkout flow redesign test, 14 days of data
## Verdict
Inconclusive — Need More Data. The treatment looks promising but the current sample cannot distinguish it from noise.
## Statistical Analysis
Control: 5,000 visitors, 250 conversions (5.00%)
Treatment: 5,100 visitors, 291 conversions (5.71%)
Observed relative lift: +14.2%
Method: two-proportion z-test, pooled variance
Pooled conversion rate: 5.36%
Standard error: 0.448 percentage points
z-score: 1.58
Two-tailed p-value: 0.114
95% confidence interval on the lift: -0.17pp to +1.59pp
The interval includes zero, so we cannot rule out that the true effect is flat or even slightly negative.
## Pitfall Check
- Peeking bias: confirm this analysis is happening at the pre-registered 14-day mark, not because someone checked the dashboard and saw a promising number mid-test and decided to look closer
- Sample ratio mismatch: 5,000 vs. 5,100 is close to the expected 50/50 split (49.5%/50.5%) — no red flag here
- Novelty effect: a checkout redesign is exactly the kind of change where early lift can reflect users' curiosity about a new layout rather than a durable improvement — worth re-checking this cohort's behavior at day 30
- Multiple comparisons: if conversion rate was one of several metrics tracked (e.g., also checking time-on-page, cart abandonment, average order value), the real significance threshold should be stricter than 0.05 to avoid a false positive from checking multiple metrics
## Business Impact
At current traffic (~10,000 visitors/week combined) and control's 5% baseline, a true 14% relative lift would be worth roughly 35 additional conversions per week. At an average order value of $85, that is approximately $3,000/week or $156,000/year if the effect is real. But given the confidence interval currently includes zero, shipping now risks realizing $0 of that value while still absorbing the engineering cost of maintaining the new flow.
## If Inconclusive: What's Needed
To detect a 14% relative lift (5.0% to 5.7%) with 80% power at α=0.05 requires approximately 16,200 visitors per variant — roughly 3.2x the current sample. At the current traffic rate of ~5,000 visitors per variant per two weeks, that means approximately 4-5 more weeks of the test running unchanged before a conclusive read is possible. Recommendation: extend the test rather than ship or kill based on the current data.
Most teams read A/B test results the same flawed way: check if the treatment number is bigger than control, glance at whether the dashboard says "significant," and ship. This prompt exists because that process misses the two failure modes that actually burn product teams — shipping noise as if it were a real effect, and killing a genuinely good idea because the test was underpowered to detect it.
Why the verdict has three options, not two
A binary ship-or-kill framing forces a decision even when the honest answer is "we don't know yet." Making "Inconclusive — Need More Data" a legitimate, first-class verdict — with an exact number attached to what more data would require — turns a vague feeling of uncertainty into an actionable choice: extend the test for a stated number of weeks, or accept the cost of deciding without full information.
Why the pitfall check runs on every result, including "wins"
A p-value below 0.05 is necessary but nowhere near sufficient. Peeking at results before a pre-registered end date inflates false positive rates far above the nominal 5%. A sample ratio mismatch between variants — one arm getting meaningfully more traffic than the 50/50 split intended — often signals a tracking bug that invalidates the whole test, not a real effect. Checking for these failure modes on every analysis, not just when a result looks suspicious, is what separates a rigorous read from a hopeful one.
Why business impact has to be in dollars, not percentage points
A 14% relative lift sounds compelling on a slide. Whether it's worth the engineering cost of maintaining a new checkout flow depends entirely on what that lift is worth in actual revenue — a number stakeholders can weigh against the cost of building and maintaining the change, in a way "statistically significant" never quite communicates on its own.