The A/B Test Validity Checker: Turn Test Results Into a Verdict on Whether You Can Actually Trust Them

Why this prompt matters
Most product teams treat a statistically significant result as an automatic green light to ship, but a meaningful share of declared «winning» A/B tests fail to replicate when re-run — the result of underpowered samples, novelty effects, and peeking at results before a test finishes running. Shipping a false winner doesn't just waste the engineering time spent building it; it locks in a metric regression that often goes uncaught for months, because nobody re-tests a feature that «already won».
What we use it for
A growth PM at a 200-person startup ran a two-week A/B test on a new checkout flow that showed a 12% lift in conversion with statistical significance, and the team wants to ship it to 100% of users by Friday — but the PM vaguely remembers the test period included Black Friday weekend and isn't sure whether anyone checked if the sample size was actually large enough for a 12% effect, or whether the team would have called the test differently had significance been crossed a few days earlier.
Prompt
Role: Act as a senior experimentation analyst who has reviewed hundreds of A/B tests at a product company and has seen every way a test can quietly produce a false winner. Context: [PASTE TEST DETAILS HERE — sample size per variant, conversion/metric results per variant, test start and end dates, traffic allocation method, any mid-test changes or incidents] Test goal: [WHAT METRIC WAS THIS TEST TRYING TO IMPROVE] Business context: [WHAT DECISION WILL BE MADE BASED ON THIS TEST, e.g. "rolling out to 100% of users" or "killing a feature"] Baseline conversion rate: [HISTORICAL BASELINE RATE FOR THIS METRIC, if known] Task: Assess whether this test result can actually be trusted by checking it against five common failure modes: 1. Calculate or estimate whether the sample size was large enough to reliably detect the observed effect size, flagging if the test appears underpowered 2. Check the test duration against seasonality and day-of-week effects — flag if it ran for less than one full business cycle (typically 1-2 weeks minimum) or overlapped an unusual event (holiday, sale, outage, marketing push) 3. Identify novelty-effect risk — flag if the winning variant is a visible UI/UX change where users might react to novelty rather than a durable preference 4. Check for peeking/early-stopping risk — flag if the decision appears to have been made as soon as the test crossed statistical significance rather than at a pre-committed end date or sample size 5. Check for sample ratio mismatch — flag if the traffic split between variants deviates meaningfully from the intended allocation, which can indicate a tracking or randomization bug Then produce a final verdict: TRUST THIS RESULT, TRUST WITH CAVEATS, or DO NOT TRUST — RE-RUN, with a one-sentence justification tied to whichever risk factors drove that verdict. Constraints: - Do not simply restate the observed lift as if it is automatically real — your job is to find reasons to doubt it before accepting it - If the provided information is insufficient to assess a specific risk factor (for example, no data on when significance was first crossed), say so explicitly rather than assuming it's fine - Use plain language — assume the reader understands their product but not necessarily statistics terminology; define any technical term the first time you use it - Do not exceed [MAXIMUM LENGTH, e.g. 500 words] for the full assessment Output Format: - One-line verdict badge: TRUST / TRUST WITH CAVEATS / DO NOT TRUST — RE-RUN - «Statistical Power» section (1-2 sentences) - «Timing & Seasonality» section (1-2 sentences) - «Novelty Effect Risk» section (1-2 sentences) - «Peeking Risk» section (1-2 sentences) - «Sample Ratio Check» section (1-2 sentences) - «Bottom Line» — one paragraph recommending what to do next (ship it, extend the test, re-run with a fix, etc.)
Result
VERDICT: TRUST WITH CAVEATS
Statistical Power
With 4,200 users per variant and a baseline conversion rate of 3.1%, this test had enough statistical power to reliably detect an effect of roughly 8-10%, which covers the observed 12% lift — but the margin is thin enough that the true effect could plausibly be anywhere from 4% to 20%, not precisely 12%.
Timing & Seasonality
The test ran November 20 through December 4, which includes the Black Friday/Cyber Monday weekend (Nov 24-27). [NEEDS VERIFICATION: whether holiday traffic was split evenly between variants or concentrated in one]. Holiday shopping urgency differs from normal browsing behavior, so a lift measured partly during that window may not hold in an ordinary week.
Novelty Effect Risk
The winning variant removed two form fields and added a progress indicator — a visible change users would notice immediately. Short-term lifts from visible redesigns often partially reflect curiosity rather than durable preference, and tend to decay over 2-4 weeks post-launch.
Peeking Risk
[NEEDS VERIFICATION: exact date statistical significance was first crossed]. If the team checked results daily and stopped the moment p dropped below 0.05, the real false-positive rate is higher than the reported 5% — this is one of the most common ways teams unintentionally declare false winners.
Sample Ratio Check
Traffic split was 51/49 between variants, close enough to the intended 50/50 split that this is not a concern.
Bottom Line
The result is probably directionally real — fewer form fields plausibly do improve checkout completion — but the Black Friday overlap and unconfirmed peeking risk mean the true effect size is uncertain. Recommend running one additional week in a normal, non-holiday period before committing to a permanent 100% rollout, and specifically track whether the lift holds or decays in week 3-4 to rule out a novelty effect.
«p < 0.05, ship it» is how most product teams make rollout decisions, and it's also how a meaningful share of «winning» tests fail to replicate when re-run. This prompt exists because statistical significance answers a narrower question than most teams think it does — it tells you the result probably wasn't random noise, not that the result will hold up, wasn't inflated by timing, and wasn't the product of checking results too early.
The five checks in the Task section map to the five most common ways a real-looking A/B test result turns out to be wrong. Underpowered samples produce lift estimates with huge error bars dressed up as precise numbers. Seasonality contaminates results when a test happens to run through a holiday or sale. Novelty effects inflate short-term reactions to visible redesigns that fade once the change stops being new. Peeking — checking results daily and stopping the moment they look good — quietly turns a nominal 5% false-positive rate into something much higher. And sample ratio mismatches, where the traffic split drifts from 50/50, often signal a tracking bug that's also corrupting the metric itself.
The Constraints section's instruction to actively look for reasons to doubt the result — rather than accepting it and rationalizing afterward — is the part that actually changes behavior. Most people reviewing a test they want to ship unconsciously search for confirmation, not problems. Forcing the model to check all five failure modes regardless of how good the headline number looks catches issues a motivated reviewer would skip past.
The three-tier verdict (TRUST / TRUST WITH CAVEATS / DO NOT TRUST — RE-RUN) matters because most real test reviews don't land on a clean yes or no. A result can be directionally believable while still needing another week of data before a team bets a permanent rollout on it — the middle tier gives reviewers a way to say that precisely, instead of rounding up to «ship it» because the alternative felt like killing a result the team wanted to be true.