Der A/B-Test-Interpreter: Verwandeln Sie rohe Experimentergebnisse in eine Ship-or-Kill-Entscheidung
Warum dieser Prompt wichtig ist
Shipping a “winning” variant that was actually statistical noise wastes engineering effort maintaining a change that does nothing, and teams burned by a false positive often overcorrect by distrusting real wins later — the cost of misreading one test result compounds across every experimentation decision that follows it, not just the one bad rollout.
Wofür wir ihn verwenden
A product manager runs a two-week A/B test on a new checkout flow, sees the treatment converting at 5.7% versus 5.0% for control — a promising-looking 14% relative lift — and needs to know before Monday's roadmap review whether that result is real or noise before recommending a full rollout.
Prompt
Role: You are a senior product analyst who has run hundreds of A/B tests and knows exactly how teams misread "significant" results. Context: - Test name and hypothesis: [TEST NAME AND HYPOTHESIS] - Raw variant data: [PASTE DATA FOR EACH VARIANT: name, sample size, number of conversions/successes, and the metric definition] - Test duration: [START DATE - END DATE, OR NUMBER OF DAYS RUN] - Business context: [WHAT DECISION DEPENDS ON THIS TEST, e.g. "deciding whether to roll the new checkout flow out to 100% of users"] - Minimum meaningful effect, if known: [MDE, e.g. "we only care about lifts bigger than 2%"] Task: Analyze the results and produce: 1. A statistical significance calculation (p-value and/or confidence interval), with the method used stated explicitly 2. A plain-English verdict: Ship / Kill / Inconclusive — Need More Data 3. If inconclusive, the approximate additional sample size or time needed to reach a conclusive result at the current traffic rate and effect size 4. A check for common pitfalls: peeking bias (was this checked before the pre-registered end date?), sample ratio mismatch between variants, novelty effect risk, and multiple-comparisons risk if more than one metric was tracked 5. The practical business impact translated into revenue or user terms, not just a percentage lift Constraints: - Do not declare significance from p < 0.05 alone without first checking for sample ratio mismatch - If sample sizes are small for the observed conversion rate, flag that the result may be underpowered regardless of the p-value - Do not recommend shipping a "winning" variant if the confidence interval for the lift includes zero or a negative value - State any assumptions explicitly if required inputs are missing Output Format: ## Verdict [Ship / Kill / Inconclusive] ## Statistical Analysis [Method used, calculation shown, p-value or CI] ## Pitfall Check [Bulleted: peeking, sample ratio mismatch, novelty effect, multiple comparisons] ## Business Impact [Translated into revenue/user terms] ## If Inconclusive: What's Needed [Additional sample size or time required, with the math shown]
Ergebnis
Example: checkout flow redesign test, 14 days of data
## Verdict
Inconclusive — Need More Data. The treatment looks promising but the current sample cannot distinguish it from noise.
## Statistical Analysis
Control: 5,000 visitors, 250 conversions (5.00%)
Treatment: 5,100 visitors, 291 conversions (5.71%)
Observed relative lift: +14.2%
Method: two-proportion z-test, pooled variance
Pooled conversion rate: 5.36%
Standard error: 0.448 percentage points
z-score: 1.58
Two-tailed p-value: 0.114
95% confidence interval on the lift: -0.17pp to +1.59pp
The interval includes zero, so we cannot rule out that the true effect is flat or even slightly negative.
## Pitfall Check
- Peeking bias: confirm this analysis is happening at the pre-registered 14-day mark, not because someone checked the dashboard and saw a promising number mid-test and decided to look closer
- Sample ratio mismatch: 5,000 vs. 5,100 is close to the expected 50/50 split (49.5%/50.5%) — no red flag here
- Novelty effect: a checkout redesign is exactly the kind of change where early lift can reflect users' curiosity about a new layout rather than a durable improvement — worth re-checking this cohort's behavior at day 30
- Multiple comparisons: if conversion rate was one of several metrics tracked (e.g., also checking time-on-page, cart abandonment, average order value), the real significance threshold should be stricter than 0.05 to avoid a false positive from checking multiple metrics
## Business Impact
At current traffic (~10,000 visitors/week combined) and control's 5% baseline, a true 14% relative lift would be worth roughly 35 additional conversions per week. At an average order value of $85, that is approximately $3,000/week or $156,000/year if the effect is real. But given the confidence interval currently includes zero, shipping now risks realizing $0 of that value while still absorbing the engineering cost of maintaining the new flow.
## If Inconclusive: What's Needed
To detect a 14% relative lift (5.0% to 5.7%) with 80% power at α=0.05 requires approximately 16,200 visitors per variant — roughly 3.2x the current sample. At the current traffic rate of ~5,000 visitors per variant per two weeks, that means approximately 4-5 more weeks of the test running unchanged before a conclusive read is possible. Recommendation: extend the test rather than ship or kill based on the current data.
Die meisten Teams lesen A/B-Testergebnisse auf dieselbe fehlerhafte Weise: prüfen, ob die Behandlungszahl größer ist als die Kontrolle, werfen einen Blick darauf, ob das Dashboard "signifikant" sagt, und liefern aus. Dieser Prompt existiert, weil dieser Prozess die beiden Fehlermodi übersieht, die Produktteams wirklich schaden — Rauschen auszuliefern, als wäre es ein echter Effekt, und eine wirklich gute Idee zu verwerfen, weil der Test zu schwach war, um sie zu erkennen.
Warum das Urteil drei Optionen hat, nicht zwei
Ein binärer Ship-or-Kill-Rahmen erzwingt eine Entscheidung, selbst wenn die ehrliche Antwort "wir wissen es noch nicht" lautet. "Nicht schlüssig — Mehr Daten Nötig" zu einem legitimen erstklassigen Urteil zu machen — mit einer genauen Zahl dazu, was mehr Daten erfordern würden — verwandelt ein vages Gefühl der Unsicherheit in eine umsetzbare Wahl.
Warum die Fallstrick-Prüfung bei jedem Ergebnis läuft, auch bei "Siegen"
Ein p-Wert unter 0,05 ist notwendig, aber bei weitem nicht ausreichend. Das vorzeitige Betrachten von Ergebnissen vor einem vorregistrierten Enddatum bläht die Falsch-Positiv-Raten weit über die nominalen 5% auf. Eine Stichproben-Verhältnis-Fehlanpassung zwischen Varianten signalisiert oft einen Tracking-Fehler, der den gesamten Test ungültig macht, keinen echten Effekt.
Warum die geschäftliche Auswirkung in Dollar sein muss, nicht in Prozentpunkten
Ein relativer Anstieg von 14% klingt auf einer Folie überzeugend. Ob es die technischen Kosten der Pflege eines neuen Checkout-Flows wert ist, hängt vollständig davon ab, wie viel dieser Anstieg in tatsächlichem Umsatz wert ist.