O Intérprete de Testes A/B: Transforme Resultados Brutos de Experimentos em uma Decisão de Lançar ou Descartar
Porque é que este prompt importa
Shipping a “winning” variant that was actually statistical noise wastes engineering effort maintaining a change that does nothing, and teams burned by a false positive often overcorrect by distrusting real wins later — the cost of misreading one test result compounds across every experimentation decision that follows it, not just the one bad rollout.
Para que o usamos
A product manager runs a two-week A/B test on a new checkout flow, sees the treatment converting at 5.7% versus 5.0% for control — a promising-looking 14% relative lift — and needs to know before Monday's roadmap review whether that result is real or noise before recommending a full rollout.
Prompt
Role: You are a senior product analyst who has run hundreds of A/B tests and knows exactly how teams misread "significant" results. Context: - Test name and hypothesis: [TEST NAME AND HYPOTHESIS] - Raw variant data: [PASTE DATA FOR EACH VARIANT: name, sample size, number of conversions/successes, and the metric definition] - Test duration: [START DATE - END DATE, OR NUMBER OF DAYS RUN] - Business context: [WHAT DECISION DEPENDS ON THIS TEST, e.g. "deciding whether to roll the new checkout flow out to 100% of users"] - Minimum meaningful effect, if known: [MDE, e.g. "we only care about lifts bigger than 2%"] Task: Analyze the results and produce: 1. A statistical significance calculation (p-value and/or confidence interval), with the method used stated explicitly 2. A plain-English verdict: Ship / Kill / Inconclusive — Need More Data 3. If inconclusive, the approximate additional sample size or time needed to reach a conclusive result at the current traffic rate and effect size 4. A check for common pitfalls: peeking bias (was this checked before the pre-registered end date?), sample ratio mismatch between variants, novelty effect risk, and multiple-comparisons risk if more than one metric was tracked 5. The practical business impact translated into revenue or user terms, not just a percentage lift Constraints: - Do not declare significance from p < 0.05 alone without first checking for sample ratio mismatch - If sample sizes are small for the observed conversion rate, flag that the result may be underpowered regardless of the p-value - Do not recommend shipping a "winning" variant if the confidence interval for the lift includes zero or a negative value - State any assumptions explicitly if required inputs are missing Output Format: ## Verdict [Ship / Kill / Inconclusive] ## Statistical Analysis [Method used, calculation shown, p-value or CI] ## Pitfall Check [Bulleted: peeking, sample ratio mismatch, novelty effect, multiple comparisons] ## Business Impact [Translated into revenue/user terms] ## If Inconclusive: What's Needed [Additional sample size or time required, with the math shown]
Resultado
Example: checkout flow redesign test, 14 days of data
## Verdict
Inconclusive — Need More Data. The treatment looks promising but the current sample cannot distinguish it from noise.
## Statistical Analysis
Control: 5,000 visitors, 250 conversions (5.00%)
Treatment: 5,100 visitors, 291 conversions (5.71%)
Observed relative lift: +14.2%
Method: two-proportion z-test, pooled variance
Pooled conversion rate: 5.36%
Standard error: 0.448 percentage points
z-score: 1.58
Two-tailed p-value: 0.114
95% confidence interval on the lift: -0.17pp to +1.59pp
The interval includes zero, so we cannot rule out that the true effect is flat or even slightly negative.
## Pitfall Check
- Peeking bias: confirm this analysis is happening at the pre-registered 14-day mark, not because someone checked the dashboard and saw a promising number mid-test and decided to look closer
- Sample ratio mismatch: 5,000 vs. 5,100 is close to the expected 50/50 split (49.5%/50.5%) — no red flag here
- Novelty effect: a checkout redesign is exactly the kind of change where early lift can reflect users' curiosity about a new layout rather than a durable improvement — worth re-checking this cohort's behavior at day 30
- Multiple comparisons: if conversion rate was one of several metrics tracked (e.g., also checking time-on-page, cart abandonment, average order value), the real significance threshold should be stricter than 0.05 to avoid a false positive from checking multiple metrics
## Business Impact
At current traffic (~10,000 visitors/week combined) and control's 5% baseline, a true 14% relative lift would be worth roughly 35 additional conversions per week. At an average order value of $85, that is approximately $3,000/week or $156,000/year if the effect is real. But given the confidence interval currently includes zero, shipping now risks realizing $0 of that value while still absorbing the engineering cost of maintaining the new flow.
## If Inconclusive: What's Needed
To detect a 14% relative lift (5.0% to 5.7%) with 80% power at α=0.05 requires approximately 16,200 visitors per variant — roughly 3.2x the current sample. At the current traffic rate of ~5,000 visitors per variant per two weeks, that means approximately 4-5 more weeks of the test running unchanged before a conclusive read is possible. Recommendation: extend the test rather than ship or kill based on the current data.
A maioria das equipes lê resultados de testes A/B da mesma forma falha: verifica se o número do tratamento é maior que o controle, dá uma olhada rápida se o painel diz "significativo" e lança. Este prompt existe porque esse processo ignora os dois modos de falha que realmente prejudicam as equipes de produto: lançar ruído como se fosse um efeito real, e descartar uma ideia genuinamente boa porque o teste não tinha poder estatístico suficiente para detectá-la.
Por que o veredito tem três opções, não duas
Um enquadramento binário de lançar-ou-descartar força uma decisão mesmo quando a resposta honesta é "ainda não sabemos". Tornar "Inconclusivo — Precisa de Mais Dados" um veredito legítimo de primeira classe — com um número exato anexado ao que mais dados exigiriam — transforma uma sensação vaga de incerteza em uma escolha acionável.
Por que a verificação de armadilhas roda em cada resultado, incluindo "vitórias"
Um valor-p abaixo de 0,05 é necessário, mas de forma alguma suficiente. Espiar resultados antes de uma data de término pré-registrada infla as taxas de falsos positivos muito além dos 5% nominais. Um desequilíbrio na proporção da amostra entre variantes frequentemente sinaliza um bug de rastreamento que invalida todo o teste, não um efeito real.
Por que o impacto nos negócios precisa estar em dólares, não em pontos percentuais
Um aumento relativo de 14% soa convincente em um slide. Se vale o custo de engenharia de manter um novo fluxo de checkout depende inteiramente de quanto esse aumento vale em receita real.