El Intérprete de Pruebas A/B: Convierte Resultados Brutos de Experimentos en una Decisión de Lanzar o Descartar
Por qué importa este prompt
Shipping a “winning” variant that was actually statistical noise wastes engineering effort maintaining a change that does nothing, and teams burned by a false positive often overcorrect by distrusting real wins later — the cost of misreading one test result compounds across every experimentation decision that follows it, not just the one bad rollout.
Para qué lo usamos
A product manager runs a two-week A/B test on a new checkout flow, sees the treatment converting at 5.7% versus 5.0% for control — a promising-looking 14% relative lift — and needs to know before Monday's roadmap review whether that result is real or noise before recommending a full rollout.
Prompt
Role: You are a senior product analyst who has run hundreds of A/B tests and knows exactly how teams misread "significant" results. Context: - Test name and hypothesis: [TEST NAME AND HYPOTHESIS] - Raw variant data: [PASTE DATA FOR EACH VARIANT: name, sample size, number of conversions/successes, and the metric definition] - Test duration: [START DATE - END DATE, OR NUMBER OF DAYS RUN] - Business context: [WHAT DECISION DEPENDS ON THIS TEST, e.g. "deciding whether to roll the new checkout flow out to 100% of users"] - Minimum meaningful effect, if known: [MDE, e.g. "we only care about lifts bigger than 2%"] Task: Analyze the results and produce: 1. A statistical significance calculation (p-value and/or confidence interval), with the method used stated explicitly 2. A plain-English verdict: Ship / Kill / Inconclusive — Need More Data 3. If inconclusive, the approximate additional sample size or time needed to reach a conclusive result at the current traffic rate and effect size 4. A check for common pitfalls: peeking bias (was this checked before the pre-registered end date?), sample ratio mismatch between variants, novelty effect risk, and multiple-comparisons risk if more than one metric was tracked 5. The practical business impact translated into revenue or user terms, not just a percentage lift Constraints: - Do not declare significance from p < 0.05 alone without first checking for sample ratio mismatch - If sample sizes are small for the observed conversion rate, flag that the result may be underpowered regardless of the p-value - Do not recommend shipping a "winning" variant if the confidence interval for the lift includes zero or a negative value - State any assumptions explicitly if required inputs are missing Output Format: ## Verdict [Ship / Kill / Inconclusive] ## Statistical Analysis [Method used, calculation shown, p-value or CI] ## Pitfall Check [Bulleted: peeking, sample ratio mismatch, novelty effect, multiple comparisons] ## Business Impact [Translated into revenue/user terms] ## If Inconclusive: What's Needed [Additional sample size or time required, with the math shown]
Resultado
Example: checkout flow redesign test, 14 days of data
## Verdict
Inconclusive — Need More Data. The treatment looks promising but the current sample cannot distinguish it from noise.
## Statistical Analysis
Control: 5,000 visitors, 250 conversions (5.00%)
Treatment: 5,100 visitors, 291 conversions (5.71%)
Observed relative lift: +14.2%
Method: two-proportion z-test, pooled variance
Pooled conversion rate: 5.36%
Standard error: 0.448 percentage points
z-score: 1.58
Two-tailed p-value: 0.114
95% confidence interval on the lift: -0.17pp to +1.59pp
The interval includes zero, so we cannot rule out that the true effect is flat or even slightly negative.
## Pitfall Check
- Peeking bias: confirm this analysis is happening at the pre-registered 14-day mark, not because someone checked the dashboard and saw a promising number mid-test and decided to look closer
- Sample ratio mismatch: 5,000 vs. 5,100 is close to the expected 50/50 split (49.5%/50.5%) — no red flag here
- Novelty effect: a checkout redesign is exactly the kind of change where early lift can reflect users' curiosity about a new layout rather than a durable improvement — worth re-checking this cohort's behavior at day 30
- Multiple comparisons: if conversion rate was one of several metrics tracked (e.g., also checking time-on-page, cart abandonment, average order value), the real significance threshold should be stricter than 0.05 to avoid a false positive from checking multiple metrics
## Business Impact
At current traffic (~10,000 visitors/week combined) and control's 5% baseline, a true 14% relative lift would be worth roughly 35 additional conversions per week. At an average order value of $85, that is approximately $3,000/week or $156,000/year if the effect is real. But given the confidence interval currently includes zero, shipping now risks realizing $0 of that value while still absorbing the engineering cost of maintaining the new flow.
## If Inconclusive: What's Needed
To detect a 14% relative lift (5.0% to 5.7%) with 80% power at α=0.05 requires approximately 16,200 visitors per variant — roughly 3.2x the current sample. At the current traffic rate of ~5,000 visitors per variant per two weeks, that means approximately 4-5 more weeks of the test running unchanged before a conclusive read is possible. Recommendation: extend the test rather than ship or kill based on the current data.
La mayoría de los equipos leen los resultados de pruebas A/B de la misma manera defectuosa: verifican si el número del tratamiento es mayor que el control, echan un vistazo a si el panel dice "significativo" y lanzan. Este prompt existe porque ese proceso pasa por alto los dos modos de fallo que realmente perjudican a los equipos de producto: lanzar ruido como si fuera un efecto real, y descartar una idea genuinamente buena porque la prueba no tenía suficiente poder estadístico para detectarla.
Por qué el veredicto tiene tres opciones, no dos
Un encuadre binario de lanzar-o-descartar fuerza una decisión incluso cuando la respuesta honesta es "todavía no lo sabemos". Convertir "No concluyente — Necesita Más Datos" en un veredicto legítimo y de primera clase — con un número exacto adjunto de lo que requerirían más datos — convierte una sensación vaga de incertidumbre en una elección accionable.
Por qué la verificación de errores se ejecuta en cada resultado, incluidas las "victorias"
Un valor p por debajo de 0.05 es necesario pero de ninguna manera suficiente. Espiar los resultados antes de una fecha de finalización preregistrada infla las tasas de falsos positivos muy por encima del 5% nominal. Un desajuste en la proporción de muestra entre variantes a menudo señala un error de seguimiento que invalida toda la prueba, no un efecto real.
Por qué el impacto empresarial debe estar en dólares, no en puntos porcentuales
Un aumento relativo del 14% suena convincente en una diapositiva. Si vale la pena el costo de ingeniería de mantener un nuevo flujo de pago depende completamente de cuánto vale ese aumento en ingresos reales.