El Pressure-Tester de Correlación vs Causalidad: Pon a Prueba una Afirmación Basada en Datos Antes de Actuar
Por qué importa este prompt
Teams that act on unexamined correlations routinely misallocate budget and headcount for months before the real cause (a channel mix shift, a concurrent feature launch, simple seasonality) surfaces — at which point the reversal costs both the wasted spend and the credibility of whoever championed the original claim.
Para qué lo usamos
Your marketing team just told leadership that a new onboarding email sequence increased 30-day retention by 12%, based on a before/after comparison, and the company is about to reallocate a team's entire budget on the strength of that one chart.
Prompt
Role: You are a skeptical senior data scientist whose job is to stress-test causal claims before they influence business decisions. You are not trying to win an argument — you are trying to find out if the claim is actually true. Context: - The claim being made: [DESCRIBE THE CLAIM, e.g. "Our new onboarding email sequence increased 30-day retention by 12%"] - The data behind it: [DESCRIBE THE COMPARISON, e.g. "We compared the 30-day retention of users who received the sequence after its Sept 1 launch against users who signed up before the launch"] - What's at stake: [DESCRIBE THE DECISION RIDING ON THIS, e.g. "Leadership wants to double the lifecycle marketing team's budget and headcount based on this result"] Task: 1. List every plausible alternative explanation for the observed pattern that does not involve the claimed cause (consider confounders, selection effects, seasonality, concurrent changes, regression to the mean, reverse causation, and measurement-window asymmetry). 2. For each alternative explanation, state what evidence would confirm or rule it out. 3. Identify what data the team already has that could test these alternatives, and what data is missing. 4. Propose the minimum additional analysis or experiment needed to meaningfully raise or lower confidence in the causal claim — prefer the cheapest diagnostic that would resolve most of the uncertainty over a full randomized controlled trial. 5. Give a final verdict: "Act on this now," "Investigate further before acting," or "Do not act — too confounded," with two to three sentences of reasoning. Constraints: - Do not accept the causal claim at face value, even if the correlation looks statistically strong. - Flag explicitly if the claim is effectively unfalsifiable given the data described. - Keep the tone rigorous but practical — this is for a business audience making a resourcing decision, not a statistics journal. - Do not pad the analysis with generic caveats that apply to literally any claim; every alternative explanation must be specific to the scenario described. Output Format: - Section 1: Alternative Explanations (numbered list, one line each) - Section 2: Distinguishing Evidence (a table: alternative explanation | what would confirm or rule it out) - Section 3: Data Gap Assessment (what the team already has vs. what's missing) - Section 4: Recommended Next Step (one clear, specific action) - Section 5: Verdict (one of the three options above, with justification)
Resultado
Section 1: Alternative Explanations
- Seasonality — the post-launch cohort signed up in September, which may systematically differ in intent or tenure from the January–August cohort used for comparison.
- Concurrent product changes — any other onboarding, pricing, or feature changes shipped around September 1 could be driving the retention lift instead of the email sequence.
- Marketing channel mix shift — if the acquisition channel mix changed around the launch date (for example, a shift from paid social toward organic or referral traffic), the new cohort may simply be higher-intent regardless of the emails.
- Regression to the mean — if pre-launch retention was unusually depressed in a specific recent window, the "lift" may partly reflect a natural reversion rather than the sequence's effect.
- Measurement-window asymmetry — the pre-launch cohort has eight months of mature, fully-observed data while the post-launch cohort is recent; if 30-day retention is still being calculated on partially-censored recent signups, the comparison is not apples-to-apples.
- Reverse causation via self-selection — if completion of the email sequence is optional or engagement-gated, only already more-engaged users may be finishing it, which would inflate the retention number for "recipients" without the emails causing anything.
Section 2: Distinguishing Evidence
| Alternative Explanation | What Would Confirm or Rule It Out |
|---|---|
| Seasonality | Compare September signups from the prior year (if the product existed then) against this September's cohort, holding the email sequence constant |
| Concurrent changes | Pull the full changelog for August 25 – September 5 and check for any other shipped change affecting new users |
| Channel mix shift | Break down both cohorts by acquisition channel and compare retention within matched channels |
| Regression to the mean | Plot weekly retention for the 12 months before launch to check whether August was an unusual trough |
| Measurement asymmetry | Confirm every user in the post-launch cohort has a fully elapsed 30-day window as of the analysis date |
| Self-selection | Check whether email sequence completion is optional; if so, compare retention of all assigned users (not just completers) against the baseline |
Section 3: Data Gap Assessment
The team already has: signup dates, retention flags at 30 days, and email send/open logs. Missing: a changelog cross-reference for the launch window, channel attribution broken out by cohort, and completion status for the email sequence (versus mere assignment to receive it).
Section 4: Recommended Next Step
Run a two-week holdback test: randomly assign 15% of new signups to skip the email sequence while keeping everything else identical, then compare 30-day retention between the holdback and recipient groups within the same calendar window. This isolates the sequence's effect from seasonality, channel mix, and concurrent changes in a fraction of the time a full RCT would take, since it only needs to resolve the sign and rough magnitude of the effect, not a precise estimate.
Section 5: Verdict
Investigate further before acting. The 12-point retention gap is large enough to be worth pursuing, but at least three of the six alternative explanations (channel mix shift, concurrent changes, and self-selection via completion) are both plausible and cheap to check. Committing to double a team's headcount on an uncontrolled before/after comparison risks a costly reversal if the holdback test shows a smaller or null effect.
La mayoría de las decisiones basadas en datos fallan no porque los datos fueran incorrectos, sino porque nadie intentó seriamente refutar la primera explicación que encajaba. Una cifra de retención sube tras un lanzamiento, y el lanzamiento recibe todo el crédito, aunque otras tres o cuatro cosas cambiaron en la misma ventana. Este prompt está diseñado para interrumpir ese patrón antes de que un equipo comprometa presupuesto real en una historia causal no verificada.
Por qué usa una estructura de alternativas forzadas
El prompt no pide simplemente “analizar estos datos”, porque ese enfoque tiende a producir confirmación de cualquier afirmación ya presente. En cambio, la sección Tarea instruye explícitamente al modelo a generar explicaciones alternativas primero, antes de evaluar la afirmación original.
Por qué el formato de salida usa una tabla de evidencia distintiva, no una conclusión
Una lista de explicaciones alternativas solo es útil si sabes cómo distinguirlas. Por eso la Sección 2 empareja cada explicación alternativa con una evidencia concreta y verificable.
Por qué recomienda la prueba más barata, no la más rigurosa
Un ensayo controlado aleatorizado completo es el estándar de oro, pero la mayoría de los equipos no tiene tiempo para ejecutarlo en cada métrica que se mueve. La sección de Restricciones le indica explícitamente al modelo preferir el diagnóstico más barato que mueva significativamente la confianza.
Cómo usarlo bien
Completa los tres campos entre corchetes con la mayor especificidad posible; entradas vagas producen explicaciones alternativas vagas. Este prompt funciona mejor cuando ya sospechas que la historia causal podría ser demasiado conveniente. Úsalo antes de que se tome una decisión de asignación de recursos, no después.