Le Testeur de Résistance Corrélation vs Causalité : Mettez une Affirmation Basée sur des Données à l’Épreuve Avant d’Agir
Pourquoi ce prompt est important
Teams that act on unexamined correlations routinely misallocate budget and headcount for months before the real cause (a channel mix shift, a concurrent feature launch, simple seasonality) surfaces — at which point the reversal costs both the wasted spend and the credibility of whoever championed the original claim.
À quoi nous l'utilisons
Your marketing team just told leadership that a new onboarding email sequence increased 30-day retention by 12%, based on a before/after comparison, and the company is about to reallocate a team's entire budget on the strength of that one chart.
Prompt
Role: You are a skeptical senior data scientist whose job is to stress-test causal claims before they influence business decisions. You are not trying to win an argument — you are trying to find out if the claim is actually true. Context: - The claim being made: [DESCRIBE THE CLAIM, e.g. "Our new onboarding email sequence increased 30-day retention by 12%"] - The data behind it: [DESCRIBE THE COMPARISON, e.g. "We compared the 30-day retention of users who received the sequence after its Sept 1 launch against users who signed up before the launch"] - What's at stake: [DESCRIBE THE DECISION RIDING ON THIS, e.g. "Leadership wants to double the lifecycle marketing team's budget and headcount based on this result"] Task: 1. List every plausible alternative explanation for the observed pattern that does not involve the claimed cause (consider confounders, selection effects, seasonality, concurrent changes, regression to the mean, reverse causation, and measurement-window asymmetry). 2. For each alternative explanation, state what evidence would confirm or rule it out. 3. Identify what data the team already has that could test these alternatives, and what data is missing. 4. Propose the minimum additional analysis or experiment needed to meaningfully raise or lower confidence in the causal claim — prefer the cheapest diagnostic that would resolve most of the uncertainty over a full randomized controlled trial. 5. Give a final verdict: "Act on this now," "Investigate further before acting," or "Do not act — too confounded," with two to three sentences of reasoning. Constraints: - Do not accept the causal claim at face value, even if the correlation looks statistically strong. - Flag explicitly if the claim is effectively unfalsifiable given the data described. - Keep the tone rigorous but practical — this is for a business audience making a resourcing decision, not a statistics journal. - Do not pad the analysis with generic caveats that apply to literally any claim; every alternative explanation must be specific to the scenario described. Output Format: - Section 1: Alternative Explanations (numbered list, one line each) - Section 2: Distinguishing Evidence (a table: alternative explanation | what would confirm or rule it out) - Section 3: Data Gap Assessment (what the team already has vs. what's missing) - Section 4: Recommended Next Step (one clear, specific action) - Section 5: Verdict (one of the three options above, with justification)
Résultat
Section 1: Alternative Explanations
- Seasonality — the post-launch cohort signed up in September, which may systematically differ in intent or tenure from the January–August cohort used for comparison.
- Concurrent product changes — any other onboarding, pricing, or feature changes shipped around September 1 could be driving the retention lift instead of the email sequence.
- Marketing channel mix shift — if the acquisition channel mix changed around the launch date (for example, a shift from paid social toward organic or referral traffic), the new cohort may simply be higher-intent regardless of the emails.
- Regression to the mean — if pre-launch retention was unusually depressed in a specific recent window, the "lift" may partly reflect a natural reversion rather than the sequence's effect.
- Measurement-window asymmetry — the pre-launch cohort has eight months of mature, fully-observed data while the post-launch cohort is recent; if 30-day retention is still being calculated on partially-censored recent signups, the comparison is not apples-to-apples.
- Reverse causation via self-selection — if completion of the email sequence is optional or engagement-gated, only already more-engaged users may be finishing it, which would inflate the retention number for "recipients" without the emails causing anything.
Section 2: Distinguishing Evidence
| Alternative Explanation | What Would Confirm or Rule It Out |
|---|---|
| Seasonality | Compare September signups from the prior year (if the product existed then) against this September's cohort, holding the email sequence constant |
| Concurrent changes | Pull the full changelog for August 25 – September 5 and check for any other shipped change affecting new users |
| Channel mix shift | Break down both cohorts by acquisition channel and compare retention within matched channels |
| Regression to the mean | Plot weekly retention for the 12 months before launch to check whether August was an unusual trough |
| Measurement asymmetry | Confirm every user in the post-launch cohort has a fully elapsed 30-day window as of the analysis date |
| Self-selection | Check whether email sequence completion is optional; if so, compare retention of all assigned users (not just completers) against the baseline |
Section 3: Data Gap Assessment
The team already has: signup dates, retention flags at 30 days, and email send/open logs. Missing: a changelog cross-reference for the launch window, channel attribution broken out by cohort, and completion status for the email sequence (versus mere assignment to receive it).
Section 4: Recommended Next Step
Run a two-week holdback test: randomly assign 15% of new signups to skip the email sequence while keeping everything else identical, then compare 30-day retention between the holdback and recipient groups within the same calendar window. This isolates the sequence's effect from seasonality, channel mix, and concurrent changes in a fraction of the time a full RCT would take, since it only needs to resolve the sign and rough magnitude of the effect, not a precise estimate.
Section 5: Verdict
Investigate further before acting. The 12-point retention gap is large enough to be worth pursuing, but at least three of the six alternative explanations (channel mix shift, concurrent changes, and self-selection via completion) are both plausible and cheap to check. Committing to double a team's headcount on an uncontrolled before/after comparison risks a costly reversal if the holdback test shows a smaller or null effect.
La plupart des décisions basées sur les données échouent non pas parce que les données étaient fausses, mais parce que personne n’a sérieusement tenté de réfuter la première explication qui collait. Un chiffre de rétention augmente après un lancement, et le lancement en reçoit tout le mérite, même si trois ou quatre autres choses ont changé dans la même période. Ce prompt est conçu pour interrompre ce schéma avant qu’une équipe n’engage un budget réel sur une histoire causale non vérifiée.
Pourquoi ce prompt utilise une structure d’alternatives forcées
Le prompt ne demande pas simplement « analyse ces données » — ce cadrage tend à produire une confirmation de l’affirmation déjà présente. À la place, la section Tâche demande explicitement au modèle de générer d’abord des explications alternatives, avant d’être autorisé à évaluer l’affirmation initiale.
Pourquoi le format de sortie utilise un tableau de preuves distinctives, pas une conclusion
Une liste d’explications alternatives n’est utile que si vous savez comment les distinguer. C’est pourquoi la Section 2 associe chaque explication alternative à une preuve concrète et vérifiable.
Pourquoi il recommande le test le moins coûteux, pas le plus rigoureux
Un essai contrôlé randomisé complet est la référence absolue pour les affirmations causales, mais la plupart des équipes n’ont ni le temps ni l’envie d’en mener un pour chaque métrique qui bouge. La section Contraintes demande explicitement au modèle de privilégier le diagnostic le moins coûteux qui fait bouger significativement la confiance.
Comment bien l’utiliser
Remplissez les trois champs entre crochets avec un maximum de précision ; des entrées vagues produisent des explications alternatives vagues. Ce prompt fonctionne mieux lorsque vous soupçonnez déjà que l’histoire causale pourrait être trop commode. Utilisez-le avant qu’une décision d’allocation de ressources ne soit prise, pas après.