مُختبِر الارتباط مقابل السببية: اختبر ادعاءً قائماً على البيانات قبل أن تتصرف بناءً عليه
لماذا تهم هذه المطالبة
Teams that act on unexamined correlations routinely misallocate budget and headcount for months before the real cause (a channel mix shift, a concurrent feature launch, simple seasonality) surfaces — at which point the reversal costs both the wasted spend and the credibility of whoever championed the original claim.
فيم نستخدمها
Your marketing team just told leadership that a new onboarding email sequence increased 30-day retention by 12%, based on a before/after comparison, and the company is about to reallocate a team's entire budget on the strength of that one chart.
المطالبة
Role: You are a skeptical senior data scientist whose job is to stress-test causal claims before they influence business decisions. You are not trying to win an argument — you are trying to find out if the claim is actually true. Context: - The claim being made: [DESCRIBE THE CLAIM, e.g. "Our new onboarding email sequence increased 30-day retention by 12%"] - The data behind it: [DESCRIBE THE COMPARISON, e.g. "We compared the 30-day retention of users who received the sequence after its Sept 1 launch against users who signed up before the launch"] - What's at stake: [DESCRIBE THE DECISION RIDING ON THIS, e.g. "Leadership wants to double the lifecycle marketing team's budget and headcount based on this result"] Task: 1. List every plausible alternative explanation for the observed pattern that does not involve the claimed cause (consider confounders, selection effects, seasonality, concurrent changes, regression to the mean, reverse causation, and measurement-window asymmetry). 2. For each alternative explanation, state what evidence would confirm or rule it out. 3. Identify what data the team already has that could test these alternatives, and what data is missing. 4. Propose the minimum additional analysis or experiment needed to meaningfully raise or lower confidence in the causal claim — prefer the cheapest diagnostic that would resolve most of the uncertainty over a full randomized controlled trial. 5. Give a final verdict: "Act on this now," "Investigate further before acting," or "Do not act — too confounded," with two to three sentences of reasoning. Constraints: - Do not accept the causal claim at face value, even if the correlation looks statistically strong. - Flag explicitly if the claim is effectively unfalsifiable given the data described. - Keep the tone rigorous but practical — this is for a business audience making a resourcing decision, not a statistics journal. - Do not pad the analysis with generic caveats that apply to literally any claim; every alternative explanation must be specific to the scenario described. Output Format: - Section 1: Alternative Explanations (numbered list, one line each) - Section 2: Distinguishing Evidence (a table: alternative explanation | what would confirm or rule it out) - Section 3: Data Gap Assessment (what the team already has vs. what's missing) - Section 4: Recommended Next Step (one clear, specific action) - Section 5: Verdict (one of the three options above, with justification)
النتيجة
Section 1: Alternative Explanations
- Seasonality — the post-launch cohort signed up in September, which may systematically differ in intent or tenure from the January–August cohort used for comparison.
- Concurrent product changes — any other onboarding, pricing, or feature changes shipped around September 1 could be driving the retention lift instead of the email sequence.
- Marketing channel mix shift — if the acquisition channel mix changed around the launch date (for example, a shift from paid social toward organic or referral traffic), the new cohort may simply be higher-intent regardless of the emails.
- Regression to the mean — if pre-launch retention was unusually depressed in a specific recent window, the "lift" may partly reflect a natural reversion rather than the sequence's effect.
- Measurement-window asymmetry — the pre-launch cohort has eight months of mature, fully-observed data while the post-launch cohort is recent; if 30-day retention is still being calculated on partially-censored recent signups, the comparison is not apples-to-apples.
- Reverse causation via self-selection — if completion of the email sequence is optional or engagement-gated, only already more-engaged users may be finishing it, which would inflate the retention number for "recipients" without the emails causing anything.
Section 2: Distinguishing Evidence
| Alternative Explanation | What Would Confirm or Rule It Out |
|---|---|
| Seasonality | Compare September signups from the prior year (if the product existed then) against this September's cohort, holding the email sequence constant |
| Concurrent changes | Pull the full changelog for August 25 – September 5 and check for any other shipped change affecting new users |
| Channel mix shift | Break down both cohorts by acquisition channel and compare retention within matched channels |
| Regression to the mean | Plot weekly retention for the 12 months before launch to check whether August was an unusual trough |
| Measurement asymmetry | Confirm every user in the post-launch cohort has a fully elapsed 30-day window as of the analysis date |
| Self-selection | Check whether email sequence completion is optional; if so, compare retention of all assigned users (not just completers) against the baseline |
Section 3: Data Gap Assessment
The team already has: signup dates, retention flags at 30 days, and email send/open logs. Missing: a changelog cross-reference for the launch window, channel attribution broken out by cohort, and completion status for the email sequence (versus mere assignment to receive it).
Section 4: Recommended Next Step
Run a two-week holdback test: randomly assign 15% of new signups to skip the email sequence while keeping everything else identical, then compare 30-day retention between the holdback and recipient groups within the same calendar window. This isolates the sequence's effect from seasonality, channel mix, and concurrent changes in a fraction of the time a full RCT would take, since it only needs to resolve the sign and rough magnitude of the effect, not a precise estimate.
Section 5: Verdict
Investigate further before acting. The 12-point retention gap is large enough to be worth pursuing, but at least three of the six alternative explanations (channel mix shift, concurrent changes, and self-selection via completion) are both plausible and cheap to check. Committing to double a team's headcount on an uncontrolled before/after comparison risks a costly reversal if the holdback test shows a smaller or null effect.
معظم القرارات القائمة على البيانات لا تفشل لأن البيانات كانت خاطئة، بل لأن لا أحد حاول جدياً دحض أول تفسير يناسب النمط. يرتفع رقم الاحتفاظ بعد إطلاق ميزة، فتُنسب كل الفضل إليها رغم أن ثلاثة أو أربعة أشياء أخرى تغيرت في النافذة الزمنية نفسها. صُمم هذا الموجه لوقف هذا النمط قبل أن يخصص فريق ميزانية حقيقية لقصة سببية غير مُتحقَّق منها.
لماذا يستخدم هذا الموجه بنية “البدائل الإجبارية”
لا يطلب الموجه من نموذج الذكاء الاصطناعي “تحليل هذه البيانات” فحسب — لأن هذه الصياغة تميل إلى إنتاج تأكيد لأي ادعاء موجود مسبقاً في الموجه. بدلاً من ذلك، يوجّه قسم المهمة النموذج صراحةً لتوليد تفسيرات بديلة أولاً، قبل السماح له بتقييم الادعاء الأصلي.
لماذا يستخدم تنسيق المخرجات جدول أدلة تمييزية، لا استنتاجاً
قائمة التفسيرات البديلة مفيدة فقط إذا عرفت كيف تميّز بينها. لهذا يقرن القسم الثاني من تنسيق المخرجات كل تفسير بديل بدليل محدد وقابل للتحقق.
لماذا يوصي بأرخص اختبار، لا الأكثر صرامة
التجربة العشوائية المُحكمة الكاملة هي المعيار الذهبي للادعاءات السببية، لكن معظم الفرق لا تملك الوقت أو الرغبة لإجرائها على كل مقياس يتغير. يخبر قسم القيود النموذج صراحةً بتفضيل أرخص تشخيص يرفع أو يخفض الثقة بشكل معقول.
كيفية الاستخدام الأمثل
املأ الحقول بين الأقواس بأكبر قدر ممكن من التحديد؛ المدخلات الغامضة تنتج تفسيرات بديلة غامضة. يعمل هذا الموجه بشكل أفضل عندما تشك مسبقاً في أن القصة السببية قد تكون مريحة أكثر من اللازم. استخدمه قبل اتخاذ قرار تخصيص الموارد، لا بعده.