تست فشار همبستگی در برابر علیت: قبل از تصمیمگیری، ادعای دادهمحورت رو زیر سوال ببر
چرا این پرامپت اهمیت دارد
Teams that act on unexamined correlations routinely misallocate budget and headcount for months before the real cause (a channel mix shift, a concurrent feature launch, simple seasonality) surfaces — at which point the reversal costs both the wasted spend and the credibility of whoever championed the original claim.
ما از آن برای چه استفاده میکنیم
Your marketing team just told leadership that a new onboarding email sequence increased 30-day retention by 12%, based on a before/after comparison, and the company is about to reallocate a team's entire budget on the strength of that one chart.
پرامپت
Role: You are a skeptical senior data scientist whose job is to stress-test causal claims before they influence business decisions. You are not trying to win an argument — you are trying to find out if the claim is actually true. Context: - The claim being made: [DESCRIBE THE CLAIM, e.g. "Our new onboarding email sequence increased 30-day retention by 12%"] - The data behind it: [DESCRIBE THE COMPARISON, e.g. "We compared the 30-day retention of users who received the sequence after its Sept 1 launch against users who signed up before the launch"] - What's at stake: [DESCRIBE THE DECISION RIDING ON THIS, e.g. "Leadership wants to double the lifecycle marketing team's budget and headcount based on this result"] Task: 1. List every plausible alternative explanation for the observed pattern that does not involve the claimed cause (consider confounders, selection effects, seasonality, concurrent changes, regression to the mean, reverse causation, and measurement-window asymmetry). 2. For each alternative explanation, state what evidence would confirm or rule it out. 3. Identify what data the team already has that could test these alternatives, and what data is missing. 4. Propose the minimum additional analysis or experiment needed to meaningfully raise or lower confidence in the causal claim — prefer the cheapest diagnostic that would resolve most of the uncertainty over a full randomized controlled trial. 5. Give a final verdict: "Act on this now," "Investigate further before acting," or "Do not act — too confounded," with two to three sentences of reasoning. Constraints: - Do not accept the causal claim at face value, even if the correlation looks statistically strong. - Flag explicitly if the claim is effectively unfalsifiable given the data described. - Keep the tone rigorous but practical — this is for a business audience making a resourcing decision, not a statistics journal. - Do not pad the analysis with generic caveats that apply to literally any claim; every alternative explanation must be specific to the scenario described. Output Format: - Section 1: Alternative Explanations (numbered list, one line each) - Section 2: Distinguishing Evidence (a table: alternative explanation | what would confirm or rule it out) - Section 3: Data Gap Assessment (what the team already has vs. what's missing) - Section 4: Recommended Next Step (one clear, specific action) - Section 5: Verdict (one of the three options above, with justification)
نتیجه
Section 1: Alternative Explanations
- Seasonality — the post-launch cohort signed up in September, which may systematically differ in intent or tenure from the January–August cohort used for comparison.
- Concurrent product changes — any other onboarding, pricing, or feature changes shipped around September 1 could be driving the retention lift instead of the email sequence.
- Marketing channel mix shift — if the acquisition channel mix changed around the launch date (for example, a shift from paid social toward organic or referral traffic), the new cohort may simply be higher-intent regardless of the emails.
- Regression to the mean — if pre-launch retention was unusually depressed in a specific recent window, the "lift" may partly reflect a natural reversion rather than the sequence's effect.
- Measurement-window asymmetry — the pre-launch cohort has eight months of mature, fully-observed data while the post-launch cohort is recent; if 30-day retention is still being calculated on partially-censored recent signups, the comparison is not apples-to-apples.
- Reverse causation via self-selection — if completion of the email sequence is optional or engagement-gated, only already more-engaged users may be finishing it, which would inflate the retention number for "recipients" without the emails causing anything.
Section 2: Distinguishing Evidence
| Alternative Explanation | What Would Confirm or Rule It Out |
|---|---|
| Seasonality | Compare September signups from the prior year (if the product existed then) against this September's cohort, holding the email sequence constant |
| Concurrent changes | Pull the full changelog for August 25 – September 5 and check for any other shipped change affecting new users |
| Channel mix shift | Break down both cohorts by acquisition channel and compare retention within matched channels |
| Regression to the mean | Plot weekly retention for the 12 months before launch to check whether August was an unusual trough |
| Measurement asymmetry | Confirm every user in the post-launch cohort has a fully elapsed 30-day window as of the analysis date |
| Self-selection | Check whether email sequence completion is optional; if so, compare retention of all assigned users (not just completers) against the baseline |
Section 3: Data Gap Assessment
The team already has: signup dates, retention flags at 30 days, and email send/open logs. Missing: a changelog cross-reference for the launch window, channel attribution broken out by cohort, and completion status for the email sequence (versus mere assignment to receive it).
Section 4: Recommended Next Step
Run a two-week holdback test: randomly assign 15% of new signups to skip the email sequence while keeping everything else identical, then compare 30-day retention between the holdback and recipient groups within the same calendar window. This isolates the sequence's effect from seasonality, channel mix, and concurrent changes in a fraction of the time a full RCT would take, since it only needs to resolve the sign and rough magnitude of the effect, not a precise estimate.
Section 5: Verdict
Investigate further before acting. The 12-point retention gap is large enough to be worth pursuing, but at least three of the six alternative explanations (channel mix shift, concurrent changes, and self-selection via completion) are both plausible and cheap to check. Committing to double a team's headcount on an uncontrolled before/after comparison risks a costly reversal if the holdback test shows a smaller or null effect.
بیشتر تصمیمهایی که بر اساس داده گرفته میشن نه به خاطر اشتباه بودن دادهها شکست میخورن، بلکه به این دلیل که هیچکس واقعاً تلاش نکرده اولین توضیحی رو که به نظر درست اومده جدی به چالش بکشه. یه عدد retention بعد از رونمایی یه محصول بالا میره، و همه این موفقیت رو به پای اون رونمایی میذارن - در حالی که توی همون بازه، سه-چهار چیز دیگه هم تغییر کرده بودن. این Prompt طراحی شده که اون الگو رو قطع کنه - قبل از اینکه یه تیم بودجه واقعی رو به یه داستان علّی تأییدنشده اختصاص بده.
چرا این از ساختار گزینههای اجباری استفاده میکنه
این Prompt از یه مدل AI نمیخواد که صرفاً «این داده رو تحلیل کن» - چون این قاببندی معمولاً منجر به تأیید هر ادعایی میشه که از پیش توی Prompt بوده، چون مدل هیچ انگیزهی ذاتیای برای مخالفت با فرض اولیه نداره. در عوض، بخش Task صریحاً به مدل دستور میده که اول توضیحات جایگزین رو تولید کنه، قبل از اینکه بتونه ادعای اصلی رو ارزیابی کنه. این ترتیب مهمه: وقتی یه مدل (یا یه نفر) روی یه داستان علّی لنگر انداخت، تولید شواهد مخالف خیلی سختتر از تولیدشون از یه صفحهی سفیده.
چرا فرمت خروجی از یه جدول شواهد تمایزدهنده استفاده میکنه نه از یه نتیجهگیری
یه لیست از توضیحات جایگزین فقط وقتی به دردت میخوره که بدونی چطور اونا رو از هم تشخیص بدی. به همین خاطر، بخش ۲ از فرمت خروجی هر توضیح جایگزین رو با یه شاهد مشخص و قابل بررسی جفت میکنه - نه یه «بیشتر بررسی کن» مبهم، بلکه یه مقایسهی ملموس (retention در سطح channel، یه cross-reference از changelog، یه cohort از سال قبل). این واکنش شکاکانه رو به یه برنامهی تشخیصی قابل اجرا تبدیل میکنه - و دقیقاً همینجاست که یه Prompt اضطرابزا از یه Prompt راهگشا جدا میشه.
چرا ارزانترین آزمایش رو توصیه میکنه نه دقیقترین رو
یه کارآزمایی تصادفی کنترلشده (randomized controlled trial) استانداردِ طلایی برای ادعاهای علّی هست، ولی بیشتر تیمها نه وقت دارن نه تمایل که برای هر متریکی که تغییر میکنه یکی اجرا کنن. بخش Constraints صریحاً به مدل میگه که ارزانترین روش تشخیصی رو ترجیح بده که میتونه سطح اطمینان رو بهطور معنادار تغییر بده، به همین خاطر هم خروجی نمونه یه holdback test کوتاه رو به جای یه RCT طولانی توصیه میکنه - هدف اینه که جهت و اندازهی تقریبی یه اثر رو سریع مشخص کنی، نه اینکه یه مقالهی peer-reviewed منتشر کنی.
چطور ازش درست استفاده کنی
سه فیلد داخل کروشه رو با هرچقدر جزئیات که میتونی تکمیل کن - ورودیهای مبهم، توضیحات جایگزین مبهم تولید میکنن. این Prompt وقتی بهترین نتیجه رو میده که از قبل شک داری داستان علّی شاید خیلی راحت جور درآمده باشه؛ اجرا کردنش روی یه ادعایی که کاملاً بهش مطمئنی، اغلب فقط همون چیزی رو تأیید میکنه که از پیش باور داشتی، که اصلاً هدف نیست. قبل از اینکه یه تصمیم برای تخصیص منابع گرفته بشه ازش استفاده کن، نه بعد از اون - تمام ارزشش در اینه که اشتباه رو موقعی بگیری که هنوز ارزون میشه درستش کرد.