مفسر اختبار A/B: حوّل نتائج التجربة الخام إلى قرار إطلاق أو إلغاء
لماذا تهم هذه المطالبة
Shipping a “winning” variant that was actually statistical noise wastes engineering effort maintaining a change that does nothing, and teams burned by a false positive often overcorrect by distrusting real wins later — the cost of misreading one test result compounds across every experimentation decision that follows it, not just the one bad rollout.
فيم نستخدمها
A product manager runs a two-week A/B test on a new checkout flow, sees the treatment converting at 5.7% versus 5.0% for control — a promising-looking 14% relative lift — and needs to know before Monday's roadmap review whether that result is real or noise before recommending a full rollout.
المطالبة
Role: You are a senior product analyst who has run hundreds of A/B tests and knows exactly how teams misread "significant" results. Context: - Test name and hypothesis: [TEST NAME AND HYPOTHESIS] - Raw variant data: [PASTE DATA FOR EACH VARIANT: name, sample size, number of conversions/successes, and the metric definition] - Test duration: [START DATE - END DATE, OR NUMBER OF DAYS RUN] - Business context: [WHAT DECISION DEPENDS ON THIS TEST, e.g. "deciding whether to roll the new checkout flow out to 100% of users"] - Minimum meaningful effect, if known: [MDE, e.g. "we only care about lifts bigger than 2%"] Task: Analyze the results and produce: 1. A statistical significance calculation (p-value and/or confidence interval), with the method used stated explicitly 2. A plain-English verdict: Ship / Kill / Inconclusive — Need More Data 3. If inconclusive, the approximate additional sample size or time needed to reach a conclusive result at the current traffic rate and effect size 4. A check for common pitfalls: peeking bias (was this checked before the pre-registered end date?), sample ratio mismatch between variants, novelty effect risk, and multiple-comparisons risk if more than one metric was tracked 5. The practical business impact translated into revenue or user terms, not just a percentage lift Constraints: - Do not declare significance from p < 0.05 alone without first checking for sample ratio mismatch - If sample sizes are small for the observed conversion rate, flag that the result may be underpowered regardless of the p-value - Do not recommend shipping a "winning" variant if the confidence interval for the lift includes zero or a negative value - State any assumptions explicitly if required inputs are missing Output Format: ## Verdict [Ship / Kill / Inconclusive] ## Statistical Analysis [Method used, calculation shown, p-value or CI] ## Pitfall Check [Bulleted: peeking, sample ratio mismatch, novelty effect, multiple comparisons] ## Business Impact [Translated into revenue/user terms] ## If Inconclusive: What's Needed [Additional sample size or time required, with the math shown]
النتيجة
Example: checkout flow redesign test, 14 days of data
## Verdict
Inconclusive — Need More Data. The treatment looks promising but the current sample cannot distinguish it from noise.
## Statistical Analysis
Control: 5,000 visitors, 250 conversions (5.00%)
Treatment: 5,100 visitors, 291 conversions (5.71%)
Observed relative lift: +14.2%
Method: two-proportion z-test, pooled variance
Pooled conversion rate: 5.36%
Standard error: 0.448 percentage points
z-score: 1.58
Two-tailed p-value: 0.114
95% confidence interval on the lift: -0.17pp to +1.59pp
The interval includes zero, so we cannot rule out that the true effect is flat or even slightly negative.
## Pitfall Check
- Peeking bias: confirm this analysis is happening at the pre-registered 14-day mark, not because someone checked the dashboard and saw a promising number mid-test and decided to look closer
- Sample ratio mismatch: 5,000 vs. 5,100 is close to the expected 50/50 split (49.5%/50.5%) — no red flag here
- Novelty effect: a checkout redesign is exactly the kind of change where early lift can reflect users' curiosity about a new layout rather than a durable improvement — worth re-checking this cohort's behavior at day 30
- Multiple comparisons: if conversion rate was one of several metrics tracked (e.g., also checking time-on-page, cart abandonment, average order value), the real significance threshold should be stricter than 0.05 to avoid a false positive from checking multiple metrics
## Business Impact
At current traffic (~10,000 visitors/week combined) and control's 5% baseline, a true 14% relative lift would be worth roughly 35 additional conversions per week. At an average order value of $85, that is approximately $3,000/week or $156,000/year if the effect is real. But given the confidence interval currently includes zero, shipping now risks realizing $0 of that value while still absorbing the engineering cost of maintaining the new flow.
## If Inconclusive: What's Needed
To detect a 14% relative lift (5.0% to 5.7%) with 80% power at α=0.05 requires approximately 16,200 visitors per variant — roughly 3.2x the current sample. At the current traffic rate of ~5,000 visitors per variant per two weeks, that means approximately 4-5 more weeks of the test running unchanged before a conclusive read is possible. Recommendation: extend the test rather than ship or kill based on the current data.
تقرأ معظم الفرق نتائج اختبار A/B بنفس الطريقة المعيبة: التحقق مما إذا كان رقم المعالجة أكبر من التحكم، إلقاء نظرة سريعة على ما إذا كانت لوحة المعلومات تقول "دال إحصائياً"، ثم الإطلاق. يوجد هذا الموجّه لأن هذه العملية تفوت وضعي فشل يحرقان فرق المنتج فعلياً — إطلاق الضوضاء كما لو كانت تأثيراً حقيقياً، وإلغاء فكرة جيدة حقاً لأن الاختبار كان ضعيف القوة لاكتشافها.
لماذا يحتوي الحكم على ثلاثة خيارات، وليس اثنين
يفرض الإطار الثنائي إطلاق-أو-إلغاء قراراً حتى عندما تكون الإجابة الصادقة "لا نعرف بعد". جعل "غير حاسم - يحتاج مزيداً من البيانات" حكماً شرعياً من الدرجة الأولى — مع رقم دقيق مرفق بما ستتطلبه بيانات إضافية — يحوّل شعوراً غامضاً بعدم اليقين إلى خيار قابل للتنفيذ.
لماذا يتم تشغيل فحص الأخطاء على كل نتيجة، بما في ذلك "الانتصارات"
قيمة p أقل من 0.05 ضرورية لكنها ليست كافية بأي حال. النظر المبكر في النتائج قبل تاريخ الانتهاء المسجل مسبقاً يضخم معدلات الإيجابية الكاذبة إلى ما هو أبعد بكثير من 5% الاسمية. عدم تطابق نسبة العينة بين المتغيرات غالباً ما يشير إلى خطأ في التتبع يبطل الاختبار بأكمله، وليس تأثيراً حقيقياً.
لماذا يجب أن يكون التأثير التجاري بالدولار، وليس بنقاط النسبة المئوية
الارتفاع النسبي بنسبة 14% يبدو مقنعاً على شريحة عرض. ما إذا كان يستحق تكلفة الهندسة للحفاظ على تدفق دفع جديد يعتمد كلياً على القيمة الحقيقية لهذا الارتفاع من الإيرادات الفعلية.