تشخیصدهنده تستهای ناپایدار: تبدیل شکستهای متناوب CI به فرضیههای ریشهیابی رتبهبندیشده
Why this prompt matters
Flaky tests that get ignored quietly erode trust in the entire suite: engineers start re-running failed CI jobs by reflex instead of reading why they failed, which means a genuine regression can slip through disguised as 'oh, that test is just flaky.' Teams that let flaky tests accumulate past a small fraction of the suite typically see their mean time to detect real production bugs get significantly worse, because the signal-to-noise ratio of CI failures has collapsed.
What we use it for
You're a backend engineer and your team's CI pipeline has a test that fails about 1 in 15 runs with no code changes. Nobody wants to spend a sprint chasing it, so it's been ignored for three weeks -- and now two more tests in the same suite are starting to flake too.
Prompt
Act as a senior test infrastructure engineer who specializes in diagnosing intermittent, non-deterministic CI test failures ("flaky tests").
CONTEXT:
- Test name / file: [TEST NAME OR FILE PATH]
- Test framework and language: [e.g. "pytest, Python" or "Jest, TypeScript"]
- Failure frequency: [e.g. "1 in 15 runs" or "roughly once a week, no clear pattern"]
- Failure logs from 3-5 recent failed runs (paste full stack traces / error output, not summaries): [PASTE FAILURE LOGS HERE, SEPARATED BY RUN]
- Recent changes to the test or the code it covers, if known: [DESCRIBE OR PASTE RELEVANT DIFF, OR WRITE "NONE KNOWN"]
- What the test is actually verifying (in plain English): [ONE-SENTENCE DESCRIPTION OF TEST INTENT]
TASK:
Analyze the failure patterns across the provided logs and produce ranked root-cause hypotheses. Consider these common flaky-test categories and rule each in or out based on the evidence: timing/race conditions, shared state or test pollution from other tests, external dependency instability (network, third-party API, database), resource exhaustion (memory, connection pool, file handles), non-deterministic test ordering, and environment differences between CI and local runs.
CONSTRAINTS:
- Do not simply conclude "the test is flaky" without committing to at least 2 specific, falsifiable hypotheses ranked by likelihood
- Every hypothesis must cite specific evidence from the pasted logs -- if the logs don't support a hypothesis, don't include it
- Clearly distinguish between issues safe to quarantine now and investigate later, versus issues that likely mask a real production bug and must be fixed before quarantining
- If the provided logs don't contain enough information to diagnose confidently, say exactly that, and specify precisely what additional logging or instrumentation to add before the next failure
OUTPUT FORMAT:
1. A ranked table: Hypothesis | Confidence (High/Medium/Low) | Evidence From Logs | Suggested Fix
2. A single recommended immediate action: Quarantine and Investigate Later / Must Fix Before Quarantining / Needs More Data First
3. If "Needs More Data First": the exact logging statements or CI configuration change to add so the next failure captures what's missingResult
Diagnosis: test_checkout_confirms_payment_status (pytest, Python)
Failure frequency: ~1 in 12 CI runs, no clear time-of-day pattern.
| Hypothesis | Confidence | Evidence From Logs | Suggested Fix |
|---|---|---|---|
| Race condition between async payment webhook handler and the assertion checking order status | High | All 3 failed runs show the assertion firing at order.status == 'pending' rather than 'confirmed' -- the webhook handler logs show it completing 40-180ms AFTER the assertion ran in 2 of 3 failures | Replace the fixed 100ms sleep before the assertion with an explicit poll-until-status-or-timeout helper (max 2s), and add a log line at the exact moment the webhook handler acquires its DB lock |
| Shared test database state from a preceding test in the same suite | Medium | One failure log shows an order ID that doesn't match the expected sequence, suggesting a prior test's order wasn't cleaned up -- but this only appears in 1 of 3 logs, so it's a contributing factor, not the primary cause | Add explicit teardown assertion verifying zero leftover orders before this test starts; run this test in isolation for 20 iterations to confirm whether it still flakes without the suite context |
| Third-party payment sandbox API latency spike | Low | No log evidence of elevated response times from the payment sandbox in any of the 3 failures -- ruled out based on available data | N/A -- not supported by current evidence |
Recommended Immediate Action: Must Fix Before Quarantining
This is not safe to quarantine. The race condition hypothesis is high-confidence and points to a genuine bug: the webhook handler and the test assertion have no synchronization mechanism between them, which means the SAME race condition could cause a real customer to see a stale "pending" payment status in production under load. Quarantining this test would hide a real timing bug, not just a test artifact. Fix the polling logic first; if the test still flakes after that fix, investigate the shared-state hypothesis as the secondary cause.
در هر تیم مهندسی، تستهای ناپایدار انباشته میشوند و هر تیم در واکنش به آن، همان عادت بد را پیدا میکند: کسی دکمه اجرای مجدد را میزند، تست در بار دوم موفق میشود و همه بدون اینکه هرگز بفهمند چرا بار اول شکست خورد، به کار خود ادامه میدهند. این عادت مخرب است. وقتی یک تیم بهطور پیشفرض شروع میکند شکستهای CI را بهعنوان نویز در نظر بگیرد، یک پسرفت واقعی در نهایت درون همان نویز پنهان میشود و به محیط تولید راه مییابد.
این پرامپت انضباط متفاوتی را ایجاب میکند. به جای پذیرش یک تشخیص مبهمِ «ناپایدار است»، از مدل میخواهد لاگهای واقعی شکست را از اجراهای متعدد بخواند و حداقل دو فرضیه مشخص و قابل ابطال ارائه دهد — فرضیههایی که بر اساس میزان اطمینان رتبهبندی شدهاند و هر کدام با یک شاهد مشخص از لاگها پشتیبانی میشوند، نه یک حدس کلی. شرایط رقابت (Race conditions)، وضعیت اشتراکی تست، ناپایداری وابستگی خارجی، تمام شدن منابع و ناهماهنگی محیط بین CI و محیط محلی رایجترین علل ریشهای هستند و پرامپت بهجای اینکه به سراغ آشناترین گزینه برود، بهصورت سیستماتیک هر یک را بررسی میکند.
مفیدترین بخش خروجی، حکمی است که اکثر تیمها کاملاً از آن چشمپوشی میکنند: آیا قرنطینه کردن این تست بیخطر است تا بعداً دوباره به آن رسیدگی شود، یا ناپایداری در واقع یک باگ واقعی را نشان میدهد که تحت شرایط مشابه میتواند بر تولید تأثیر بگذارد؟ برای مثال، یک شرط رقابت بین handler وبهوک ناهمگام و یک بررسی وضعیت، فقط یک مشکل تست نیست — این همان باگی است که یک مشتری واقعی میتواند تحت بار با آن مواجه شود. قرنطینه کردن آن تست، یک مشکل تولید را پشت یک تیک سبز پنهان میکند. تشخیص درست این تفاوت در هر بار اجرا، چیزی است که تیمهای دارای سیگنال CI قابل اعتماد را از تیمهایی که بیصدا از اعتماد به مجموعه تست خود دست کشیدهاند جدا میکند.