تریاژ تست‌های فلیکی: تست‌های ناموفق CI رو به یه پلن قرنطینه‌ی رتبه‌بندی‌شده بر اساس ریشه‌ی مشکل تبدیل کنEXCERPT: یه Prompt ساختاریافته برای AI که یه عالمه تست ناموفقِ گاه‌به‌گاهِ CI رو به جدول تریاژ ریشه‌ی مشکل، لیست قرنطینه‌ی همون روز و پلن اسپرینت تبدیل می‌کنه — ساخته‌شده برای تیم‌هایی که از کلیک رفلکسی روی دکمه‌ی «دوباره اجرا» خسته شدن.
چرا این پرامپت اهمیت دارد
Flaky tests that go untriaged do not stay flat — they compound, because "just re-run it" becomes the team's default response to any red build. A widely cited Google engineering study found flaky tests account for roughly 1 in every 300 test runs in large CI suites, and engineering teams that track the human cost put it far higher: SREs commonly estimate several lost engineer-hours per week spent manually re-running or waiting out intermittent failures across a mid-sized team. Left untriaged, flaky tests are also one of the most common reasons teams eventually weaken or disable CI gating altogether — trading the short-term convenience of a green build for the exact safety net CI exists to provide, right up until a real regression slips through disguised as "probably just flaky."
ما از آن برای چه استفاده می‌کنیم
Your team's CI dashboard has quietly accumulated 18 tests tagged "flaky" over the past two months. Nobody owns the list, and new hires are simply told to click re-run when one of them turns red. On Thursday afternoon, ahead of Friday's release freeze, the QA lead exports the flaky-test tracker — test names, failure rates, and the last few stack traces for each — and runs it through this prompt to get a sprint-ready triage: which tests to quarantine today, which ones trace back to one shared root cause, and which genuinely need another engineer's eyes before anyone touches them.
پرامپت
Role: Act as a senior test infrastructure engineer / staff SDET with 10+ years of experience stabilizing CI pipelines at scale. You have seen every category of flaky test and know the difference between a real intermittent bug and a badly-written test. Context: My team's CI system is [CI SYSTEM NAME, e.g. GitHub Actions / CircleCI / Jenkins]. Over the past [TIME WINDOW, e.g. "6 weeks"], the following tests have been flagged as flaky (intermittently failing with no corresponding code change in the failing area): [PASTE FLAKY TEST LIST — for each test include: test name, approximate failure rate (e.g. "fails 1 in 8 runs"), the error message or stack trace snippet from the last 2-3 failures, and how long it has been flaky] Relevant recent changes to the codebase or infrastructure: [RECENT CODE/INFRA CHANGES, e.g. "migrated CI runners to new containers 3 weeks ago", "added a Redis caching layer", "none known"] Engineering capacity available to fix tests this sprint: [TEAM SIZE / CAPACITY, e.g. "2 engineers, 1 day each"] Task: For each test in the list, triage it into a root-cause category, assign a confidence level, and recommend a specific next action. Then identify any systemic patterns across multiple tests (e.g. several tests failing because of one shared fixture or one piece of shared infrastructure) and surface those separately, since fixing one shared root cause may resolve several "individual" flaky tests at once. Constraints: - Do not guess a root cause without citing specific evidence from the error message or stack trace provided. If the evidence given is insufficient to form a confident hypothesis, say so explicitly and mark the test as "Needs More Data" rather than inventing a plausible-sounding cause. - Do not recommend "just add a retry" or "increase the timeout" as a fix unless the evidence specifically points to a timing/race condition where that is a legitimate mitigation — and if you do recommend it, label it clearly as a MITIGATION, not a FIX, and note that the underlying bug is still open. - Keep every recommended action realistic for the stated engineering capacity — do not recommend 5 fixes if the team only has 1 day available; rank by impact instead. - If a test appears to be testing something that no longer matters (e.g. a deprecated feature), flag it as a candidate for deletion rather than a fix. - Be direct about tests that are poorly designed (e.g. depend on wall-clock time, test execution order, or shared mutable state) versus tests exposing a genuine product bug — these require different fixes. Output Format: 1. A table with these exact columns: Test Name | Suspected Root Cause Category (Shared State / Race Condition / External Dependency / Environment-Infra / Genuine Product Bug / Poorly Designed Test / Needs More Data) | Confidence (High/Medium/Low) | Evidence Cited | Recommended Action | Suggested Owner/Skillset 2. A "Quarantine Now" section: a bare list of test names that should be disabled or skipped immediately, each with a one-line justification, so they stop blocking merges while the real fix is scheduled. 3. A "Systemic Issues" section: any root cause that explains 2+ tests in the list, with the shared fix that would resolve multiple rows at once, ranked by how many tests it would unblock. 4. A "This Sprint" section: given the stated capacity, the single highest-leverage action to take first and why.
نتیجه
Here is the triage based on the 6 tests and context provided:
| Test Name | Root Cause Category | Confidence | Evidence Cited | Recommended Action | Owner | |---|---|---|---|---|---| | test_checkout_coupon_apply | Shared State | High | Failure log shows AssertionError: expected balance 42.00, got 38.50 only when run after test_checkout_refund_flow in the same worker — both tests share a module-level TEST_WALLET fixture that is never reset between runs | Fix Now: move TEST_WALLET to a per-test fixture with explicit teardown | Backend/test-infra engineer, ~2 hrs | | test_session_refresh_race | Race Condition | High | Stack trace shows TokenExpiredError thrown 40ms after the refresh call returned 200 — the token refresh writes to cache asynchronously | Mitigation only: await the cache write explicitly in the test. This is NOT a fix — the underlying async write-then-read gap may also affect production under load; file a separate ticket | Platform engineer | | test_search_results_count | External Dependency | Medium | 3 of 4 recent failures logged ConnectionTimeout against the staging Elasticsearch cluster, not the application code | Quarantine Now + file an infra ticket for staging ES cluster capacity | Infra/SRE | | test_legacy_sms_notification | Unknown | Low | Only one failure instance provided and the error message is truncated — not enough evidence to confirm a root cause | Needs More Data: capture the full stack trace on the next 2 failures before assigning an owner | QA | | test_deprecated_loyalty_tier_calc | Poorly Designed Test / Deletion Candidate | High | Test covers the loyalty-tier feature that was flagged for removal in last quarter's roadmap doc | Delete rather than fix | Feature owner | | test_cart_total_rounding | Shared State | Medium | Same TEST_WALLET fixture referenced in the failure context as test_checkout_coupon_apply | Resolved by the same fix as row 1 | — |
Quarantine Now: test_search_results_count — blocked on a staging infrastructure problem, not application code; disabling it stops it from blocking unrelated merges while SRE addresses cluster capacity.
Systemic Issues: The TEST_WALLET shared fixture explains 2 of the 6 failures (test_checkout_coupon_apply and test_cart_total_rounding). Fixing the fixture's teardown logic once resolves both rows and should be prioritized over fixing either individually — this is the highest-leverage single change in the list.
This Sprint: With 1 engineer-day available, fix the TEST_WALLET fixture teardown first. It resolves 2 of 6 flaky tests in roughly 2 hours, leaving the rest of the day to quarantine test_search_results_count and file the Elasticsearch capacity ticket with infra.
هر تیم مهندسی به یه شکل مشخصی تست‌های فلیکی جمع می‌کنه: یکی از تست‌ها به‌صورت تصادفی خراب می‌شه، یکی می‌زنه «re-run»، تست سبز می‌شه، و تیکتِ واقعی برای بررسی هیچ‌وقت ثبت نمی‌شه. شش ماه بعد یه لیست توی ترکر هست با بیش از ۲۰ تست که هیچ‌کس مسئولش نیست، و اعتماد تیم به سیگنال سبز/قرمز CI بی‌صدا از بین رفته.
این Prompt به‌عنوان یه چارچوب ساختاریافته‌ی تریاژ ساخته شده، نه یه درخواست ساده‌ی «خطا رو بچسبون، فیکس بگیر»، چون تشخیص تست فلیکی پیش از هر چیز یه مسئله‌ی دسته‌بندی‌ست تا یه مسئله‌ی فیکس کردن. بخش Role نقش AI رو به‌عنوان یه مهندس ارشد زیرساخت تست لنگر می‌ندازه، دقیقاً برای اینکه الگوها رو با دسته‌های شناخته‌شده‌ی تست فلیکی تطبیق بده (state مشترک، race condition، وابستگی‌های خارجی، زیرساخت، باگ واقعی، طراحی بد تست) نه اینکه هر شکست رو یه معمای بی‌همتا حساب کنه.
بخش Constraints بیشترین بار رو به دوش می‌کشه. صریحاً دو حالت شکست رایج دیباگ با کمک AI رو ممنوع می‌کنه: اختراع یه علت ریشه‌ای که منطقی به نظر می‌رسه وقتی شواهد پشتیبانیش نمی‌کنه، و توصیه‌ی کلیشه‌ایِ «یه retry اضافه کن» انگار که فیکس باشه نه باندِ زخم. هر دو دستور مدل رو مجبور می‌کنن درباره‌ی ابهام صادق باشه — که اینجا بیشتر از اکثر Promptها مهمه، چون یه حدس غلط درباره‌ی علت ریشه‌ای مهندس رو می‌فرسته تو یه بیراهه‌ی نصف‌روزه دنبال فیکس اشتباه.
بخش Output Format عمداً به چهار بخش تقسیم شده نه یه جدول، چون ارزش واقعی تریاژ خود ردیف‌ها نیستن — بلکه اینه که بفهمی سه تست فلیکیِ «بی‌ربط» همگی به یه fixture مشترک برمی‌گردن. بخش اختصاصی «Systemic Issues» این ترکیب‌بندی رو صریحاً اجباری می‌کنه تا تو یه جدولی که یه مهندس خسته سرسری از روش رد می‌شه دفن نشه. بخش «This Sprint» خروجی رو به ظرفیت واقعی تیم گره می‌زنه، تا Prompt یه پلنی تولید کنه که یه نفر بتونه همون امروز اجراش کنه، نه یه لیست آرزویی از ۲۰ فیکس که هیچ‌کس وقتش رو نداره.
هفته‌ای یه بار یا قبل از هر فریزِ ریلیز اجراش کن و هر چیزی رو که داشبورد CI از دفعه‌ی قبل علامت زده بچسبون — این کار تمیزکاری تست فلیکی رو از یه بک‌لاگی که هیچ‌کس بهش دست نمی‌زنه تبدیل می‌کنه به یه تریاژِ ۱۵ دقیقه‌ایِ تکرارشونده.