The Observability Gap Finder: Turn a Production Incident Into the Monitoring You Should Have Had

Why this prompt matters
Postmortems that conclude "we need better monitoring" as an action item almost never get follow-through, because nobody can turn "better monitoring" into something assignable, estimable, or verifiable as done. The next incident repeats the same multi-service log bisection because the generic action item was never specific enough to act on.
What we use it for
An SRE team just finished a 90-minute incident where a checkout service silently failed for 12% of users before anyone noticed. The only way the team found out was a spike in support tickets, and engineers then spent 40 minutes manually bisecting logs across six services to find the root cause — with the postmortem meeting starting in two hours and nothing specific written down yet.
Prompt
Role: Act as a senior SRE/observability engineer who has run blameless postmortems and knows the difference between "we should monitor everything" (useless) and specific, actionable instrumentation gaps. Context: Here is the incident timeline: [PASTE A TIMELINE OF WHAT HAPPENED, WHEN IT STARTED, HOW IT WAS DETECTED, AND THE STEPS TAKEN TO DIAGNOSE AND RESOLVE IT] Here is our current monitoring stack: [LIST YOUR METRICS PLATFORM, LOGGING PLATFORM, ALERTING RULES, AND TRACING SETUP IF ANY, e.g. "Datadog dashboards on API latency and error rate, PagerDuty alerts on 5xx rate >1%, no distributed tracing"] Here is how the incident was actually detected: [e.g. "a customer support ticket," "an engineer noticed in Slack," "an alert fired 40 minutes after impact started"] Task: 1. Identify the specific moment in the incident timeline where better observability would have shortened detection or diagnosis time, and name exactly what signal was missing — not "better monitoring," but the specific metric, log field, trace span, or alert rule that didn't exist. 2. For each gap, classify it as a missing metric, a missing log field, a missing distributed trace span, or a missing/misconfigured alert threshold — these require different engineering fixes. 3. Rank the gaps by how much detection or diagnosis time each one would plausibly have saved, not by how easy each would be to implement. 4. Flag any gap that is actually a process gap disguised as a monitoring gap (for example, "the alert existed but nobody was on call for that service") — these need a different kind of fix than instrumentation. Constraints: - Do not recommend "add more logging everywhere." Every recommendation must name a specific signal and roughly where in the system it would be added. - Do not recommend new tools or platforms as the fix unless the actual gap is a genuine platform limitation, not a configuration or instrumentation gap within the current stack. - If the timeline doesn't give you enough detail to pinpoint a specific gap at some point, say so explicitly rather than inventing one. - Separate "would have caught it faster" from "would have prevented it" — most observability gaps only address the former, and conflating the two overstates what the fix will actually accomplish. Output Format: A table: Timeline Moment | Missing Signal Type | Specific Signal | Estimated Time Saved | Implementation Note. Followed by a section titled "Process Gaps (Not Monitoring Gaps)" if any apply.
Result
| Timeline Moment | Missing Signal Type | Specific Signal | Est. Time Saved | Implementation Note |
|---|---|---|---|---|
| T+0 to T+22min — checkout silently failing for 12% of users before anyone noticed | Missing metric | Per-payment-provider success rate, segmented by provider ID, not just the overall aggregate checkout success rate | ~20 min detection time | Add a metric tagged by provider_id with an alert threshold on any single provider dropping more than 15% week-over-week, independent of the aggregate rate |
| T+22 to T+35min — support tickets triggered the first manual investigation | Missing/misconfigured alert | Per-provider error rate was already being logged, but no alert rule existed on it — only the aggregate 5xx rate had an alert | ~13 min | This is a configuration gap, not a missing-instrumentation gap — the data existed, nobody wired an alert to it |
| T+35 to T+75min — 40 minutes spent bisecting logs across 6 services by hand | Missing distributed trace | No trace context was propagated from checkout-gateway through payment-router to the provider-adapter services; logs had to be manually correlated by timestamp and user ID | ~30 min | This is the real cost center. Implement W3C trace-context propagation across the checkout path; current logs share no request ID across services |
| T+75 to T+90min — root cause confirmed: Provider X's API silently changed a response field shape | Missing log field | Raw response schema/fingerprint from the provider adapter wasn't logged on successful (200) responses, only on explicit errors | ~10 min | Log a response-schema fingerprint on every provider call, not just failures — this bug returned a 200 with a changed field shape, so it never triggered existing error logging |
Process Gaps (Not Monitoring Gaps): The on-call engineer for payment-router was on PTO, and the secondary on-call wasn't informed they'd been escalated to primary. This added roughly 8 minutes to initial response time that no amount of better instrumentation would have fixed — it needs an on-call escalation policy review, not a monitoring change.
Most postmortem action items that say "improve monitoring" die quietly in a backlog, because nobody can turn that sentence into a ticket with a clear definition of done. The failure isn't a lack of good intentions — it's that the postmortem never named the specific signal that was missing, so the fix has no shape. This prompt exists to force that specificity before the meeting ends, not after.
The instruction to classify each gap as a metric, log field, trace span, or alert rule is the part doing the real work. These four categories require entirely different engineering effort and entirely different owners — a missing alert on existing data is a five-minute config change, while missing distributed tracing across six services is a multi-sprint project. Lumping them together as "add more observability" is exactly how a specific, fixable gap turns into a permanent backlog item nobody prioritizes.
Ranking gaps by time saved rather than ease of implementation is a deliberate correction to a common bias: teams naturally gravitate toward whatever's easiest to ship, which is rarely what would have actually mattered most in the incident. The prompt forces the ranking to reflect the incident timeline's actual bottleneck, even when that bottleneck is the hardest thing on the list to fix.
The "process gaps" section exists because postmortems routinely misdiagnose an on-call or ownership failure as a monitoring failure. An alert that fired correctly but reached nobody isn't an observability problem — it's an escalation-policy problem, and treating it as the former means the actual fix (checking who's covering what, and whether handoffs are confirmed) never gets written down.
Use case: an SRE team just finished a 90-minute incident where a checkout service silently failed for 12% of users before anyone noticed. The only way the team found out was a spike in support tickets, and engineers then spent 40 minutes manually bisecting logs across six services to find the root cause — with the postmortem meeting starting in two hours and nothing specific written down yet.
Why it matters: postmortems that conclude "we need better monitoring" as an action item almost never get follow-through, because nobody can turn "better monitoring" into something assignable, estimable, or verifiable as done. The next incident repeats the same multi-service log bisection because the generic action item was never specific enough to act on.
This works best when you have an actual incident timeline with real timestamps to analyze — it's a diagnostic tool for a specific incident, not a general audit of your monitoring maturity. Run it against a vague "we should monitor more" feeling rather than a concrete timeline, and it has nothing to anchor the analysis to.