The Survivorship Bias Detector: Find What Your Dataset Silently Excludes Before You Trust Its Conclusion
Why this prompt matters
Teams that build roadmaps from survivor-only data routinely ship the feature their happiest customers want most, watch retention stay flat, and don't find out why until a full product cycle has been spent on the wrong problem — the cost isn't the survey, it's the quarter or two of engineering time spent optimizing for people who were never going to leave.
What we use it for
A product manager is reviewing survey results from a company's most engaged, long-tenured customers to decide which feature to prioritize on the roadmap, and needs to know whether the survey population quietly excludes the exact customers the roadmap is supposed to help retain.
Prompt
Act as a rigorous data skeptic whose specialty is finding survivorship bias — cases where a dataset silently excludes everyone or everything that didn't make it to the point of measurement. CONTEXT: Dataset: [DESCRIBE THE DATASET, e.g. "survey responses from customers who have used our product for 12+ months and are flagged as highly engaged"] How it was collected or filtered: [DESCRIBE THE FILTER OR SELECTION PROCESS THAT PRODUCED THIS POPULATION] Conclusion currently being drawn from it: [STATE THE CONCLUSION OR DECISION THIS DATA IS BEING USED TO SUPPORT] TASK: 1. Identify who or what is structurally absent from this dataset — not random noise, but a systematic exclusion created by how the data was collected. 2. For each absence, state what the current conclusion implicitly assumes about the missing group, and whether that assumption is plausible or just convenient. 3. Describe the specific way the conclusion could flip, weaken, or reverse if the missing group behaved differently than assumed. 4. Rank the exclusions by how much they threaten the conclusion if the assumption turns out to be wrong. 5. For the top exclusion, propose one concrete way to check it — a data source to pull, a question to ask, or a quick analysis to run — rather than just flagging it as a limitation. CONSTRAINTS: - Do not flag generic "sample size" or "margin of error" issues — only genuine survivorship/selection exclusions created by how the population got into the dataset, not random variance. - Assume the person asking has already heard "correlation isn't causation" warnings; go past that to the specific mechanism of exclusion. - If an exclusion is already accounted for in the analysis, say so explicitly and move to the next most severe exclusion instead of inventing a problem. OUTPUT FORMAT: ## Structural Exclusions (ranked by threat to the conclusion) 1. [Who/what is missing] -> [What the conclusion assumes about them] -> [How the conclusion could flip if wrong] ... ## Highest-Priority Check [One concrete, doable check for the #1 exclusion] ## Verdict [One paragraph: does the conclusion survive scrutiny as-is, does it need a caveat, or shouldn't it be trusted until the top check is done?]
Result
Structural Exclusions (ranked by threat to the conclusion)
1. Customers who churned before reaching 12 months are structurally absent from this survey. The conclusion assumes the feature preferences of people who stuck around for a year represent what would retain MORE customers, or even represent the full customer base's needs. High threat: if most churn happens in months 1-3, the 68% of long-tenured survivors wanting advanced reporting tells you nothing about why 60% of new signups leave before month 3 — you could ship advanced reporting and see zero change in overall retention, because you optimized for people who were never at risk of leaving.
2. "Highly engaged" is itself a filter that selects for people who already found value. The conclusion implicitly assumes engaged users' stated feature requests represent unmet needs rather than wishlist additions from people who are already satisfied. Medium-high threat: power users asking for advanced reporting might just want to do more with a tool they already love, while the actual retention-blocking gap — say, a confusing onboarding flow — never surfaces because nobody who hit that wall is in the room to report it.
3. Survey non-response within the power-user group itself. The conclusion assumes the roughly 30-40% of power users who didn't respond share the same preferences as respondents. Medium threat: if non-respondents are disproportionately the busiest, most senior users, the voice actually driving revenue may be underrepresented in the 68% figure.
4. The survey only asks about features the product team already thought to list. This doesn't exclude people, but excludes possible answers — the conclusion assumes the menu of options covers the real need. Low-to-medium threat: usually resolved by an open-text field, but worth flagging since it compounds the other exclusions.
Highest-Priority Check
Pull churn-stage data for the last two quarters: what percentage of churned customers left in months 1-3 versus after month 6? If the majority churn early, cross-reference exit surveys or support tickets from that group — even a sample of 20-30 — for what they cite as the blocker. That single data pull will tell you whether advanced reporting addresses the group actually at risk, or only deepens satisfaction among people who were never going to leave.
Verdict
The 68% figure is real and the feature request is legitimate, but the conclusion that it should top the roadmap for retention reasons doesn't survive scrutiny yet — it conflates "what remaining users want more of" with "what would stop users from leaving," which are different questions answered by different populations. Advanced reporting may still be the right call for expansion revenue or power-user satisfaction, but that is a different justification than the one currently being used, and the team should re-run this with early-churn data before treating it as a retention fix.
The classic survivorship bias story is the WWII bomber: engineers wanted to reinforce the parts of returning planes with the most bullet holes, until statistician Abraham Wald pointed out that the planes in front of them had survived those hits — the parts worth reinforcing were the ones with no holes on the surviving planes, because planes hit there never made it back to be studied. The bias isn't bad data; it's a population that was filtered before anyone started measuring it.
This prompt translates that insight into a repeatable business check. The context section forces three separate inputs — the dataset, how it was collected, and the conclusion being drawn — because survivorship bias lives specifically in the gap between those three things. A dataset can be perfectly accurate and the conclusion still wrong, because the selection mechanism that produced the dataset is doing work nobody accounted for.
The instruction to rank exclusions by threat to the conclusion, rather than just listing them, is what keeps this useful instead of becoming a generic "there are always caveats" exercise. Not every absence matters equally — a dataset can have three or four structural gaps and only one of them is actually load-bearing for the specific decision at hand. Forcing a single "highest-priority check" at the end, rather than a list of things that would be nice to verify, is what turns the output into something a team can actually act on this week instead of filing away as a caveat nobody follows up on.
The constraint excluding generic sample-size or margin-of-error complaints exists because those are a different, much more commonly flagged problem, and a model without that constraint will often default to them instead of doing the harder work of finding the actual selection mechanism. The instruction to acknowledge when an exclusion is already handled, rather than inventing a problem, matters for the same reason the constraint on the Analogy Stress-Tester prompt does: a reviewer that only ever finds fault stops being useful the moment every analysis gets the same treatment regardless of its actual quality.
This is most valuable exactly where survivorship bias causes the most expensive mistakes: any time a team is using a population that "made it" to a measurement point — retained customers, completed projects, surviving portfolio companies, employees who didn't quit — to draw conclusions meant to apply to the people who didn't make it there at all.