The AI Vendor Claims Auditor: Turn a Sales Pitch Into a List of Questions That Expose Overselling
Why this prompt matters
AI vendor contracts are typically 12-24 month commitments signed after a handful of demos, and claims like '95% accuracy' or 'seamless integration' are almost never independently verified before signing. When the real numbers surface during implementation, often eight to ten weeks in, the buyer is already locked into the contract and has sunk engineering time into integration work the vendor never scoped. A structured audit before signing catches the gap between the pitch and the product while there's still leverage to renegotiate, request a pilot on real data, or walk away.
What we use it for
Your operations team is three weeks into evaluating an AI-powered customer support platform for a 200-person SaaS company. The vendor's deck claims '95% ticket resolution without human intervention,' 'seamless integration with your existing helpdesk in under a week,' and 'enterprise-grade accuracy.' Nobody on the five-person buying committee has an AI engineering background, so no one in the room can tell which claims are standard industry performance and which are marketing inflation designed to close the deal before technical due diligence catches up.
Prompt
Act as a skeptical, technically literate AI procurement consultant whose job is to protect [YOUR COMPANY NAME] from buying AI capability that won't actually work as advertised. Context: We are evaluating an AI vendor for [SPECIFIC USE CASE, e.g., customer support automation, sales forecasting, document processing]. Here is the vendor's pitch, marketing copy, or sales deck content: [PASTE VENDOR PITCH / MARKETING CLAIMS HERE]. Our current technical environment is: [DESCRIBE YOUR STACK, DATA VOLUME, TEAM SIZE]. Our must-have requirements are: [LIST NON-NEGOTIABLE REQUIREMENTS]. Task: Extract every concrete, testable claim the vendor makes (performance numbers, integration promises, accuracy rates, time-to-value estimates). For each claim, assess how plausible it is given current AI capabilities, identify the specific conditions under which it is likely true versus misleading, and write one pointed verification question we should ask the vendor directly. Constraints: - Do not dismiss every claim as marketing fluff. Distinguish claims that are technically reasonable from claims that require unusual, unstated conditions to be true. - Flag any claim that depends on an unstated baseline (for example, "95% accuracy" without saying accuracy on what dataset, measured how). - Note where the vendor's language shifts from concrete numbers to vague terms like "seamless," "enterprise-grade," or "state-of-the-art" — these phrases usually signal an unverifiable claim. - Rate each claim's overselling risk as Low, Medium, or High, with a one-sentence justification. Output Format: A table with columns: Claim | Plausibility Assessment | Verification Question to Ask Vendor | Overselling Risk (Low/Medium/High). Follow the table with a 3-5 bullet summary of the two or three highest-risk claims and what evidence we should demand before signing.
Result
| Claim | Plausibility Assessment | Verification Question | Overselling Risk |
|---|---|---|---|
| "95% ticket resolution without human intervention" | Plausible only for a narrow band of simple, repetitive ticket types (password resets, order status, basic FAQs). Across a full mixed ticket queue including billing disputes and technical escalations, published industry benchmarks cluster closer to 60-75% autonomous resolution. The 95% figure almost certainly reflects a filtered subset, not your full queue. | "What ticket categories and complexity levels were included in the 95% figure, and what was the resolution rate on your three most complex recurring ticket types specifically?" | High — unstated baseline, classic cherry-picked metric. |
| "Seamless integration with your existing helpdesk in under a week" | Plausible if you use one of the 3-4 major helpdesk platforms with a pre-built connector and don't need custom field mapping. Becomes implausible fast if your ticket taxonomy, custom fields, or approval workflows deviate from the vendor's default template. | "Can you name three customers on our exact helpdesk platform and ticket volume who went live in under a week, and what customization, if any, did they skip?" | Medium — plausible under specific conditions the pitch doesn't disclose. |
| "Enterprise-grade accuracy" | Not a measurable claim as stated. "Enterprise-grade" has no industry-standard definition or benchmark attached to it. | "What specific accuracy metric, dataset, and evaluation methodology does 'enterprise-grade' refer to, in numbers?" | High — vague qualifier standing in for a number. |
| "Trained on millions of support conversations across industries" | Plausible as a training-data claim, but training data volume does not predict performance on your specific product, terminology, or customer base without domain adaptation. | "How much of that training data is from our industry vertical, and what's the accuracy delta between your general model and a model fine-tuned on our historical tickets?" | Medium — technically true but doesn't imply what it's meant to imply. |
Summary — top three risks before signing:
- The 95% resolution claim needs a breakdown by ticket complexity before it means anything; demand a pilot on your actual historical ticket data, not a vendor demo dataset, before accepting this number.
- "Enterprise-grade accuracy" should be rejected as a contract term entirely — replace it with a specific, measurable SLA (e.g., "≥85% resolution rate on tickets tagged Tier 1-2, measured monthly") before signing.
- Request three reference customers matching your exact helpdesk platform, ticket volume, and industry — not generic case studies — and verify their actual go-live timeline and post-launch resolution rate directly, not through the vendor's summary.
Most AI procurement audits fail for a structural reason: the person reading the pitch deck is rarely the person who understands where AI claims tend to be technically shaky. A sales leader can spot an unrealistic revenue projection instantly, but "95% ticket resolution" or "enterprise-grade accuracy" sounds authoritative to someone without a technical background, precisely because those phrases are designed to sound authoritative rather than to convey verifiable information.
This prompt exists to close that specific gap — not by making the buyer cynical about AI in general, but by giving them a repeatable method for separating claims that are technically reasonable from claims that are structurally unverifiable as stated.
Why the structure works
The Role instruction — "skeptical, technically literate" — matters because a generic "review this pitch" prompt tends to produce a summary, not an audit. Explicitly assigning the adversarial-but-fair posture is what pushes the model to extract testable claims rather than restate the pitch in different words.
The Constraints section does the real work. Telling the model not to dismiss every claim as fluff prevents a lazy, uniformly negative output that would be as useless as uncritical acceptance. The instruction to flag unstated baselines is the single most valuable line in the prompt — it's the pattern behind nearly every misleading AI performance claim: a real number, measured under unstated and usually favorable conditions.
Why the output format is a table, not prose
A buying committee needs to walk into a vendor call with specific questions, not a general impression. The four-column format forces one verification question per claim, which is what turns this from an internal risk assessment into an actual negotiating tool you can use live on a vendor call.
Where this prompt earns its keep
This is most valuable in the window between a vendor demo and contract signature — after the initial excitement, before the money moves. Running the full pitch deck or sales email through this prompt takes minutes and routinely surfaces two or three questions that would otherwise only surface eight weeks into implementation, when the leverage to renegotiate has already disappeared.