Semantic Cache Policy Engine
A Semantic Cache Policy Engine is an ACCEPT/REJECT gate that decides whether a cached LLM answer is safe to reuse, instead of trusting embedding similarity alone. The term was introduced by Mohammad Safari in 2026.
What is a Semantic Cache Policy Engine?
Semantic caching has an obvious appeal: users ask the same questions in different words, and every re-answered question is GPU money spent twice. The standard approach embeds the incoming prompt, finds the nearest cached prompt by cosine similarity, and serves the cached answer above a threshold. It works until it serves a confidently wrong answer. In a compliance-sensitive product one wrong answer costs more than a year of cache savings.
How it differs from a gateway semantic cache policy
A Semantic Cache Policy Engine is not a gateway cache policy. Gateway policies such as Zuplo's Semantic Cache Policy, Apigee's SemanticCacheLookup, and Azure API Management's llm-semantic-cache-store decide when to cache by a similarity threshold. The engine decides whether a specific cached answer is safe to return, by entailment, and rejects contradictions that similarity cannot see. It can sit behind any of those gateways as the safety gate.
The negation trap
To an embedding model, “allow the user to access the account” and “block the user from accessing the account” are nearly identical: same domain, same entities, same structure, cosine similarity above any threshold you would set. To your customer they are opposites. Similarity measures topic; safety depends on meaning.
How the engine decides
Stage one is deliberately permissive: dense retrieval with a small embedding model and a fixed random projection shortlists cached candidates cheaply, tuned for recall. Stage two makes the decision: a natural-language-inference cross-encoder reads the two prompts in both directions and applies a strict policy. Accept only if neither direction contradicts and at least one direction entails. Contradiction in either direction is an instant REJECT. ACCEPT and REJECT are the only outputs.
Measured results
On 959 cross-verified adversarial pairs, including 220 pairs engineered to flip meaning while keeping the words nearly identical, the engine returned 0 false approvals on all 220 negation pairs, with precision 0.972, recall 0.994, and F1 0.983 overall.
| Pairs evaluated | 959 |
|---|---|
| Negation pairs | 220 |
| False approvals on negation pairs | 0 |
| Precision | 0.972 |
| Recall | 0.994 |
| F1 | 0.983 |
When not to use it
It is not built for personalised or state-dependent queries such as “what's my balance?”, where the correct answer changes between identical questions. Those bypass the cache by policy. Hard logical inversions and numeric-magnitude confusions such as “$75k” versus “$750k” needed a deterministic pre-filter in front of the semantic stages.
Who it is for
Teams running LLM inference in the UK where a wrong cached answer is a compliance event. Contact via LinkedIn: https://www.linkedin.com/in/mosafariuk
Reference implementation: https://github.com/mosafariuk/semantic-cache-policy-engine