Prompt caching is cutting AI inference costs by up to 90 percent, and most teams aren't using it right

Most teams paying for LLM API access are leaving 50 to 90 percent of potential savings on the table, and the mechanism to capture it has existed in production for over a year. Prompt caching lets an API provider reuse the computation it already did for the unchanging parts of a request — system instructions, tool definitions, long reference documents — instead of reprocessing them on every single call.

The problem is not that prompt caching doesn't work. It's that most engineering teams structure their prompts in a way that defeats it before it ever gets a chance to help, then conclude the feature "didn't move the needle" for their use case.

How the Cache Actually Works

When a transformer model processes a prompt, it computes key-value (KV) pairs for every token in the input — the internal representations the model uses to generate its next-token predictions. This computation is the expensive part of inference, scaling with input length. Prompt caching stores these KV pairs on the provider's servers after a first request, keyed to the exact prefix of tokens that produced them.

On a subsequent request, if the new prompt shares that same prefix, the provider skips recomputing the KV pairs for the shared portion and only processes the new tokens appended at the end. Anthropic's implementation, for example, charges roughly 10 percent of the base input token rate for cache reads and about 25 percent more than the base rate for the initial cache write — a tradeoff that pays off quickly for any prefix reused more than a couple of times. OpenAI's automatic prompt caching applies a 50 percent discount on cached input tokens with no separate write cost, and Google's context caching on Gemini models works similarly with an explicit cache object you create and reference.

Where the Real Savings Come From

The savings aren't theoretical. A customer support agent that includes a 3,000-token system prompt, a 5,000-token product knowledge base excerpt, and a 2,000-token set of tool definitions on every call is paying for 10,000 tokens of input before the user's actual question — maybe 50 tokens — even enters the picture. Across a high-volume application processing tens of thousands of requests a day, that overhead dwarfs the marginal cost of the query itself.

With caching correctly configured, that 10,000-token prefix gets computed once and then read from cache on every subsequent call within the cache's time-to-live window (typically 5 minutes on Anthropic's standard tier, extendable to an hour on some providers). The practical effect for a RAG pipeline with a large retrieved-context window, or an agent with an extensive tool-calling schema, is a 60-80 percent reduction in effective input cost — without changing model quality, output behavior, or latency in any way a user would notice. Latency actually improves slightly, since cached tokens skip the forward pass entirely.

The Mistake That Kills Cache Hit Rates

Prompt caching only works on exact prefix matches. If a single token changes anywhere before the cached boundary, the entire cache entry misses and the provider recomputes from scratch. This is where most implementations quietly sabotage themselves.

The most common failure pattern: putting dynamic content — a timestamp, a session ID, a randomly-ordered list of retrieved documents — near the beginning of the prompt, ahead of the static system instructions and tool definitions. Every request then has a unique prefix, and the cache never has a chance to build up hits. The fix is mechanical but requires discipline: structure every prompt so static content comes first (system instructions, few-shot examples, tool schemas, stable reference documents) and variable content comes last (the user's actual message, current date if needed, session-specific state).

A second common mistake is over-aggressive prompt personalization. Teams that inject user-specific preferences or account details directly into the system prompt — rather than passing them as a separate, clearly-delimited block after the cacheable prefix — destroy their own cache hit rate for the sake of a marginal quality improvement that often isn't measurable. If personalization data changes per user or per session, it belongs after the cache boundary, not woven into the shared instructions.

Retrieved Context Is the Hardest Case

RAG applications face a genuine architectural tension: the whole point of retrieval is to surface different documents for different queries, which by definition breaks prefix matching on the retrieved content itself. The highest-leverage fix here isn't caching the retrieved chunks — it's caching everything around them. Keep the system prompt, the retrieval instructions, and the output format specification in a stable prefix, and treat the retrieved documents as the variable suffix. Teams running large tool libraries can get an additional win by caching the tool definition block separately from the retrieved content, since tool schemas rarely change within a session even when the query does.

Some teams take this further with a two-tier cache strategy: a long-lived cache for genuinely static content (product documentation, coding standards, brand voice guidelines) refreshed only when the source material changes, and a short-lived cache for session-level context that's reused across a multi-turn conversation but discarded once the session ends.

Measuring It Correctly

Provider dashboards typically report cache read and cache write token counts separately from standard input tokens, which makes it straightforward to calculate actual hit rate: cache reads divided by total prefix-eligible tokens. Teams should track this number explicitly rather than inferring savings from the total bill, since a low hit rate can hide behind an otherwise-reasonable invoice if request volume is also fluctuating.

Takeaways

• Audit your prompt structure first: static content (system instructions, tool schemas, reference documents) must come before any per-request variable content, with zero exceptions.
• Don't personalize inside the cacheable prefix — inject user-specific data as a clearly separated block after the shared instructions.
• For RAG pipelines, cache the stable scaffolding around retrieved content, not the retrieved content itself.
• Track cache hit rate as an explicit metric, not an inferred one from total spend.
• Check your provider's cache TTL and request patterns — traffic gaps longer than the TTL window will force a fresh cache write regardless of prompt structure.