Prompt Caching
Prompt caching reuses the model's work on a stable prompt prefix across calls: mark a long, unchanging span (a system prompt, a big document, a tool schema, few-shot examples) as cacheable, and subsequent requests that share that prefix skip re-processing it — cutting both cost and time-to-first-token on the repeated portion. The economic shape is the point: when the same large context is sent many times (an agent loop re-sending its instructions each turn, a RAG app re-sending the same corpus, a chatbot with a fat system prompt), the prefix is the dominant cost and caching it is the single biggest lever. The ordering constraint follows directly — put the stable content first and the volatile content last, so the cacheable prefix is as long as possible before the first byte that changes (this is why it pairs with Evergreen vs Volatile Context). Speculative prompt caching extends the idea by warming the cache for a prefix the system predicts will be reused, before the reuse actually arrives.
Anthropic's Claude Cookbooks ship this as a first-party recipe
(misc/prompt_caching.ipynb and
misc/speculative_prompt_caching.ipynb), attributed here as
the source's; the concept is a durable inference-economics technique
independent of any one vendor's cache-lifetime or pricing.
Claims
- Reusing a cached prompt prefix cuts the cost and latency of the repeated context. principle — durable inference economics: recomputing an identical prefix on every call is pure waste; caching it is a structural saving, not a tuning trick.
- Order the prompt stable-prefix-first, volatile-suffix-last, to maximize the cacheable span. best practice — context: prompts with a large fixed part (system prompt, document, tools) and a small changing part (the user turn); "best" because cache hits require an unchanged prefix, so any early-placed volatile token shortens what can be cached. Contingent on the volatile part being genuinely separable — see Evergreen vs Volatile Context.
- Prompt caching pays off in proportion to how often the same prefix is re-sent. best practice — context: agent loops, RAG, and long-system-prompt chat that re-send a big fixed context every turn; for one-off single calls there is nothing to reuse and the technique is moot.
- Speculative prompt caching warms the cache for a
predicted-reuse prefix before the reuse arrives.
observation — the source ships
a distinct
speculative_prompt_cachingrecipe; attributed as the cookbook's, a refinement of the base technique. - The cached "prefix" is a chain of KV blocks hashed by their token sequence; the first differing token ends the chain. principle — the mechanism underneath the ordering rule above. Prefix caching is not a bespoke feature but a consequence of Paged Attention's content-addressed pages: two requests sharing a system prompt resolve to the same physical GPU pages. This is why "stable content first" is structural rather than stylistic.
- A cached prefix is not free — it occupies physical VRAM that could otherwise hold concurrent requests. principle — at the API boundary the cost is invisible because the vendor absorbs it; anyone running their own serving stack pays it directly. The KV Cache is a memory-for- compute trade, and cache reuse spends the memory side of that ledger.
- Prefix-cache hit rates of 75–95% are common in RAG pipelines, multi-turn chat, and coding agents. observation — reported by the IBM Technology vLLM explainer, an independent second source converging on the same lever; check-worthy.
Related
- Evergreen vs Volatile Context — the stable/volatile split is what makes a long cacheable prefix possible; prompt caching is that distinction cashed out as cost/latency savings.
- Reasoning Effort Control — a sibling inference-economics lever (spend on thinking vs. spend on re-processing context); both are about paying only for the tokens that earn their cost.
- Model-Tier Routing — the other main cost lever (cheap model for easy calls); caching reduces the cost of the context, routing reduces the cost of the model.
- Paged Attention — the serving-layer mechanism that makes prefix reuse possible; block hashing is why the stable-prefix-first rule exists.
- KV Cache — the cache being reused; prompt caching is the same cache with a wider reuse boundary (across requests, not just within one).
- Prefill and Decode Phases — a prefix-cache hit is prefill skipped entirely, which is exactly why it moves time-to-first-token.
- Distillate: Anthropic's Claude Cookbooks — the canonical recipe index — first-party base + speculative recipes.
- Distillate: KV Cache and Paged Attention: Why LLM Serving Is Memory-Bound — the mechanism from underneath (vLLM paged attention + block hashing), and the VRAM cost the API surface hides.