Every RAG or agent application I’ve reviewed re-sends the same few thousand tokens on every request: the system prompt, tool schemas, a hundred pages of policy docs, the conversation so far. Those tokens are parsed and re-processed (re-KV-encoded) on every call, which shows up as both a line item and as TTFT. Prompt caching exists precisely because of this repetition, and the engineering is entirely in where you put the static content.

The mechanism is simple: the provider keeps the KV cache of a prompt prefix warm on GPU memory for a TTL, keyed by an exact token match of that prefix. On a cache hit, the prefill work for the cached prefix is skipped, so you pay almost nothing for it and the first new token arrives almost immediately. On a miss, you pay full prefill price and full latency. Everything that follows is about maximizing the hit rate, because the economics are not symmetric: on Anthropic, writing a cached block costs 1.25x base input price, reading it costs 0.1x; the docs claim up to 90% cost and 85% latency reduction on long prompts, and we reproduce those numbers at 10k+ token prefixes.

Layout rules that actually move the hit rate

The constraint that drives everything: caches match exact prefixes, and they grow from the front. Any dynamic token before a stable block invalidates that block, so ordering is the whole strategy.

• Static first, dynamic last. System prompt and instructions, then tool definitions, then the immutable corpus (policy docs, schema dumps, few-shot examples), then the genuinely per-request tail. If a request id or timestamp leaks into the first 100 tokens, you have just invalidated every cached block behind it. We strip timestamps, nonces, and request ids from prompt assembly entirely — they go into a header or a metadata field, never the prompt.
• Design explicit breakpoints. Providers let you tag where a cached segment ends (Anthropic’s cache_control breakpoints; Gemini’s cachedContent). A breakpoint costs write-price for the segment that precedes it, so breakpoints should partition your prompt at stable boundaries: one for the system prompt, one for the document corpus, none in the dynamic tail. A checkpoint every few paragraphs is how you pay 1.25x on five overlapping segments instead of one.
• Watch the minimums. Anthropic requires 1024 tokens before a cache_control segment is eligible; Gemini’s automatic caching kicks in above a 32k-token threshold and its explicit cache has its own minimum. Enabling caching on a 300-token system prompt does nothing except add a validation round trip — we filter out any tag on segments below the floor.
• Interleaving is the enemy. RAG that injects a different retrieved chunk in the middle of the prompt destroys caching for everything after it. Put all retrieval in one final block, or better, re-rank and truncate retrieval so the corpus prefix stays byte-identical.

Here is the shape we ship, with breakpoints on the two stable segments:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
import anthropic

client = anthropic.Anthropic()

system_blocks = [
    {
        "type": "text",
        "text": STAFF_SYSTEM_PROMPT,          # ~800 tokens, stable
        "cache_control": {"type": "ephemeral"},  # 5-minute TTL
    },
    {
        "type": "text",
        "text": POLICY_CORPUS[:100_000],      # the static document pool
        "cache_control": {"type": "ephemeral"},
    },
]

response = client.messages.create(
    model="claude-3-5-sonnet-20240620",
    max_tokens=2048,
    system=system_blocks,
    messages=[{"role": "user", "content": request_tail}],  # dynamic, uncached
)

hits = response.usage.input_tokens
cache_hits = response.usage.cache_read_input_tokens

We gate that call on the 1024-token floor, so cache_control only appears when the corpus is large enough to justify it. And we read the cache_read_input_tokens field back into a Prometheus counter: hit rate is the single most useful SLO for this system, and a silent drop from 95% to 60% hit rate is how you learn that someone started stamping a build hash into the system prompt.

Multi-turn is where this pays off hardest

The biggest wins come from conversational traffic: an agent thread where the system prompt, tool definitions, and every prior user/assistant turn form a shared prefix. Each turn appends to the cached prefix instead of rebuilding it, so the second, fifth, and fifteenth turns all pay the 0.1x read price on the accumulated conversation. The failure mode we hit in production was keeping the cached prefix but letting the agent’s scratchpad (thinking traces, intermediate tool results) pollute the message history — a tiny byte difference in turn N invalidates turns N+1 through N+15. Our rule: cache only what the client re-sends byte-for-byte.

TTL and economics

The ephemeral cache lives 5 minutes; Anthropic also offers a 1-hour TTL with a more expensive write. For bursty traffic (a load spike every few minutes) 5 minutes gives you near-zero waste. For steady streams of identical prefixes — an eval harness hammering the same prompt, a nightly batch job — the 1-hour block pays for itself and we use it. The waste case is a low-TTL cache on traffic that never repeats; set TTL from the measured inter-request gap of the static prefix, not vibes.

Production lessons

• Treat hit rate as a code-review topic. Caching layout is a design property of the prompt template; reviewers should block on dynamic tokens in the prefix like they’d block on an unbounded query.
• Measure cache-read tokens, not just cost. Latency variance from a cache miss is large — a cold start can be 10x the warm TTFT. If your p99 TTFT is spiky, check hit rate first.
• Don’t cache secrets into prompt templates. A cached block is provider-side GPU memory shared across your traffic; keep credentials out of the corpus, same as you would in logs.

Prompt caching is the cheapest latency optimization in LLM engineering: it requires no model changes, no infra, and no sampling tricks — just the discipline to keep your static prefixes static.