Prompt caching, hands-on

1 · What prompt caching is essential

Here's a bill that surprises everyone the first time. You build a support agent with a 20,000-token system prompt — a company handbook pasted into the context so the model always has the policies to hand. It works beautifully. Then the invoice arrives, and you realize you paid to process those same 20,000 tokens on every single request, thousands of times a day, even though not one byte of the handbook ever changed. You were charged full price, over and over, to re-read a document the model had already seen a second ago. Prompt caching is the fix — and once you see the waste, you can't unsee it.

The reason this happens is that a language model has no memory between API calls. Each request is a fresh, stateless event: the model wakes up knowing nothing, reads your entire prompt from the first token to the last, and only then starts answering. There's no "oh, you again — I still have your handbook loaded"; there is just a giant block of text arriving cold. The expensive first phase of that — where the model chews through all your input to build up its internal state before it writes a single word of reply — is called prefill, and prefill cost scales directly with how many input tokens you send. A big fixed prefix means a big fixed prefill bill, paid in full every time.

The tangible analogy: imagine a brilliant consultant who, by contract, must re-read the entire 200-page policy binder from scratch at the start of every meeting before answering your one new question. Genius answers — absurd process. What you obviously want is for them to read the binder once, keep it fresh in their head, and on the next meeting just field the new question. Prompt caching gives the API exactly that: process the stable material once, remember the computed state, and on later calls skip straight to the new part.

The common mistake is treating this as a discount you bolt on at the end without changing how you build the prompt. It isn't — it's an architectural choice about ordering. Caching only helps the part of your prompt that stays byte-for-byte identical across calls, and only the part that comes before the changing content. If your stable handbook sits after this-request's question, there is no reusable prefix and nothing to cache. So the real skill here isn't a magic parameter — it's learning to assemble prompts as "big stable thing first, small changing thing last," which is exactly the shape the rest of this page drills into.

Every time you call the model, the whole prompt — your system prompt, tool definitions, and the conversation so far — is re-read and re-processed from scratch. If a big chunk of that prompt is identical from one call to the next (a long system prompt, a big document you keep asking about, a fixed list of tool definitions), you are paying to process the same tokens over and over.

Prompt caching lets the API remember the computed state of a stable prefix and reuse it on the next call. The first call writes the cache; later calls with the same prefix read it — skipping the re-computation. You mark where the stable part ends with a cache breakpoint (cache_control), and the API caches everything up to and including that point.

The rule that governs everything: caching is a prefix match. The cache key is the exact bytes up to the breakpoint. On the first call that prefix is computed and stored (a cache write). On later calls with a byte-identical prefix, the stored state is reused (a cache read) and only the variable suffix after the breakpoint is processed fresh.

2 · Why it saves cost and latency essential

Most people reach for caching to save money, then discover the latency win is what their users actually feel. Two separate bills come due on every input token: you pay in dollars (input tokens are billed) and you pay in time (the model has to prefill those tokens before it can start replying). A cached prefix cuts both at once — and if you serve a chat UI, the time cut is the one people notice, because it's the difference between a reply that starts typing instantly and one that stalls for two seconds while the model re-reads a document it saw moments ago.

Think of it like a compiler's build cache. Compiling from scratch is slow and burns CPU; a warm build cache reuses the work it already did and finishes in a fraction of the time. Prompt caching is the same trade for a language model: the "compiled" internal state of your stable prefix is stored, and later calls reload it instead of rebuilding it. The reload isn't quite free — there's a lookup and a small serving cost — but it's a tiny fraction of doing the whole prefill over.

The mistake to guard against is thinking the discount is automatic and enormous on every call. It isn't: the very first call is more expensive, not less, because someone has to pay to build the cache in the first place. Caching is an investment that pays back only on reuse — the win comes from the second call onward, and it compounds the more times you hit the same warm prefix. Which is why the two numbers below — the write premium and the read discount — are the whole economic story, and section 5 turns them into arithmetic you can run.

Reusing the computed prefix is cheaper on both axes. On cost: a cache read is billed at roughly 0.1× the normal input price for those tokens, versus the full 1× you'd otherwise pay every call. On latency: the model skips the prefill work for the cached tokens, so time-to-first-token drops — often dramatically when the prefix is large (tens of thousands of tokens of system prompt or document).

The catch is the write: the first call pays a small premium — about 1.25× for the default 5-minute cache (or 2× for the 1-hour TTL) — to store the prefix. So caching pays off once you reuse the prefix enough to amortize that write. With the 5-minute cache, that's just two calls.

3 · The request shape — cache_control on a content block essential

This is the section where "caching" stops being a concept and becomes one line of JSON. After all the theory, the actual mechanism is almost anticlimactic: you attach a small marker — "cache_control": {"type": "ephemeral"} — to a block of your prompt, and everything up to and including that block becomes cacheable. There is no separate "cache API," no key you manage, no store to configure. You just point at the end of your stable material and say "cache up to here."

The mental model that makes this click: think of it like a bookmark, not a highlighter. You aren't selecting a region to cache — you're dropping a single marker that says "the stable prefix ends here." The API caches from the very start of the prompt up to that bookmark, and treats everything after it as fresh, variable content. That's why placement is everything: the marker's job is to sit exactly on the boundary between "never changes" and "changes every call."

One detail trips up nearly everyone, and it's worth pausing on: to attach cache_control, the system field (or a message's content) has to be a list of content blocks, not a plain string. A bare string has nowhere to hang the marker. So the very first change caching forces on your code is turning system="..." into system=[{"type": "text", "text": "...", "cache_control": {...}}]. And the ordering rule underneath it is fixed and non-negotiable: the API renders your request as tools → system → messages. Because a breakpoint caches everything before it, dropping one marker on the last system block conveniently captures your tool definitions and your whole system prompt together — the two things most likely to be stable — in a single stroke.

You cache by putting "cache_control": {"type": "ephemeral"} on the last content block of the stable prefix. Render order is tools → system → messages, so a breakpoint on the last system block caches the tools and the system prompt together. The block below is real Anthropic SDK — it needs a key and a network connection to run.

cache_system.py# needs: pip install anthropic + ANTHROPIC_API_KEY
# ▶ needs API key + network — this really calls the API; it will NOT run offline.
import anthropic

client = anthropic.Anthropic()

LARGE_SYSTEM = open("company_handbook.txt").read()   # e.g. tens of thousands of tokens

resp = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LARGE_SYSTEM,
            # Breakpoint at the END of the stable prefix. Everything up to and
            # including this block is cached; the user message after it is not.
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[{"role": "user", "content": "Summarize the PTO policy in 3 bullets."}],
)

print(resp.content[0].text)

4 · Reading the usage fields — proof of a hit intermediate

Here's the trap that catches confident engineers: caching fails silently. When it doesn't work, nothing breaks. No error, no warning, no exception — your agent answers exactly as before. The only symptom is a bill that's quietly 10× higher than you assumed, discovered weeks later when someone finally reads the invoice. A stray timestamp in your system prompt can turn every "cached" call into a full-price cold write, and your code will never say a word about it. That's why this section isn't optional polish — it's the safety check that tells you whether the thing you built is actually saving money or just looking like it should.

The good news is that the API hands you a receipt on every single response. Buried in the usage object are three token counters that add up to a complete story of what just happened: how many tokens were written to the cache, how many were read from it, and how many were processed fresh at full price. Reading them is like checking the itemized receipt after a shop tells you a discount was applied — you don't take their word for it, you look for the line that says the discount actually came off.

The tell you're hunting for is a simple flip between the first call and the second. On a cold prefix, the write counter is large and the read counter is zero — you paid to populate the cache. On the very next call with a byte-identical prefix, they swap: the write counter drops to zero and the read counter jumps up to (roughly) the same number. That flip — write on call one, read on call two — is the fingerprint of caching working. The common mistake is skipping this check entirely and assuming the marker "did its thing"; the second most common is seeing the read counter stubbornly stuck at zero and blaming the API, when the real cause is almost always something in your own prefix changing between the two calls.

Never assume caching worked — measure it. Every response carries a usage object with three token fields that together tell the whole story:

On the first call you expect a big cache_creation_input_tokens and a zero cache_read_input_tokens. On the second identical-prefix call they flip: cache_creation drops to 0 and cache_read jumps up. The block below reads those fields — real SDK, needs a key and network.

read_usage.py# needs: pip install anthropic + ANTHROPIC_API_KEY
# ▶ needs API key + network — real API calls; will NOT run offline.
import anthropic

client = anthropic.Anthropic()

system = [{
    "type": "text",
    "text": open("company_handbook.txt").read(),
    "cache_control": {"type": "ephemeral"},
}]

def ask(question):
    resp = client.messages.create(
        model="claude-opus-4-8",
        max_tokens=256,
        system=system,                                   # byte-identical prefix each call
        messages=[{"role": "user", "content": question}],
    )
    u = resp.usage
    print(f"created={u.cache_creation_input_tokens} "
          f"read={u.cache_read_input_tokens} "
          f"fresh={u.input_tokens}")
    return resp

ask("Summarize the PTO policy.")     # 1st: created=<big>  read=0
ask("What about parental leave?")    # 2nd: created=0      read=<big>

5 · The cost math (runs offline) intermediate

Before you wire up a single API call, you can answer "is caching even worth it here?" with grade-school arithmetic. That matters because caching is not free and not always a win — and the failure mode is subtle. Cache a prefix you only ever use once and you've made things worse: you paid the write premium for a discount you never collected. The whole decision turns on one question — how many times will this exact prefix be reused? — and that's a number you can estimate on a napkin.

The analogy is a warehouse-club membership. You pay an up-front fee (the write premium) to unlock a lower per-item price (the cheap reads). One visit and the membership was a waste — you'd have been better off paying retail. But shop there regularly and the fee amortizes to nothing while the per-item savings pile up. Prompt caching has exactly this shape: a one-time cost you're betting you'll earn back, and then some, through repetition.

Only two multipliers drive the whole model, and they're the ones from section 2: a write costs about 1.25× a normal input token (the membership fee), and a read costs about 0.1× (the members' price). Everything else is counting. Uncached, N calls pay full price N times. Cached, you pay 1.25× once and 0.1× on the other N−1 calls. The mistake to avoid is reasoning about this in your head at scale — the intuition that "caching saves ~90%" is only true in the limit of heavy reuse, and wildly wrong at low reuse where the fixed write premium dominates. The tiny function below makes the crossover concrete: it's pure standard library, runs with a plain python cache_savings.py, and prints both the steady-state savings and the exact break-even point where caching stops being a loss and starts being a win.

You can decide whether caching is worth it with arithmetic, no API needed. Without caching, N calls each pay full price (1×) for the prefix. With caching, the first call pays the write premium (~1.25×) and the other N−1 calls pay the read rate (~0.1×). The helper below is pure stdlib — it runs with a plain python file.py — and reports the tokens saved as a percentage.

cache_savings.pydef cache_savings(prefix_tokens, calls, write_mult=1.25, read_mult=0.1):
    """Compare paying full price N times vs. write-once + read (N-1) times.
    Returns (no_cache_units, with_cache_units, percent_saved) in token-equivalents.
    """
    no_cache = prefix_tokens * calls
    with_cache = prefix_tokens * write_mult + prefix_tokens * read_mult * (calls - 1)
    saved_pct = round(100 * (no_cache - with_cache) / no_cache)
    return round(no_cache), round(with_cache), saved_pct

no_cache, with_cache, saved = cache_savings(prefix_tokens=10_000, calls=100)
print(f"no cache:   {no_cache} token-units")
print(f"with cache: {with_cache} token-units")
print(f"saved:      {saved}% on the cached prefix")

# Break-even: at how few calls does caching already win?
one, two = cache_savings(10_000, 1)[1], cache_savings(10_000, 2)[1]
print(f"1 call costs {one} (write premium, a loss); 2 calls cost {two} (already a win)")

no cache:   1000000 token-units
with cache: 111500 token-units
saved:      89% on the cached prefix
1 call costs 12500 (write premium, a loss); 2 calls cost 13500 (already a win)

Two lessons from the numbers. First, at 100 reuses of a 10K-token prefix you save ~89% of the prefix cost. Second, a single call costs 1.25× — caching a prefix you'll use only once is a small loss. It turns a profit at the second call and climbs from there.

6 · TTL and what invalidates the cache advanced

You've proven a cache hit and done the math — now here's what silently takes the hit away again. Two forces expire a cache entry, and confusing them is the source of most "it worked yesterday" caching bugs. The first is time: an entry doesn't live forever, it decays. The second is change: alter the prefix by a single byte and the entry you had is instantly worthless. Understanding both is what separates a cache that quietly delivers savings for months from one that "works in the demo" and mysteriously stops.

Take time first. The word ephemeral in {"type": "ephemeral"} is a literal promise: the entry is short-lived. But the clock works like a car's motion-sensor light, not an egg timer — every time someone hits the cache, the countdown resets to full. So a steady drip of traffic keeps the prefix warm forever, while a quiet gap longer than the window lets it lapse and the next caller pays to rebuild it. That's the mental model for choosing a TTL: match the window to how often you expect the prefix to be touched, and remember that a longer window buys survival across idle gaps at the cost of a steeper write premium.

Now change — and this is where the prefix-match rule from section 1 comes back to bite. Because the cache key is the exact bytes up to the breakpoint, invalidation is total and unforgiving: one altered byte anywhere before the marker doesn't degrade the cache, it destroys the entire cached prefix after that point. The analogy is a book index built for one specific edition — reflow the text by inserting a single word on page one and every page number in the index is wrong; the index isn't "mostly right," it's useless. The mistake that burns people is thinking of invalidators as "the prompt" only. They're anything that changes those bytes — and that includes things that don't feel like prompt content at all: a tool list rebuilt in a different order, a model swap, a json.dumps() whose keys came out in a different order this time, an iterated set. The list below names the usual suspects so you can hunt them by category rather than one painful surprise at a time.

A cache entry is ephemeral: by default it lives about 5 minutes, and the clock resets on every hit. So a steady stream of requests keeps the prefix warm indefinitely; a 6-minute gap lets it expire and the next call pays the write again. For bursty traffic with long gaps you can request a 1-hour TTL with {"type": "ephemeral", "ttl": "1h"} — but the write premium doubles to ~2×, so it needs more reads to pay off.

Invalidation follows straight from the prefix-match rule: any byte change anywhere before the breakpoint invalidates everything after it. That includes changes you might not think of as "the prompt":

What invalidates a cache entry

1. Editing the prefix text — a changed system prompt, a re-ordered document, a stray whitespace edit.
2. Changing the tool set — adding, removing, or re-ordering tools. Tools render at position 0, so this invalidates everything.
3. Switching models — caches are per-model; the same prefix on a different model is a cold write.
4. Non-deterministic serialization — json.dumps() without sorted keys, iterating a set, or an interpolated timestamp/UUID.
5. Letting it expire — no hit within the TTL window and the entry is gone.

7 · Multiple breakpoints & multi-turn caching professional

A single breakpoint is fine for a one-shot call, but a real agent holds a conversation — and that's where one marker isn't enough. The problem: in a chat, some of your prompt never changes (the system prompt, the tools) while another part grows a little with every turn (the conversation history). A single breakpoint forces you to pick one boundary, but you actually have two things worth caching that move at completely different speeds. This section is about caching both at once.

The mental model is geological strata, or tree rings. Your prompt has layers of different stability: a bedrock layer that's frozen for the life of the app (system + tools), and above it a layer that accretes one thin band per turn (each new user/assistant exchange). You want a cache marker at the top of each stable layer, so the bedrock is read once and reused forever while each new band gets written once and then read cheaply on every turn after. The API gives you up to 4 breakpoints per request precisely so you can mark several stability boundaries like this.

The move that makes multi-turn caching pay off is walking the breakpoint forward: each turn, you place a marker on the last block of the newest message. That freezes everything said so far as a cacheable prefix, so the next turn reads the entire prior conversation from cache and writes only its own new slice. Caching accrues incrementally — the read total climbs turn by turn while the fresh cost stays a thin sliver. The mistake that quietly kills this is rebuilding the stable layer slightly differently in a background job — a summarizer or sub-agent that reassembles system, tools, or model even a hair differently from the main loop misses the shared entry completely and pays cold writes. The rule: copy the parent's stable fields verbatim and append fork-specific content strictly after the breakpoint.

You get up to 4 breakpoints per request, which lets you cache at multiple stability boundaries. A common layout: one breakpoint after the tool definitions + frozen system prompt (never changes), and a second on the last block of the most recent conversation turn (grows as the chat continues). Each turn re-reads the entire prior prefix cheaply and only writes the newest slice. The block below shows both — real SDK, needs a key and network.

multi_breakpoint.py# needs: pip install anthropic + ANTHROPIC_API_KEY
# ▶ needs API key + network — real API call; will NOT run offline.
import anthropic

client = anthropic.Anthropic()

# Breakpoint #1: the frozen system prompt (+ any tools) — cached once, reused forever.
system = [{
    "type": "text",
    "text": open("company_handbook.txt").read(),
    "cache_control": {"type": "ephemeral"},
}]

# Breakpoint #2: the last block of the newest turn — the conversation prefix grows,
# so each turn reads everything before it cheaply and writes only the new slice.
messages = [
    {"role": "user", "content": "What's our refund window?"},
    {"role": "assistant", "content": "30 days from delivery."},
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "And for digital goods?",
                "cache_control": {"type": "ephemeral"},
            }
        ],
    },
]

resp = client.messages.create(
    model="claude-opus-4-8", max_tokens=256, system=system, messages=messages,
)
u = resp.usage
print(f"read={u.cache_read_input_tokens} created={u.cache_creation_input_tokens}")

A subtle production detail: caches are scoped by TTL, model, and prefix bytes. If a background job (summarization, a sub-agent) rebuilds system/tools/model even slightly differently from the main loop, it misses the main cache entirely. Copy the parent's stable fields verbatim and append fork-specific content after the breakpoint.

8 · Tech-lead — a caching strategy that gets hits tech-lead

The gap between "we use caching" and "caching actually saves us money" is where most teams quietly lose the win. An individual engineer places a cache_control marker, sees a read on their test call, and moves on. Then the codebase evolves: someone adds a per-user greeting to the system prompt, a refactor reorders the tool list, a new feature interpolates a request ID "just for logging." Each change is reasonable in isolation, and each silently zeroes the hit rate. Six months later the cache is decorative — every marker still there, every call a cold write. A lead's job is to make sure that decay never happens invisibly.

The reframe that separates a lead from a contributor is treating hit rate as a monitored metric, not a hopeful annotation. The analogy is a cache in any other system — a CDN, a database query cache: no serious team ships one and simply assumes it's working. They graph the hit rate, they alert when it drops, they treat a falling hit rate as an incident because it maps directly to a cost regression. Prompt caching deserves the same discipline. The single most useful signal is production cache_read_input_tokens: watch it in aggregate, and the moment someone lands a change that poisons the prefix, reads collapse toward zero and your dashboard tells you before the invoice does.

The mistake that defines this level is optimizing the marker placement while leaving the prompt assembly undisciplined. Caching is a property of how you build the request, so the design has to guarantee stability by construction: a frozen prefix (system + deterministically serialized, sorted tool definitions), all per-request content pushed strictly after the last breakpoint, a written breakpoint plan for which of the 4 slots caches what and why, and a TTL matched to traffic shape. Own those four things — the frozen contract, the breakpoint plan, the TTL choice, and the hit-rate alarm — and a large, repetitive prompt surface stays both fast and affordable as the system grows. Skip any one and the cache erodes the first time the code changes hands.

A lead owns the cache hit rate as a real, monitored metric — not a hopeful cache_control sprinkled on and forgotten. That means designing the prompt assembly so the stable content is genuinely stable, and instrumenting cache_read_input_tokens in production to catch regressions the moment someone interpolates a timestamp into the system prompt.

The contract to own: a frozen prefix (system + deterministic, sorted tool definitions) with dynamic context pushed after the last breakpoint; a breakpoint plan (which of the 4 slots caches what, and why); a TTL choice matched to traffic shape (5-minute for steady load, 1-hour for bursty); and a hit-rate dashboard that alerts when reads fall toward zero. Get those right and a large, repetitive prompt surface stays both fast and affordable at scale.

import anthropic
client = anthropic.Anthropic()

resp = client.messages.create(
    model="claude-opus-4-8", max_tokens=512,
    system=[{
        "type": "text",
        "text": LARGE_SHARED_CONTEXT,          # e.g. a 20k-token manual
        "cache_control": {"type": "ephemeral"},  # 5-minute TTL by default
    }],
    messages=[{"role": "user", "content": "Summarize section 3."}],
)
print(resp.usage.cache_creation_input_tokens)  # >0 on the FIRST call (cache write)
print(resp.usage.cache_read_input_tokens)      # >0 on LATER calls  (cache read)

IN = 5.0 / 1e6   # $ per token, Opus 4.8 input

def uncached(prefix, n):
    return prefix * IN * n                       # full price every time

def cached(prefix, n):
    write = prefix * IN * 1.25                   # first request: 1.25x
    reads = prefix * IN * 0.10 * (n - 1)         # rest: 0.1x
    return write + reads

p, n = 20_000, 10
print(f"uncached: ${uncached(p, n):.4f}")   # $1.0000
print(f"cached  : ${cached(p, n):.4f}")     # $0.2150
print(f"saved   : {100*(1-cached(p,n)/uncached(p,n)):.0f}%")  # 78%

# BROKEN — the timestamp changes every request, so the prefix never repeats:
system = [{"type": "text",
           "text": f"Today is {datetime.now()}. " + MANUAL,
           "cache_control": {"type": "ephemeral"}}]

# FIXED — freeze the prefix; move volatile content AFTER the last breakpoint:
system = [{"type": "text", "text": MANUAL,
           "cache_control": {"type": "ephemeral"}}]
messages = [{"role": "user",
             "content": f"Today is {datetime.now()}. Summarize section 3."}]

def step(messages):
    # mark the last block of the last message as the cache breakpoint
    last = messages[-1]
    if isinstance(last["content"], list):
        last["content"][-1]["cache_control"] = {"type": "ephemeral"}
    resp = client.messages.create(
        model="claude-opus-4-8", max_tokens=1024,
        system=[{"type": "text", "text": SYSTEM,
                 "cache_control": {"type": "ephemeral"}}],
        messages=messages,
    )
    print("read:", resp.usage.cache_read_input_tokens)  # grows each turn
    return resp

import json

def render_prefix(req):
    # tools -> system -> messages is the render order; hash the stable prefix
    return json.dumps(req.get("system"), sort_keys=True)

def audit(build_prompt):
    a = build_prompt(question="Q1")
    b = build_prompt(question="Q2")   # only the question should differ
    if render_prefix(a) != render_prefix(b):
        return "BROKEN: system prefix differs between requests -> no cache reads"
    return "OK: prefix stable; volatile content is after the breakpoint"

# Usual suspects it catches: datetime.now()/uuid in system, unsorted json.dumps,
# per-user string interpolation in the prefix, tools reordered/rebuilt per user.

def build(tenant_docs, question, tenant_id):
    return dict(
        model="claude-opus-4-8", max_tokens=1024,
        system=[
            # Tier 1: frozen, shared across ALL tenants -> global cache
            {"type": "text", "text": GLOBAL_SYSTEM,
             "cache_control": {"type": "ephemeral", "ttl": "1h"}},
            # Tier 2: per-tenant retrieved docs -> per-tenant cache, bursty traffic
            {"type": "text", "text": tenant_docs,
             "cache_control": {"type": "ephemeral", "ttl": "1h"}},
        ],
        # Tier 3: volatile per-request question -> NO breakpoint, always full price
        messages=[{"role": "user", "content": question}],
    )

Knowledge check check yourself