Prompt caching, hands-on

1 · What prompt caching is essential

Every time you call the model, the whole prompt — your system prompt, tool definitions, and the conversation so far — is re-read and re-processed from scratch. If a big chunk of that prompt is identical from one call to the next (a long system prompt, a big document you keep asking about, a fixed list of tool definitions), you are paying to process the same tokens over and over.

Prompt caching lets the API remember the computed state of a stable prefix and reuse it on the next call. The first call writes the cache; later calls with the same prefix read it — skipping the re-computation. You mark where the stable part ends with a cache breakpoint (cache_control), and the API caches everything up to and including that point.

The rule that governs everything: caching is a prefix match. The cache key is the exact bytes up to the breakpoint. On the first call that prefix is computed and stored (a cache write). On later calls with a byte-identical prefix, the stored state is reused (a cache read) and only the variable suffix after the breakpoint is processed fresh.

2 · Why it saves cost and latency essential

Reusing the computed prefix is cheaper on both axes. On cost: a cache read is billed at roughly 0.1× the normal input price for those tokens, versus the full 1× you'd otherwise pay every call. On latency: the model skips the prefill work for the cached tokens, so time-to-first-token drops — often dramatically when the prefix is large (tens of thousands of tokens of system prompt or document).

The catch is the write: the first call pays a small premium — about 1.25× for the default 5-minute cache (or 2× for the 1-hour TTL) — to store the prefix. So caching pays off once you reuse the prefix enough to amortize that write. With the 5-minute cache, that's just two calls.

3 · The request shape — cache_control on a content block essential

You cache by putting "cache_control": {"type": "ephemeral"} on the last content block of the stable prefix. Render order is tools → system → messages, so a breakpoint on the last system block caches the tools and the system prompt together. The block below is real Anthropic SDK — it needs a key and a network connection to run.

cache_system.py# needs: pip install anthropic + ANTHROPIC_API_KEY
# ▶ needs API key + network — this really calls the API; it will NOT run offline.
import anthropic

client = anthropic.Anthropic()

LARGE_SYSTEM = open("company_handbook.txt").read()   # e.g. tens of thousands of tokens

resp = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LARGE_SYSTEM,
            # Breakpoint at the END of the stable prefix. Everything up to and
            # including this block is cached; the user message after it is not.
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[{"role": "user", "content": "Summarize the PTO policy in 3 bullets."}],
)

print(resp.content[0].text)

4 · Reading the usage fields — proof of a hit intermediate

Never assume caching worked — measure it. Every response carries a usage object with three token fields that together tell the whole story:

On the first call you expect a big cache_creation_input_tokens and a zero cache_read_input_tokens. On the second identical-prefix call they flip: cache_creation drops to 0 and cache_read jumps up. The block below reads those fields — real SDK, needs a key and network.

read_usage.py# needs: pip install anthropic + ANTHROPIC_API_KEY
# ▶ needs API key + network — real API calls; will NOT run offline.
import anthropic

client = anthropic.Anthropic()

system = [{
    "type": "text",
    "text": open("company_handbook.txt").read(),
    "cache_control": {"type": "ephemeral"},
}]

def ask(question):
    resp = client.messages.create(
        model="claude-opus-4-8",
        max_tokens=256,
        system=system,                                   # byte-identical prefix each call
        messages=[{"role": "user", "content": question}],
    )
    u = resp.usage
    print(f"created={u.cache_creation_input_tokens} "
          f"read={u.cache_read_input_tokens} "
          f"fresh={u.input_tokens}")
    return resp

ask("Summarize the PTO policy.")     # 1st: created=<big>  read=0
ask("What about parental leave?")    # 2nd: created=0      read=<big>

5 · The cost math (runs offline) intermediate

You can decide whether caching is worth it with arithmetic, no API needed. Without caching, N calls each pay full price (1×) for the prefix. With caching, the first call pays the write premium (~1.25×) and the other N−1 calls pay the read rate (~0.1×). The helper below is pure stdlib — it runs with a plain python file.py — and reports the tokens saved as a percentage.

cache_savings.pydef cache_savings(prefix_tokens, calls, write_mult=1.25, read_mult=0.1):
    """Compare paying full price N times vs. write-once + read (N-1) times.
    Returns (no_cache_units, with_cache_units, percent_saved) in token-equivalents.
    """
    no_cache = prefix_tokens * calls
    with_cache = prefix_tokens * write_mult + prefix_tokens * read_mult * (calls - 1)
    saved_pct = round(100 * (no_cache - with_cache) / no_cache)
    return round(no_cache), round(with_cache), saved_pct

no_cache, with_cache, saved = cache_savings(prefix_tokens=10_000, calls=100)
print(f"no cache:   {no_cache} token-units")
print(f"with cache: {with_cache} token-units")
print(f"saved:      {saved}% on the cached prefix")

# Break-even: at how few calls does caching already win?
one, two = cache_savings(10_000, 1)[1], cache_savings(10_000, 2)[1]
print(f"1 call costs {one} (write premium, a loss); 2 calls cost {two} (already a win)")

no cache:   1000000 token-units
with cache: 111500 token-units
saved:      89% on the cached prefix
1 call costs 12500 (write premium, a loss); 2 calls cost 13500 (already a win)

Two lessons from the numbers. First, at 100 reuses of a 10K-token prefix you save ~89% of the prefix cost. Second, a single call costs 1.25× — caching a prefix you'll use only once is a small loss. It turns a profit at the second call and climbs from there.

6 · TTL and what invalidates the cache advanced

A cache entry is ephemeral: by default it lives about 5 minutes, and the clock resets on every hit. So a steady stream of requests keeps the prefix warm indefinitely; a 6-minute gap lets it expire and the next call pays the write again. For bursty traffic with long gaps you can request a 1-hour TTL with {"type": "ephemeral", "ttl": "1h"} — but the write premium doubles to ~2×, so it needs more reads to pay off.

Invalidation follows straight from the prefix-match rule: any byte change anywhere before the breakpoint invalidates everything after it. That includes changes you might not think of as "the prompt":

What invalidates a cache entry

1. Editing the prefix text — a changed system prompt, a re-ordered document, a stray whitespace edit.
2. Changing the tool set — adding, removing, or re-ordering tools. Tools render at position 0, so this invalidates everything.
3. Switching models — caches are per-model; the same prefix on a different model is a cold write.
4. Non-deterministic serialization — json.dumps() without sorted keys, iterating a set, or an interpolated timestamp/UUID.
5. Letting it expire — no hit within the TTL window and the entry is gone.

7 · Multiple breakpoints & multi-turn caching professional

You get up to 4 breakpoints per request, which lets you cache at multiple stability boundaries. A common layout: one breakpoint after the tool definitions + frozen system prompt (never changes), and a second on the last block of the most recent conversation turn (grows as the chat continues). Each turn re-reads the entire prior prefix cheaply and only writes the newest slice. The block below shows both — real SDK, needs a key and network.

multi_breakpoint.py# needs: pip install anthropic + ANTHROPIC_API_KEY
# ▶ needs API key + network — real API call; will NOT run offline.
import anthropic

client = anthropic.Anthropic()

# Breakpoint #1: the frozen system prompt (+ any tools) — cached once, reused forever.
system = [{
    "type": "text",
    "text": open("company_handbook.txt").read(),
    "cache_control": {"type": "ephemeral"},
}]

# Breakpoint #2: the last block of the newest turn — the conversation prefix grows,
# so each turn reads everything before it cheaply and writes only the new slice.
messages = [
    {"role": "user", "content": "What's our refund window?"},
    {"role": "assistant", "content": "30 days from delivery."},
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "And for digital goods?",
                "cache_control": {"type": "ephemeral"},
            }
        ],
    },
]

resp = client.messages.create(
    model="claude-opus-4-8", max_tokens=256, system=system, messages=messages,
)
u = resp.usage
print(f"read={u.cache_read_input_tokens} created={u.cache_creation_input_tokens}")

A subtle production detail: caches are scoped by TTL, model, and prefix bytes. If a background job (summarization, a sub-agent) rebuilds system/tools/model even slightly differently from the main loop, it misses the main cache entirely. Copy the parent's stable fields verbatim and append fork-specific content after the breakpoint.

8 · Tech-lead — a caching strategy that gets hits tech-lead

A lead owns the cache hit rate as a real, monitored metric — not a hopeful cache_control sprinkled on and forgotten. That means designing the prompt assembly so the stable content is genuinely stable, and instrumenting cache_read_input_tokens in production to catch regressions the moment someone interpolates a timestamp into the system prompt.

The contract to own: a frozen prefix (system + deterministic, sorted tool definitions) with dynamic context pushed after the last breakpoint; a breakpoint plan (which of the 4 slots caches what, and why); a TTL choice matched to traffic shape (5-minute for steady load, 1-hour for bursty); and a hit-rate dashboard that alerts when reads fall toward zero. Get those right and a large, repetitive prompt surface stays both fast and affordable at scale.

import anthropic
client = anthropic.Anthropic()

resp = client.messages.create(
    model="claude-opus-4-8", max_tokens=512,
    system=[{
        "type": "text",
        "text": LARGE_SHARED_CONTEXT,          # e.g. a 20k-token manual
        "cache_control": {"type": "ephemeral"},  # 5-minute TTL by default
    }],
    messages=[{"role": "user", "content": "Summarize section 3."}],
)
print(resp.usage.cache_creation_input_tokens)  # >0 on the FIRST call (cache write)
print(resp.usage.cache_read_input_tokens)      # >0 on LATER calls  (cache read)

IN = 5.0 / 1e6   # $ per token, Opus 4.8 input

def uncached(prefix, n):
    return prefix * IN * n                       # full price every time

def cached(prefix, n):
    write = prefix * IN * 1.25                   # first request: 1.25x
    reads = prefix * IN * 0.10 * (n - 1)         # rest: 0.1x
    return write + reads

p, n = 20_000, 10
print(f"uncached: ${uncached(p, n):.4f}")   # $1.0000
print(f"cached  : ${cached(p, n):.4f}")     # $0.2150
print(f"saved   : {100*(1-cached(p,n)/uncached(p,n)):.0f}%")  # 78%

# BROKEN — the timestamp changes every request, so the prefix never repeats:
system = [{"type": "text",
           "text": f"Today is {datetime.now()}. " + MANUAL,
           "cache_control": {"type": "ephemeral"}}]

# FIXED — freeze the prefix; move volatile content AFTER the last breakpoint:
system = [{"type": "text", "text": MANUAL,
           "cache_control": {"type": "ephemeral"}}]
messages = [{"role": "user",
             "content": f"Today is {datetime.now()}. Summarize section 3."}]

def step(messages):
    # mark the last block of the last message as the cache breakpoint
    last = messages[-1]
    if isinstance(last["content"], list):
        last["content"][-1]["cache_control"] = {"type": "ephemeral"}
    resp = client.messages.create(
        model="claude-opus-4-8", max_tokens=1024,
        system=[{"type": "text", "text": SYSTEM,
                 "cache_control": {"type": "ephemeral"}}],
        messages=messages,
    )
    print("read:", resp.usage.cache_read_input_tokens)  # grows each turn
    return resp

import json

def render_prefix(req):
    # tools -> system -> messages is the render order; hash the stable prefix
    return json.dumps(req.get("system"), sort_keys=True)

def audit(build_prompt):
    a = build_prompt(question="Q1")
    b = build_prompt(question="Q2")   # only the question should differ
    if render_prefix(a) != render_prefix(b):
        return "BROKEN: system prefix differs between requests -> no cache reads"
    return "OK: prefix stable; volatile content is after the breakpoint"

# Usual suspects it catches: datetime.now()/uuid in system, unsorted json.dumps,
# per-user string interpolation in the prefix, tools reordered/rebuilt per user.

def build(tenant_docs, question, tenant_id):
    return dict(
        model="claude-opus-4-8", max_tokens=1024,
        system=[
            # Tier 1: frozen, shared across ALL tenants -> global cache
            {"type": "text", "text": GLOBAL_SYSTEM,
             "cache_control": {"type": "ephemeral", "ttl": "1h"}},
            # Tier 2: per-tenant retrieved docs -> per-tenant cache, bursty traffic
            {"type": "text", "text": tenant_docs,
             "cache_control": {"type": "ephemeral", "ttl": "1h"}},
        ],
        # Tier 3: volatile per-request question -> NO breakpoint, always full price
        messages=[{"role": "user", "content": question}],
    )

Knowledge check check yourself