Scale & cost war-stories

How to triage a scale problem essential

Picture the moment this skill is for. It's 9:40 on a Tuesday, a dashboard is red, Slack has three threads, and someone from finance is asking why the API bill tripled overnight. Your instinct — everyone's instinct — is to open the codebase and start changing things. That instinct is the trap. Under pressure the fastest engineers do the opposite of what feels productive: they spend the first five minutes naming the problem before touching a single line, because the wrong fix applied confidently makes an outage longer, not shorter.

The reason a name helps so much is that LLM scaling failures aren't infinitely varied. Think of them like the four ways a car fails on the highway: it's guzzling fuel, it's crawling, the engine is cutting out, or it's steering wrong. You don't rebuild the engine when the tank is empty. Almost every production LLM incident sorts cleanly into one of four classes — cost (the bill is too high), latency (responses are too slow), reliability (the API is erroring under load), or quality (the answers got worse) — and each class has a tiny, well-known set of usual root causes and one canonical fix. Classification isn't bureaucracy; it's what turns an unbounded search over your whole system into a lookup in a table of four rows.

Here is the distinction that separates a patch from a fix, and it's the whole spine of this lesson. A symptom is what the dashboard shows ("cost is 3× up"). A root cause is the systemic gap that let it happen ("we re-send a 50k-token prefix uncached on every call") — and crucially, a systemic gap is never "someone should have known better." People forgetting is a constant of the universe; a resilient system assumes it. So the last and most important step in the tree below isn't the hot-fix — it's the platform default (a token ceiling, a shared client with backoff baked in, an eval that runs on every change) that makes the entire class of failure impossible next quarter. The common mistake is to stop one box too early: you add a cache to the one endpoint that broke, close the ticket, and the sequel ships from a different team three months later. As you read the tree, watch for that final column — it's where real engineering happens.

Every incident in this lesson starts the same way: something is on fire and you have minutes, not hours. The fastest way to the right fix is to classify the problem before you touch anything. Almost every LLM scaling failure is one of four shapes — the bill is too high, responses are too slow, the API is throwing errors under load, or the answers got worse — and each shape has a small set of usual root causes and a canonical fix. The triage tree below is the map; the six sections after it walk one representative story per branch (with cost and quality getting the extra depth they earn in practice).

Notice the last column of the tree: the real work is not the hot-fix, it's the platform default that makes the class impossible next quarter. A cache you added to one endpoint is a fix; a default token ceiling and a shared API client with backoff baked in are what stop the sequel. Each story below ends on that class-killing move.

1 · Cost blowout — a huge shared prefix, uncached essential

You have almost certainly written the code that causes this one. It's the most natural thing in the world: your feature needs the model to answer questions about a document — a policy manual, an API spec, a knowledge base — so you paste the document into the prompt as context and append the user's question. It works beautifully in the demo. Then it ships, usage grows, and the bill grows with it in a way nobody flagged, because the code never changed and no single request looks expensive. The blowout is invisible precisely because it's spread across thousands of individually-reasonable calls.

To feel why this is wasteful, use an analogy. Imagine a librarian who, every time you ask a question about a 300-page report, re-reads the entire report from page one before answering — even though it's the same report and you're the tenth person to ask today. That re-reading is the "input token" cost, and the model pays it in full on every single call. The document is the shared prefix: a large chunk of the prompt that is byte-for-byte identical across many requests, sitting in front of a small, unique tail (the question). The precise definition matters: it's not "big prompt" that's the problem, it's "big and repeated and stable." A prompt that's huge but different every time can't be cached; a prompt that's tiny but repeated isn't worth caching. The sweet spot — and it's an enormous one in practice — is a large stable block reused across calls.

The fix is prompt caching: you mark the stable prefix once, the provider stores its processed form, and subsequent calls read it back at a small fraction of the price instead of reprocessing it from scratch. Why is that allowed to be so much cheaper? Because the expensive work — turning those 50k tokens into the model's internal representation — was already done on the first call; the rest are just reusing it. The mistake this section is built to prevent isn't failing to cache so much as caching fragilely: because a cache hit requires the prefix to match exactly and sit at the very start of the prompt, dropping one volatile token into that block (a timestamp, a request id, the user's name) silently turns every call back into a full-price write — and because nothing errors, you won't notice for a week. Keep that fragility in mind as you read the money math below; the number it produces is real, but it evaporates the day someone interpolates a variable in the wrong place.

What finance saw: a document-QA feature answered every user question by stuffing the same ~50k-token policy manual into the prompt as context, then adding the user's short question. Traffic was flat, but as usage grew to ~1,000 calls a day the input-token bill dominated everything else. Nobody had changed the code; the cost simply scaled linearly with a prefix that never changed. (All figures illustrative.)

Root cause. The expensive part of each request — the 50k-token manual — was identical across every call and was being re-sent and re-processed from scratch each time. That is exactly the situation prompt caching exists for: mark the stable prefix with cache_control, and after the first call the model reads it from cache at a fraction of the price.

Fix. Add a cache breakpoint on the shared prefix. Anthropic bills a cache write at roughly 1.25× the base input price (5-minute TTL) and a cache read at about 0.1× — so a prefix reused across many calls is close to a tenth of its uncached cost. The lab does the money math for the exact scenario above (verify the multipliers in current docs):

cache_savings.pydef cache_savings(prefix_tokens, unique_tokens, calls,
                  in_price=3.0, write_mult=1.25, read_mult=0.10):
    """Cost of a big shared prefix WITHOUT vs WITH prompt caching.
    Prices are per-million input tokens; write_mult/read_mult are the
    published cache multipliers (5-min write ~1.25x, read ~0.1x).
    Verify current multipliers in Anthropic's docs."""
    M = 1_000_000
    per_uncached = (prefix_tokens + unique_tokens) * in_price / M
    no_cache = per_uncached * calls
    # cached: first call writes the prefix, the rest read it cheaply
    first = (prefix_tokens * write_mult + unique_tokens) * in_price / M
    rest = (prefix_tokens * read_mult + unique_tokens) * in_price / M * (calls - 1)
    with_cache = first + rest
    saved = no_cache - with_cache
    pct = saved / no_cache * 100
    return no_cache, with_cache, saved, pct

no_cache, with_cache, saved, pct = cache_savings(
    prefix_tokens=50_000, unique_tokens=500, calls=1_000)
print(f"no cache : ${no_cache:8.2f}")
print(f"cached   : ${with_cache:8.2f}")
print(f"saved    : ${saved:8.2f}  ({pct:.1f}%)")

no cache : $  151.50
cached   : $   16.67
saved    : $  134.83  (89.0%)

Lesson. When a large, stable chunk of the prompt is reused across requests, caching it is usually the single biggest cost lever you have — here it erases ~89% of spend. The class-killing move is to make caching the default for any endpoint with a shared system prompt or knowledge prefix, and to alarm on the cache-hit rate so a refactor that accidentally breaks the prefix (and the cache) shows up as a cost regression, not a silent one.

2 · Latency SLO missed under load intermediate

Here's the failure you'll defend against most often, because it hides so well. Everything is fast in staging. You test with a handful of requests, the p95 is comfortably under target, you ship. Then real traffic arrives — not more complexity, just more concurrency — and the tail latency balloons past your SLO while the median still looks fine. What makes this maddening is that no request got slower on its own; they started queueing behind each other, and the 95th-percentile user is the one waiting at the back of the line.

Two design choices almost always set the trap, and they're worth naming precisely because they feel like "keeping it simple" at the time. The first is one model for everything: you pick the most capable model so you never have to think about routing, and now a one-line "is this spam?" classification pays the same latency as a hard multi-step analysis. That's like sending every letter — a postcard and a legal contract alike — by armored courier. The second is no streaming: the client waits for the entire response to finish generating before showing the user anything, so even a fast answer feels slow because the screen sits blank behind a spinner. Note these are two different quantities — total generation time versus time-to-first-token — and confusing them is the common mistake. Streaming does nothing for total time; it transforms perceived latency, which is often the number users actually judge you on.

The fix has two independent levers. Model tiering routes each request to the cheapest model that can clear its complexity bar — a small fast model (Haiku) for the simple bulk, a mid model (Sonnet) for the middle, the heavy model (Opus) only for the genuinely hard tail — so 95% of traffic stops waiting behind the worst case's cost and latency. Streaming is orthogonal and nearly free: flip it on and time-to-first-token drops to near-instant regardless of which tier answered. The subtle thing the simulation below will show is that tiering helps the p95 specifically because real traffic is mostly-simple with a thin hard tail — when the tail is thin, moving it off the common path is exactly what rescues the percentile the SLO measures. If your traffic were uniformly hard, tiering would buy you far less, which is itself a useful thing to know before you build the router.

What users saw: a p95 latency SLO of "first token in under 1s, done in under 3s" held fine in staging and blew up under real load. Every request — a one-line classification, a short rewrite, a hard multi-step analysis — was routed to the largest model, and the client waited for the entire response before showing anything. Under a traffic peak the tail latency ballooned and the SLO went red. (Latency numbers illustrative.)

Root cause. Two mismatches. First, one heavy model for everything: most traffic was simple and did not need Opus-class reasoning, but paid Opus-class latency anyway. Second, no streaming: users stared at a spinner for the full generation instead of seeing tokens as they arrived.

Fix. Route by task complexity across the Haiku / Sonnet / Opus tiers — cheap-and-fast Haiku for the simple bulk, Sonnet for the middle, Opus only for the genuinely hard tail — and turn on streaming so time-to-first-token drops to near-instant regardless of tier. The offline simulation compares everything-on-Opus against tiered routing on a realistic traffic mix (mostly simple with a thin hard tail):

tiering_sim.pyTIERS = {"haiku": 0.25, "sonnet": 3.0, "opus": 15.0}  # illustrative $/Mtok output

def route(task_complexity):
    """Cheapest tier that clears the task's complexity bar (0..1)."""
    if task_complexity < 0.34:
        return "haiku"
    if task_complexity < 0.75:
        return "sonnet"
    return "opus"

def simulate(tasks, out_tokens=800):
    """Compare 'everything on Opus' vs tiered routing + a p95 latency proxy."""
    M = 1_000_000
    all_opus = flat = 0.0
    lat_flat, lat_tier = [], []
    base = {"haiku": 300, "sonnet": 700, "opus": 1500}  # ms, illustrative
    for c in tasks:
        all_opus += out_tokens * TIERS["opus"] / M
        tier = route(c)
        flat += out_tokens * TIERS[tier] / M
        lat_flat.append(base["opus"])
        lat_tier.append(base[tier])
    p95 = lambda xs: sorted(xs)[max(0, int(len(xs) * 0.95) - 1)]
    return all_opus, flat, p95(lat_flat), p95(lat_tier)

# mostly-simple traffic with a thin tail of hard tasks (typical of real workloads)
tasks = ([0.1, 0.2, 0.3] * 40 + [0.5, 0.6] * 45 + [0.9] * 8 + [0.95] * 2)
all_opus, tiered, p95_flat, p95_tier = simulate(tasks)
print(f"all-opus cost : ${all_opus:7.2f}   p95 latency {p95_flat} ms")
print(f"tiered   cost : ${tiered:7.2f}   p95 latency {p95_tier} ms")
print(f"cost cut      : {(1 - tiered / all_opus) * 100:.0f}%")

all-opus cost : $   2.64   p95 latency 1500 ms
tiered   cost : $   0.36   p95 latency 700 ms
cost cut      : 86%

Lesson. Tiering cuts both the bill (here ~86%) and the p95 latency (1500ms → 700ms), because the common case stops paying for the worst case. Streaming is orthogonal and free: it fixes perceived latency (time-to-first-token) even when total generation time is unchanged. The class-killing move is a small routing layer with an explicit complexity signal, plus streaming on by default — so the SLO is defended by architecture, not by hoping traffic stays light. Genuinely async work (no user waiting) should leave the realtime path entirely — that is story 4.

3 · Rate-limit thundering-herd outage advanced

This is the one where you cause your own outage — twice — with code that looks defensive. A traffic spike pushes you past your rate limit, the API returns 429 Too Many Requests, and your client does the "obvious" thing: it retries the failed request. Immediately. On every failure. What feels like resilience is actually pouring fuel on the fire, and the sickening part is that the second dip is usually caused by the fix a well-meaning engineer applies to the first ("let's just bump the retry count").

The mechanism has a name — a thundering herd — and an analogy makes it visceral. Picture a doorway that's briefly blocked. A calm crowd waits, then filters through. But if everyone shoves forward the instant they're refused, and shoves again the instant they're refused again, the doorway stays jammed by the very people trying to get through it. That's positive feedback: more failures trigger more retries, which add more load, which cause more failures. A rate limit is supposed to push back when you exceed it; the outage isn't the limit malfunctioning, it's your retry loop amplifying the limit's correct signal into a self-inflicted denial of service. Naming it as feedback (not "the API is flaky") is what points you at the real fix.

Retrying isn't wrong — retrying rudely is. The fix is to retry politely, and it stacks three ideas. Exponential backoff waits longer after each failure (0.5s, 1s, 2s, 4s…) so a struggling dependency gets breathing room instead of a fresh wave. Jitter — a random offset added to each delay — is the small detail that does the heavy lifting: without it, a thousand clients that failed at the same instant all back off by the same amount and retry in perfect synchrony, recreating the herd one step later; jitter smears them across the window so the load flattens. And a circuit breaker is the meta-move: after N consecutive failures it "trips," fast-failing every request for a cool-down period so the client stops hammering a service that's already down and lets it recover. The mistake to avoid is treating these as optional polish — under stress a client's job is to shed load, not add it, and a naive retry loop does the exact opposite. The simulation below makes the difference concrete and deterministic.

What users saw: the assistant went from slow to fully down for minutes, twice, in a self-inflicted cycle. A traffic spike pushed request volume past the account's rate limit, the API began returning 429 Too Many Requests, and the client retried every failure immediately. The retries stacked onto new requests — a thundering herd — so the system generated more load exactly when it needed to generate less, and the provider throttled harder.

Root cause. The client had no load-shedding behavior under failure. Retries were unconditional and instantaneous, making the failure mode positive feedback: more failures → more retries → more load → more failures. The rate limit did precisely what a rate limit is for; the outage was amplified by our own retry loop. The naive fix — "just retry the 429s" — is what caused the second dip.

Fix. Retry politely. Anthropic's guidance is exponential backoff on 429/5xx, and adding jitter spreads retries out so they don't synchronize into a fresh herd. Pair it with a circuit breaker that trips after N consecutive failures and fast-fails during a cool-down (so the client stops hammering a dependency that is already down and lets it recover), and a queue in front of the API to smooth spikes into a steady drain. The lab simulates the difference deterministically:

backoff_sim.pyimport random

def backoff_delays(attempts, base=0.5, cap=30.0, jitter=True, seed=0):
    """Exponential backoff with FULL JITTER, as recommended for 429/5xx:
    delay = random(0, min(cap, base * 2**attempt)). Deterministic here via seed."""
    rng = random.Random(seed)
    delays = []
    for a in range(attempts):
        window = min(cap, base * (2 ** a))
        delays.append(round(rng.uniform(0, window), 3) if jitter else round(window, 3))
    return delays

class CircuitBreaker:
    """Trip after `threshold` consecutive failures; fast-fail during cool-down
    so the client stops hammering a dependency that is already down."""
    def __init__(self, threshold=3):
        self.threshold = threshold
        self.fails = 0
        self.open = False
    def record(self, ok):
        if ok:
            self.fails = 0
            self.open = False
        else:
            self.fails += 1
            if self.fails >= self.threshold:
                self.open = True
        return self.open

print("no-backoff retries (thundering herd): all fire at t=0 ->", [0.0] * 5)
print("full-jitter delays (spread out)     :", backoff_delays(5))

cb = CircuitBreaker(threshold=3)
seq = [False, False, False, False, True]
states = ["OPEN" if cb.record(ok) else "closed" for ok in seq]
print("breaker states over", seq, "->", states)

no-backoff retries (thundering herd): all fire at t=0 -> [0.0, 0.0, 0.0, 0.0, 0.0]
full-jitter delays (spread out)     : [0.422, 0.758, 0.841, 1.036, 4.09]
breaker states over [False, False, False, False, True] -> ['closed', 'closed', 'OPEN', 'OPEN', 'closed']

Lesson. Under stress a client must shed load, not add it. Immediate retries fire at t=0 together; full-jitter delays fan out across the window so the dependency can recover, and the breaker cuts the feedback loop entirely once failures cluster. Also respect the retry-after header when the API sends one. The class-killing move: make backoff+jitter and a breaker the default in the shared API client, so no service can ship a naive retry loop again (verify your account's current rate limits in Anthropic's docs).

4 · Bulk nightly job too slow and too expensive advanced

Some work is answered "right now" and some is answered "by morning" — and shipping the second kind through the first kind's door is one of the most common and most fixable money leaks in production. You have a nightly job: re-classify a few hundred thousand documents, backfill embeddings, re-run evals. Someone wires it up the way they already know how — loop over the items, fire each one at the realtime API, collect the answers. It works. It's also slow, expensive, and quietly sabotaging your daytime traffic, and none of that is obvious until you look.

The insight is a mismatch of shapes. Think of shipping options: overnight courier versus a standing freight contract. When a customer is waiting on the other end of a chat, you're paying for a courier — speed at a premium, and worth it. But a nightly batch has slack on latency: no human is staring at a spinner, and "done within 24 hours" is a perfectly good deadline. Forcing that job through the realtime path pays courier prices for freight you had all night to move. Worse, it competes with your interactive users for the same rate-limit headroom, so your throughput job actively degrades the latency-critical path it's running alongside — a throughput-shaped workload jammed into a latency-shaped channel.

The fix is the Message Batches API, which exists precisely for this shape: submit all the items at once, let them process asynchronously within a window (up to 24 hours), and collect the results matched back by an id you assign. Because the provider can schedule this work on its own terms across separate capacity, it's priced at roughly half the standard per-token cost — so you're paid to give up latency you never needed. Just as importantly, it runs off your interactive rate-limit budget, so the nightly job stops stealing headroom from daytime chat. The mistake to avoid is the mirror image: don't batch a user-facing call to save money — you'll trade a 50% discount for a response that arrives hours late to someone waiting now. The rule that kills the whole class is a design-review reflex: if nobody is waiting on the response this second, it probably doesn't belong on the realtime path. The lab does the money math.

What the platform team saw: a nightly job re-classified and summarized ~200k documents by firing them one at a time through the realtime API. It competed with daytime interactive traffic for rate-limit headroom, took hours, and paid full realtime token prices for work that no user was waiting on. (Volume and cost illustrative.)

Root cause. A throughput-shaped job was forced through a latency-shaped channel. The realtime API is optimized for "answer now"; a bulk offline pass wants "answer all of these within a window, as cheaply as possible." That is exactly what the Message Batches API is for.

Fix. Move the job to Message Batches: submit all items as one batch, let it process asynchronously within its window (up to 24 hours), and retrieve results matched by custom_id. Anthropic prices batch token usage at about 50% of the standard per-token cost, and it runs on separate capacity so it stops fighting interactive traffic for rate limits. The lab does the cost comparison (verify the discount and window in current docs):

batch_cost.pydef nightly_job_cost(n_items, in_tok, out_tok,
                     in_price=3.0, out_price=15.0, batch_discount=0.50):
    """Realtime vs Message Batches for a bulk nightly job.
    Batches run within a 24h window at ~50% of standard token cost.
    Verify the current discount/window in Anthropic's docs."""
    M = 1_000_000
    per_item = (in_tok * in_price + out_tok * out_price) / M
    realtime = per_item * n_items
    batched = realtime * (1 - batch_discount)
    return realtime, batched

realtime, batched = nightly_job_cost(n_items=200_000, in_tok=1_200, out_tok=600)
print(f"realtime : ${realtime:9.2f}")
print(f"batched  : ${batched:9.2f}")
print(f"saved    : ${realtime - batched:9.2f}  (50% off)")

realtime : $  2520.00
batched  : $  1260.00
saved    : $  1260.00  (50% off)

Lesson. If nobody is waiting on the response right now, it probably does not belong on the realtime path. Batching halves the token bill here and removes the job from your interactive rate-limit budget entirely. The class-killing move is a rule of thumb baked into design review: any bulk, deadline-not-latency job goes through Batches — nightly re-indexing, backfills, evals, bulk classification. Combine it with story 1 (cache the shared prefix inside the batch) and the savings stack.

5 · Silent quality regression after a model change professional

The scariest incidents are the ones with no alarm. A crash pages you; a 500 shows up on a dashboard; a cost spike hits a budget alert. But this failure is a slow leak with no siren: the bot still answers, still sounds fluent, still returns something — and yet, over a week, a downstream parser starts dropping records and the "wrong/mis-formatted answer" tickets tick upward. Nobody deployed anything on your side. There was no error, no page, no obvious moment to point at. That silence is exactly what makes it a professional-tier story: catching it requires machinery you have to build before you need it.

The root of it is treating the model as if it were a constant when it's actually a dependency. Contrast two ways to reference a model. A mutable alias — asking for "the latest" version — is like pinning your build to latest on a package: convenient until the thing underneath changes without a version bump and your code inherits behavior you never reviewed. A pinned version is the opposite: you name the exact model, and any upgrade becomes a deliberate change you opt into. The trap with the alias isn't that models get worse — they usually get better on average — it's that "better on average" can still mean "different in the specific shape your strict consumer depended on." A newer model that's more conversational might wrap its JSON in a sentence of prose, and a parser expecting bare JSON silently discards it. The change is real; your ability to see it is what's missing.

So the fix is two parts, and the ordering of blame matters. First, pin the model, so an upgrade is a reviewed pull request rather than a surprise that arrives while you sleep. Second — and this is the part that actually defends you — build a small eval gate: a set of golden inputs paired with the output shape or quality you require, run automatically on every model or prompt change, blocking rollout if the pass rate drops below a threshold. Resist the tempting conclusion "someone should have pinned it." That's the symptom talking. The systemic cause is a release process that let a core dependency change with no review and no automated quality check — and "someone should remember" is not a control. The lab below is a miniature of exactly this gate: it diffs a pinned model against a candidate and refuses the rollout when quality regresses, turning a week-long silent bleed into a red build in CI.

What users saw: nothing dramatic — the bot still answered fluently — but over a week, a downstream parser that expected strict JSON began silently dropping records, and "the bot gave a wrong/mis-formatted answer" tickets climbed. The app referenced its model by a mutable alias rather than a pinned version, and when the underlying model rolled, its output shape drifted just enough to break the strict consumer. No deploy on our side, no error, no page — a slow bleed.

Root cause. The application depended on a mutable model reference and had no eval gate to detect a quality change. "Someone should have pinned it" is the symptom, not the cause; the real cause is a release process that let a core dependency change with no review and no automated quality check.

Fix. Pin the exact model version so any upgrade is a deliberate, reviewed change — a PR, not a surprise — and build a small eval suite of golden inputs with an expected output shape/quality, run on every model change and gating rollout on it. A regression is then caught in CI, not by users a week later. The lab is a miniature eval gate: it diffs a pinned model against a candidate and blocks the rollout when the pass rate drops below threshold:

eval_gate.pyGOLDEN = [
    ("refund order 88",   "json"),
    ("cancel my plan",    "json"),
    ("what's my balance", "json"),
    ("update address",    "json"),
]

def run_model(prompt, model):
    """Stand-in for a real API call: returns the output SHAPE the model produced.
    The 'candidate' model regresses to prose on 2 of 4 goldens."""
    if model == "pinned":
        return "json"
    return "prose" if prompt in ("cancel my plan", "update address") else "json"

def eval_gate(model, threshold=0.95):
    """Fraction of goldens whose shape still matches. Block rollout below threshold."""
    passed = sum(run_model(p, model) == want for p, want in GOLDEN)
    score = passed / len(GOLDEN)
    verdict = "PASS -> rollout" if score >= threshold else "FAIL -> block rollout"
    return score, verdict

for model in ("pinned", "candidate"):
    score, verdict = eval_gate(model)
    print(f"{model:10} score={score:.2f}  {verdict}")

pinned     score=1.00  PASS -> rollout
candidate  score=0.50  FAIL -> block rollout

Lesson. Treat the model like any other versioned dependency: pinned, changelogged, and eval-gated. The candidate above passes only 50% of the goldens and is blocked automatically — the difference between a caught regression and a week-long silent one. The class-killing move is making "pin + eval-on-change" a platform rule, so no service can consume a mutable model reference. See CH6 and the evals material for building the golden set.

6 · Context-window blowup from unbounded history tech-lead

This is the failure that punishes your best users, which is why it's the tech-lead story. A chat assistant, built the natural way, appends the whole conversation to every request so the model "remembers" what was said. Short sessions are flawless. But the users who love your product — the ones who chat the longest — are the ones whose transcript quietly grows until a request hits the context limit and throws, and every call before that gets steadily slower and more expensive as the history bloats. A bug that scales with engagement, striking your most engaged users first, is about the cruelest place for a defect to live.

The naive framing is "we ran out of context," but the real problem is a missing decision. Sending the entire transcript every turn means per-request cost and latency grow linearly with session length, and eventually the assembled prompt simply exceeds the window. The systemic gap is that there was no token budget on the assembled prompt and no policy for what to do when the history exceeds it. This is where the tech-lead judgment comes in, because the obvious fixes each have a sharp edge. Just truncate to the last N turns? You'll amputate the crucial fact the user gave you in turn two. Summarize everything? You lose the exact wording that recent turns depend on and you pay to summarize on every request. The skill isn't picking one lever; it's composing them so their weaknesses cancel.

The durable answer is to bound the assembled context to a token budget with a keep-newest-plus-summarize policy: keep the most recent turns verbatim, because recency carries the most conversational weight, and fold the older overflow into a single rolling summary so nothing important is silently dropped. Then cache that stable summarized head (story 1 again — the levers compose) so the retained context is not just small but nearly free to re-send each turn. The mental model to carry away: you are packing a fixed-size suitcase, not filling an infinite one — the newest items go in whole, the rest get compressed to fit, and the total never exceeds what the window allows. The mistake this prevents is the one that feels safe today and pages you at 2am when your happiest customer's session finally tips over the limit. The lab implements the packing policy against a hard budget so you can watch a 39k-token transcript fold down to something that fits.

What users saw: a long-running chat assistant appended the entire conversation to every request. Short sessions were fine; long ones grew until requests hit the context limit and threw — and every call before that got steadily slower and more expensive as the history bloated. The failure was correlated with your best users (the ones who chat the longest), which is the cruelest place for a bug to live. This is the tech-lead story because the naive fixes each have a sharp edge.

Root cause. Unbounded history with no packing strategy. Sending the whole transcript every turn makes per-request cost and latency grow with session length and eventually exceeds the window entirely. The systemic gap: there was no token budget on the assembled prompt and no policy for what to do when the history exceeds it.

Fix. Bound the assembled context to a token budget: keep the newest turns verbatim (recency matters most), and replace the older overflow with a single rolling summary so nothing important is simply forgotten. Then cache the stable head of the prompt (story 1) so the retained context is cheap to re-send. The offline lab implements the packing policy against a hard budget:

pack_history.pydef pack_history(turns, budget, summary_cost=120):
    """Bound an unbounded chat history to a token BUDGET.
    Keep newest turns verbatim; replace the older overflow with one summary.
    turns: list of (role, tokens), oldest first. Returns
    (kept, n_summarized, total_tokens)."""
    kept, running = [], 0
    for role, tok in reversed(turns):          # newest first
        if running + tok <= budget:
            kept.append((role, tok))
            running += tok
        else:
            break
    kept.reverse()
    dropped = turns[:len(turns) - len(kept)]
    if dropped:
        running += summary_cost                # one rolling summary replaces them
    return kept, len(dropped), running

turns = [("user", 400), ("assistant", 900)] * 30   # 60 turns, ~39k tokens
raw = sum(t for _, t in turns)
kept, summarized, total = pack_history(turns, budget=4_000)
print(f"raw history      : {raw:,} tokens over {len(turns)} turns")
print(f"kept verbatim    : {len(kept)} turns")
print(f"summarized away  : {summarized} older turns -> 1 summary")
print(f"packed total     : {total:,} tokens (fits budget 4,000)")

raw history      : 39,000 tokens over 60 turns
kept verbatim    : 6 turns
summarized away  : 54 older turns -> 1 summary
packed total     : 4,020 tokens (fits budget 4,000)

Lesson. Never send an unbounded prompt. A budget with a keep-newest + summarize-the-rest policy turns a 39k-token transcript into ~4k tokens that fits, stays cheap, and keeps the thread coherent. Which levers combine: trim to a budget, summarize the overflow so context isn't lost, and cache the stable prefix so the retained history is nearly free to re-send. The class-killing move is a shared context-assembly component with a mandatory budget — so no feature can ship an unbounded prompt, and the context-limit error becomes structurally impossible.