Scale & cost war-stories

How to triage a scale problem essential

Every incident in this lesson starts the same way: something is on fire and you have minutes, not hours. The fastest way to the right fix is to classify the problem before you touch anything. Almost every LLM scaling failure is one of four shapes — the bill is too high, responses are too slow, the API is throwing errors under load, or the answers got worse — and each shape has a small set of usual root causes and a canonical fix. The triage tree below is the map; the six sections after it walk one representative story per branch (with cost and quality getting the extra depth they earn in practice).

Notice the last column of the tree: the real work is not the hot-fix, it's the platform default that makes the class impossible next quarter. A cache you added to one endpoint is a fix; a default token ceiling and a shared API client with backoff baked in are what stop the sequel. Each story below ends on that class-killing move.

1 · Cost blowout — a huge shared prefix, uncached essential

What finance saw: a document-QA feature answered every user question by stuffing the same ~50k-token policy manual into the prompt as context, then adding the user's short question. Traffic was flat, but as usage grew to ~1,000 calls a day the input-token bill dominated everything else. Nobody had changed the code; the cost simply scaled linearly with a prefix that never changed. (All figures illustrative.)

Root cause. The expensive part of each request — the 50k-token manual — was identical across every call and was being re-sent and re-processed from scratch each time. That is exactly the situation prompt caching exists for: mark the stable prefix with cache_control, and after the first call the model reads it from cache at a fraction of the price.

Fix. Add a cache breakpoint on the shared prefix. Anthropic bills a cache write at roughly 1.25× the base input price (5-minute TTL) and a cache read at about 0.1× — so a prefix reused across many calls is close to a tenth of its uncached cost. The lab does the money math for the exact scenario above (verify the multipliers in current docs):

cache_savings.pydef cache_savings(prefix_tokens, unique_tokens, calls,
                  in_price=3.0, write_mult=1.25, read_mult=0.10):
    """Cost of a big shared prefix WITHOUT vs WITH prompt caching.
    Prices are per-million input tokens; write_mult/read_mult are the
    published cache multipliers (5-min write ~1.25x, read ~0.1x).
    Verify current multipliers in Anthropic's docs."""
    M = 1_000_000
    per_uncached = (prefix_tokens + unique_tokens) * in_price / M
    no_cache = per_uncached * calls
    # cached: first call writes the prefix, the rest read it cheaply
    first = (prefix_tokens * write_mult + unique_tokens) * in_price / M
    rest = (prefix_tokens * read_mult + unique_tokens) * in_price / M * (calls - 1)
    with_cache = first + rest
    saved = no_cache - with_cache
    pct = saved / no_cache * 100
    return no_cache, with_cache, saved, pct

no_cache, with_cache, saved, pct = cache_savings(
    prefix_tokens=50_000, unique_tokens=500, calls=1_000)
print(f"no cache : ${no_cache:8.2f}")
print(f"cached   : ${with_cache:8.2f}")
print(f"saved    : ${saved:8.2f}  ({pct:.1f}%)")

no cache : $  151.50
cached   : $   16.67
saved    : $  134.83  (89.0%)

Lesson. When a large, stable chunk of the prompt is reused across requests, caching it is usually the single biggest cost lever you have — here it erases ~89% of spend. The class-killing move is to make caching the default for any endpoint with a shared system prompt or knowledge prefix, and to alarm on the cache-hit rate so a refactor that accidentally breaks the prefix (and the cache) shows up as a cost regression, not a silent one.

2 · Latency SLO missed under load intermediate

What users saw: a p95 latency SLO of "first token in under 1s, done in under 3s" held fine in staging and blew up under real load. Every request — a one-line classification, a short rewrite, a hard multi-step analysis — was routed to the largest model, and the client waited for the entire response before showing anything. Under a traffic peak the tail latency ballooned and the SLO went red. (Latency numbers illustrative.)

Root cause. Two mismatches. First, one heavy model for everything: most traffic was simple and did not need Opus-class reasoning, but paid Opus-class latency anyway. Second, no streaming: users stared at a spinner for the full generation instead of seeing tokens as they arrived.

Fix. Route by task complexity across the Haiku / Sonnet / Opus tiers — cheap-and-fast Haiku for the simple bulk, Sonnet for the middle, Opus only for the genuinely hard tail — and turn on streaming so time-to-first-token drops to near-instant regardless of tier. The offline simulation compares everything-on-Opus against tiered routing on a realistic traffic mix (mostly simple with a thin hard tail):

tiering_sim.pyTIERS = {"haiku": 0.25, "sonnet": 3.0, "opus": 15.0}  # illustrative $/Mtok output

def route(task_complexity):
    """Cheapest tier that clears the task's complexity bar (0..1)."""
    if task_complexity < 0.34:
        return "haiku"
    if task_complexity < 0.75:
        return "sonnet"
    return "opus"

def simulate(tasks, out_tokens=800):
    """Compare 'everything on Opus' vs tiered routing + a p95 latency proxy."""
    M = 1_000_000
    all_opus = flat = 0.0
    lat_flat, lat_tier = [], []
    base = {"haiku": 300, "sonnet": 700, "opus": 1500}  # ms, illustrative
    for c in tasks:
        all_opus += out_tokens * TIERS["opus"] / M
        tier = route(c)
        flat += out_tokens * TIERS[tier] / M
        lat_flat.append(base["opus"])
        lat_tier.append(base[tier])
    p95 = lambda xs: sorted(xs)[max(0, int(len(xs) * 0.95) - 1)]
    return all_opus, flat, p95(lat_flat), p95(lat_tier)

# mostly-simple traffic with a thin tail of hard tasks (typical of real workloads)
tasks = ([0.1, 0.2, 0.3] * 40 + [0.5, 0.6] * 45 + [0.9] * 8 + [0.95] * 2)
all_opus, tiered, p95_flat, p95_tier = simulate(tasks)
print(f"all-opus cost : ${all_opus:7.2f}   p95 latency {p95_flat} ms")
print(f"tiered   cost : ${tiered:7.2f}   p95 latency {p95_tier} ms")
print(f"cost cut      : {(1 - tiered / all_opus) * 100:.0f}%")

all-opus cost : $   2.64   p95 latency 1500 ms
tiered   cost : $   0.36   p95 latency 700 ms
cost cut      : 86%

Lesson. Tiering cuts both the bill (here ~86%) and the p95 latency (1500ms → 700ms), because the common case stops paying for the worst case. Streaming is orthogonal and free: it fixes perceived latency (time-to-first-token) even when total generation time is unchanged. The class-killing move is a small routing layer with an explicit complexity signal, plus streaming on by default — so the SLO is defended by architecture, not by hoping traffic stays light. Genuinely async work (no user waiting) should leave the realtime path entirely — that is story 4.

3 · Rate-limit thundering-herd outage advanced

What users saw: the assistant went from slow to fully down for minutes, twice, in a self-inflicted cycle. A traffic spike pushed request volume past the account's rate limit, the API began returning 429 Too Many Requests, and the client retried every failure immediately. The retries stacked onto new requests — a thundering herd — so the system generated more load exactly when it needed to generate less, and the provider throttled harder.

Root cause. The client had no load-shedding behavior under failure. Retries were unconditional and instantaneous, making the failure mode positive feedback: more failures → more retries → more load → more failures. The rate limit did precisely what a rate limit is for; the outage was amplified by our own retry loop. The naive fix — "just retry the 429s" — is what caused the second dip.

Fix. Retry politely. Anthropic's guidance is exponential backoff on 429/5xx, and adding jitter spreads retries out so they don't synchronize into a fresh herd. Pair it with a circuit breaker that trips after N consecutive failures and fast-fails during a cool-down (so the client stops hammering a dependency that is already down and lets it recover), and a queue in front of the API to smooth spikes into a steady drain. The lab simulates the difference deterministically:

backoff_sim.pyimport random

def backoff_delays(attempts, base=0.5, cap=30.0, jitter=True, seed=0):
    """Exponential backoff with FULL JITTER, as recommended for 429/5xx:
    delay = random(0, min(cap, base * 2**attempt)). Deterministic here via seed."""
    rng = random.Random(seed)
    delays = []
    for a in range(attempts):
        window = min(cap, base * (2 ** a))
        delays.append(round(rng.uniform(0, window), 3) if jitter else round(window, 3))
    return delays

class CircuitBreaker:
    """Trip after `threshold` consecutive failures; fast-fail during cool-down
    so the client stops hammering a dependency that is already down."""
    def __init__(self, threshold=3):
        self.threshold = threshold
        self.fails = 0
        self.open = False
    def record(self, ok):
        if ok:
            self.fails = 0
            self.open = False
        else:
            self.fails += 1
            if self.fails >= self.threshold:
                self.open = True
        return self.open

print("no-backoff retries (thundering herd): all fire at t=0 ->", [0.0] * 5)
print("full-jitter delays (spread out)     :", backoff_delays(5))

cb = CircuitBreaker(threshold=3)
seq = [False, False, False, False, True]
states = ["OPEN" if cb.record(ok) else "closed" for ok in seq]
print("breaker states over", seq, "->", states)

no-backoff retries (thundering herd): all fire at t=0 -> [0.0, 0.0, 0.0, 0.0, 0.0]
full-jitter delays (spread out)     : [0.422, 0.758, 0.841, 1.036, 4.09]
breaker states over [False, False, False, False, True] -> ['closed', 'closed', 'OPEN', 'OPEN', 'closed']

Lesson. Under stress a client must shed load, not add it. Immediate retries fire at t=0 together; full-jitter delays fan out across the window so the dependency can recover, and the breaker cuts the feedback loop entirely once failures cluster. Also respect the retry-after header when the API sends one. The class-killing move: make backoff+jitter and a breaker the default in the shared API client, so no service can ship a naive retry loop again (verify your account's current rate limits in Anthropic's docs).

4 · Bulk nightly job too slow and too expensive advanced

What the platform team saw: a nightly job re-classified and summarized ~200k documents by firing them one at a time through the realtime API. It competed with daytime interactive traffic for rate-limit headroom, took hours, and paid full realtime token prices for work that no user was waiting on. (Volume and cost illustrative.)

Root cause. A throughput-shaped job was forced through a latency-shaped channel. The realtime API is optimized for "answer now"; a bulk offline pass wants "answer all of these within a window, as cheaply as possible." That is exactly what the Message Batches API is for.

Fix. Move the job to Message Batches: submit all items as one batch, let it process asynchronously within its window (up to 24 hours), and retrieve results matched by custom_id. Anthropic prices batch token usage at about 50% of the standard per-token cost, and it runs on separate capacity so it stops fighting interactive traffic for rate limits. The lab does the cost comparison (verify the discount and window in current docs):

batch_cost.pydef nightly_job_cost(n_items, in_tok, out_tok,
                     in_price=3.0, out_price=15.0, batch_discount=0.50):
    """Realtime vs Message Batches for a bulk nightly job.
    Batches run within a 24h window at ~50% of standard token cost.
    Verify the current discount/window in Anthropic's docs."""
    M = 1_000_000
    per_item = (in_tok * in_price + out_tok * out_price) / M
    realtime = per_item * n_items
    batched = realtime * (1 - batch_discount)
    return realtime, batched

realtime, batched = nightly_job_cost(n_items=200_000, in_tok=1_200, out_tok=600)
print(f"realtime : ${realtime:9.2f}")
print(f"batched  : ${batched:9.2f}")
print(f"saved    : ${realtime - batched:9.2f}  (50% off)")

realtime : $  2520.00
batched  : $  1260.00
saved    : $  1260.00  (50% off)

Lesson. If nobody is waiting on the response right now, it probably does not belong on the realtime path. Batching halves the token bill here and removes the job from your interactive rate-limit budget entirely. The class-killing move is a rule of thumb baked into design review: any bulk, deadline-not-latency job goes through Batches — nightly re-indexing, backfills, evals, bulk classification. Combine it with story 1 (cache the shared prefix inside the batch) and the savings stack.

5 · Silent quality regression after a model change professional

What users saw: nothing dramatic — the bot still answered fluently — but over a week, a downstream parser that expected strict JSON began silently dropping records, and "the bot gave a wrong/mis-formatted answer" tickets climbed. The app referenced its model by a mutable alias rather than a pinned version, and when the underlying model rolled, its output shape drifted just enough to break the strict consumer. No deploy on our side, no error, no page — a slow bleed.

Root cause. The application depended on a mutable model reference and had no eval gate to detect a quality change. "Someone should have pinned it" is the symptom, not the cause; the real cause is a release process that let a core dependency change with no review and no automated quality check.

Fix. Pin the exact model version so any upgrade is a deliberate, reviewed change — a PR, not a surprise — and build a small eval suite of golden inputs with an expected output shape/quality, run on every model change and gating rollout on it. A regression is then caught in CI, not by users a week later. The lab is a miniature eval gate: it diffs a pinned model against a candidate and blocks the rollout when the pass rate drops below threshold:

eval_gate.pyGOLDEN = [
    ("refund order 88",   "json"),
    ("cancel my plan",    "json"),
    ("what's my balance", "json"),
    ("update address",    "json"),
]

def run_model(prompt, model):
    """Stand-in for a real API call: returns the output SHAPE the model produced.
    The 'candidate' model regresses to prose on 2 of 4 goldens."""
    if model == "pinned":
        return "json"
    return "prose" if prompt in ("cancel my plan", "update address") else "json"

def eval_gate(model, threshold=0.95):
    """Fraction of goldens whose shape still matches. Block rollout below threshold."""
    passed = sum(run_model(p, model) == want for p, want in GOLDEN)
    score = passed / len(GOLDEN)
    verdict = "PASS -> rollout" if score >= threshold else "FAIL -> block rollout"
    return score, verdict

for model in ("pinned", "candidate"):
    score, verdict = eval_gate(model)
    print(f"{model:10} score={score:.2f}  {verdict}")

pinned     score=1.00  PASS -> rollout
candidate  score=0.50  FAIL -> block rollout

Lesson. Treat the model like any other versioned dependency: pinned, changelogged, and eval-gated. The candidate above passes only 50% of the goldens and is blocked automatically — the difference between a caught regression and a week-long silent one. The class-killing move is making "pin + eval-on-change" a platform rule, so no service can consume a mutable model reference. See CH6 and the evals material for building the golden set.

6 · Context-window blowup from unbounded history tech-lead

What users saw: a long-running chat assistant appended the entire conversation to every request. Short sessions were fine; long ones grew until requests hit the context limit and threw — and every call before that got steadily slower and more expensive as the history bloated. The failure was correlated with your best users (the ones who chat the longest), which is the cruelest place for a bug to live. This is the tech-lead story because the naive fixes each have a sharp edge.

Root cause. Unbounded history with no packing strategy. Sending the whole transcript every turn makes per-request cost and latency grow with session length and eventually exceeds the window entirely. The systemic gap: there was no token budget on the assembled prompt and no policy for what to do when the history exceeds it.

Fix. Bound the assembled context to a token budget: keep the newest turns verbatim (recency matters most), and replace the older overflow with a single rolling summary so nothing important is simply forgotten. Then cache the stable head of the prompt (story 1) so the retained context is cheap to re-send. The offline lab implements the packing policy against a hard budget:

pack_history.pydef pack_history(turns, budget, summary_cost=120):
    """Bound an unbounded chat history to a token BUDGET.
    Keep newest turns verbatim; replace the older overflow with one summary.
    turns: list of (role, tokens), oldest first. Returns
    (kept, n_summarized, total_tokens)."""
    kept, running = [], 0
    for role, tok in reversed(turns):          # newest first
        if running + tok <= budget:
            kept.append((role, tok))
            running += tok
        else:
            break
    kept.reverse()
    dropped = turns[:len(turns) - len(kept)]
    if dropped:
        running += summary_cost                # one rolling summary replaces them
    return kept, len(dropped), running

turns = [("user", 400), ("assistant", 900)] * 30   # 60 turns, ~39k tokens
raw = sum(t for _, t in turns)
kept, summarized, total = pack_history(turns, budget=4_000)
print(f"raw history      : {raw:,} tokens over {len(turns)} turns")
print(f"kept verbatim    : {len(kept)} turns")
print(f"summarized away  : {summarized} older turns -> 1 summary")
print(f"packed total     : {total:,} tokens (fits budget 4,000)")

raw history      : 39,000 tokens over 60 turns
kept verbatim    : 6 turns
summarized away  : 54 older turns -> 1 summary
packed total     : 4,020 tokens (fits budget 4,000)

Lesson. Never send an unbounded prompt. A budget with a keep-newest + summarize-the-rest policy turns a 39k-token transcript into ~4k tokens that fits, stays cheap, and keeps the thread coherent. Which levers combine: trim to a budget, summarize the overflow so context isn't lost, and cache the stable prefix so the retained history is nearly free to re-send. The class-killing move is a shared context-assembly component with a mandatory budget — so no feature can ship an unbounded prompt, and the context-limit error becomes structurally impossible.