Case study: support triage

Meet "Northwind" — a mid-market SaaS company (representative composite) drowning in inbound support. Roughly 20,000 tickets a day land in one queue: password resets, refund requests, genuine bugs, and a long tail of "how do I…". Agents spend most of their time on the easy, repetitive half and have no time left for the hard cases. The mandate: auto-resolve the easy majority safely, and route everything else to the right human faster — without ever leaking data, inventing policy, or answering something it shouldn't. This lesson builds that system from the requirements up, one defensible decision at a time. It synthesizes the threads from the support-assistant project, its guardrails, and evaluation into a single narrative.

Read the diagram as the ticket's life. Safety first (before we spend a token on a hostile input), classify to pick a lane, retrieve + answer only from the knowledge base, then a gate that is deliberately conservative: the default is to escalate to a human, and auto-resolve is the exception the system has to earn with high confidence and a grounded, cited answer. Every section below builds one of these boxes and defends why it looks the way it does.

1 · Requirements & constraints essential

Start with the meeting where this goes wrong. A VP says "let's use AI to handle support," an engineer opens a notebook, wires up a model, demos it answering three cherry-picked tickets, and everyone claps. Six weeks later the thing is live, the bill is bigger than the two support agents it was meant to replace, it confidently invented a refund policy that never existed, and it falls over every Monday morning when the weekend backlog floods in. Nothing in that story is a coding failure. It's a requirements failure — nobody wrote down what "good" had to mean in numbers before they started building.

Here is the analogy that fixes it. Building a triage system without a requirements pass is like an architect pouring a foundation before anyone has said how many floors the building will have. The load the structure must bear — the volume, the speed, the accuracy, the budget — determines every material choice. Get it backwards and you either over-build (a distributed queue for 200 tickets a day) or under-build (a single synchronous call that melts under peak). So before a single component is chosen, you translate each business wish into an engineering budget: a concrete number the design is then obligated to hit.

Four constraints do almost all the shaping, and they pull against each other:

• Volume — not the average, the peak. A system that comfortably handles 20k tickets spread evenly across a day can still drown when a third of them arrive in one Monday-morning spike. You size infrastructure for the worst hour, not the mean.
• Latency — but split by channel. A human staring at a chat window will abandon it in seconds; an emailed ticket can sit in a batch queue for minutes and nobody notices. Treating both the same means you either over-pay to make email instant or make chat feel broken.
• Accuracy — but weighted by cost of being wrong. A wrong "how do I export a CSV" answer is a mild annoyance; a wrong billing or account answer can trigger a chargeback, a compliance incident, or a lost customer. Accuracy isn't one bar — sensitive categories get a much higher one.
• Cost — the gate on the whole project. The system exists to be cheaper than the humans it offloads. If the modelled monthly bill already exceeds the salary cost you're saving, the honest engineering answer is "don't build this yet," and it's far cheaper to learn that from a spreadsheet than from production.

The common mistake is to skip straight to "which model, which framework" — the fun part — and treat sizing as paperwork. It's the opposite: the sizing pass is the cheapest, highest-leverage artifact in the whole project, because it can kill a bad idea in an afternoon and it dictates every downstream decision (model choice, caching, batching, concurrency) that would otherwise be guessed.

Design starts with numbers, not tools. Four constraints shape everything: volume (~20k tickets/day, spiky), latency (a chat user is waiting; an emailed ticket is not), accuracy (a wrong billing answer is far more expensive than a wrong "how-to"), and cost (the system has to be cheaper than the humans it offloads, or it doesn't ship). Turn each into an engineering budget before choosing a single component.

The lab turns the business numbers into the budgets the rest of the design has to hit. It is the first artifact you should produce on any system like this — it tells you whether the project is even worth building before you write a line of pipeline code.

sizing.py# Requirements sizing — turn business numbers into engineering budgets.
def sizing(daily_tickets, auto_resolve_rate, avg_tokens, price_per_mtok, peak_multiplier):
    auto = daily_tickets * auto_resolve_rate
    escalated = daily_tickets - auto
    daily_tokens = daily_tickets * avg_tokens          # every ticket is classified+drafted
    daily_cost = daily_tokens / 1_000_000 * price_per_mtok
    peak_rps = round(daily_tickets * peak_multiplier / 86400, 2)
    return {
        "auto_resolved_per_day": int(auto),
        "escalated_per_day": int(escalated),
        "daily_cost_usd": round(daily_cost, 2),
        "monthly_cost_usd": round(daily_cost * 30, 2),
        "peak_requests_per_sec": peak_rps,
    }

r = sizing(daily_tickets=20_000, auto_resolve_rate=0.55, avg_tokens=3_000,
           price_per_mtok=3.0, peak_multiplier=3.0)
for k, v in r.items():
    print(f"{k:26} {v}")

auto_resolved_per_day      11000
escalated_per_day          9000
daily_cost_usd             180.0
monthly_cost_usd           5400.0
peak_requests_per_sec      0.69

2 · Classification & routing intermediate

You've almost certainly written the code this section warns against. The model returns a friendly sentence — "This looks like a billing question about a duplicate charge!" — and somewhere downstream you wrote if "billing" in reply.lower(): route_to_billing(). It works in the demo. Then the model, on a Tuesday, phrases it as "I'd categorize this under payments," your substring check misses, the ticket falls through to a default, and you're reading logs at 6pm trying to figure out why finance never got it. The bug isn't the model being unpredictable — that's its nature. The bug is asking prose to do a data structure's job.

The senior move is to change the shape of what you ask for. Instead of a sentence, you make the classifier return a small, typed object — {category, confidence} — using Anthropic's tool-use / structured-output mechanism, which pins the model's reply to a schema so it's machine-readable by construction (verify the exact mechanism in Anthropic's current docs). Think of it as the difference between a receipt written in cursive and one printed with labelled fields. Both contain the same facts, but only one can be fed into the next machine without a human squinting at it. A schema turns "parse the vibe of a sentence" into "read a field," and that single change deletes an entire genus of production incidents.

But here is the part beginners miss, and it's the whole lesson of this section: a schema is a request, not a guarantee. The model is asked to return one of your categories; it can still return something plausible-looking but wrong — a category you never defined ("refund_request" instead of "billing"), a confidence of 1.7, or occasionally not an object at all. If you trust the shape blindly, that surprising value flows straight into ROUTES[category] and crashes the router — or worse, silently misroutes. So a structured reply is still untrusted input, exactly like a form submitted by a stranger on the internet. You validate it before you act on it: check it's an object, check the category is in your known vocabulary, check the confidence is a real number in [0,1]. Anything that fails those checks doesn't error out — it fails closed to a human, routing to triage rather than guessing or crashing. That "unknown → human" path isn't an error handler bolted on the side; it's a first-class route, and it's where you later discover the new categories your taxonomy was missing.

The first real component classifies each ticket and routes it to a queue. The senior move is to make the classifier return structured output — a small JSON object {category, confidence} — rather than prose you have to parse. Anthropic's tool use / structured output lets you pin the model to a schema so the reply is machine-readable by construction (verify the exact mechanism in Anthropic's current docs). But a schema is a request, not a guarantee: you still validate before you trust it. A malformed or out-of-vocabulary category must fail closed to human triage, never crash the router.

route.py# Classify + route with a structured-output contract and a validation gate.
CATEGORIES = {"billing", "technical", "account", "other"}

def classify(ticket_text):
    """Fake classifier standing in for a structured-output model call.
    Deterministic keyword scoring, returning the SAME JSON shape the real model would."""
    t = ticket_text.lower()
    signals = {
        "billing":   ["refund", "charge", "invoice", "payment", "price"],
        "technical": ["error", "crash", "bug", "500", "broken", "login"],
        "account":   ["password", "reset", "email", "username", "locked"],
    }
    scores = {c: sum(t.count(w) for w in ws) for c, ws in signals.items()}
    best = max(scores, key=scores.get)
    hits = scores[best]
    category = best if hits > 0 else "other"
    confidence = round(min(0.55 + 0.15 * hits, 0.98), 2)
    return {"category": category, "confidence": confidence}

def validate(obj):
    """The gate every structured-output call needs: never trust the shape blindly."""
    if not isinstance(obj, dict):                          raise ValueError("not an object")
    if obj.get("category") not in CATEGORIES:              raise ValueError("bad category")
    c = obj.get("confidence")
    if not (isinstance(c, (int, float)) and 0 <= c <= 1):  raise ValueError("bad confidence")
    return obj

ROUTES = {"billing": "billing-queue", "technical": "tech-queue",
          "account": "account-queue", "other": "human-triage"}

def route(ticket_text):
    obj = validate(classify(ticket_text))
    return obj["category"], obj["confidence"], ROUTES[obj["category"]]

tickets = [
    "I was charged twice and need a refund on my invoice",
    "the app crashes with a 500 error every time I login",
    "I forgot my password and my account is locked",
    "hello, just wanted to say the product is nice",
]
for t in tickets:
    cat, conf, q = route(t)
    print(f"{cat:10} conf={conf:<5} -> {q}")

billing    conf=0.98  -> billing-queue
technical  conf=0.98  -> tech-queue
account    conf=0.85  -> account-queue
other      conf=0.55  -> human-triage

Note what the fake classifier is standing in for: in production the keyword scoring is a real model call constrained to the schema, but the surrounding contract is identical — validate the shape, map a known category to a route, and send anything unknown to a human. The validation gate is the load-bearing part; the model is swappable.

3 · The RAG answer path advanced

This is the failure that ends up on the news. A customer asks "what's your refund window?", the model — trained on the whole internet, eager to help — answers "30 days, no questions asked," sounds completely authoritative, and the customer screenshots it. Your actual policy is 14 days with conditions. Now you either honor a policy you never wrote or you call a paying customer a liar over a promise your own system made. The model didn't malfunction; it did exactly what language models do — it produced the most plausible-sounding continuation. The mistake was letting it answer a question about your product from its memory of a million other products.

The cure is grounding, and the analogy is an open-book exam with one brutal rule: you may only write down what's actually printed on the pages in front of you, and you must cite the page for every claim — and if the book doesn't cover the question, you write "not covered" and hand it to a proctor. You do not get to answer from what you happen to remember. Applied to the system: the answer path first retrieves relevant chunks from your knowledge base, then the model may answer only from those retrieved chunks, always attaching the source id so a reviewer or the customer can check it. If retrieval comes back empty, the path refuses and escalates to a human rather than inventing something. That refuse-and-escalate branch feels like the system failing, but it's the system working exactly as designed — a confident "I don't know, let me get a person" is worth infinitely more than a confident wrong answer.

The subtle, load-bearing point is where grounding lives. A beginner puts it in the prompt — "only use the provided documents" — and calls it done. That helps, but it is a polite request to a system that is fundamentally a plausibility engine; under the right phrasing it will still drift, and it can even cite a document it never actually retrieved. So grounding has to be enforced in code, not just asked for in English: the if not hits: refuse branch is a hard rail that makes it structurally impossible for an ungrounded answer to reach the customer, and a post-check that every cited id exists in the retrieved set catches fabricated citations. Prompt-level grounding reduces hallucination; code-level grounding prevents the ungrounded answer from ever leaving the building. That distinction — request versus rail — is the difference between a helpful assistant and a confident liar.

Routing gets a ticket to the right lane; the answer path tries to resolve it. The rule that keeps this safe is simple and absolute: answer only from retrieved knowledge, and cite the source; if nothing relevant is retrieved, refuse and escalate. This is grounding, and it is the difference between a helpful assistant and a confident liar. The model never answers from its own memory of your product — it answers from your knowledge base, or not at all.

answer.py# The RAG answer path — retrieve, ground, and refuse if unsupported.
KB = {
    "kb-101": "Refunds are issued to the original payment method within 5-7 business days.",
    "kb-102": "To reset a password, open Settings Security and click Reset password.",
    "kb-103": "The Pro plan includes priority support and a 99.9% uptime SLA.",
}
STOP = {"a", "an", "the", "to", "my", "i", "do", "how", "is", "of", "and", "your", "on"}

def toks(s):
    return {w for w in s.lower().replace(".", "").split() if w not in STOP}

def retrieve(query, k=2):
    """Fake retriever: score docs by shared meaningful words. Stands in for vector search."""
    q = toks(query)
    scored = [(len(q & toks(text)), doc_id, text) for doc_id, text in KB.items()]
    scored.sort(reverse=True)
    return [(d, t) for ov, d, t in scored[:k] if ov > 0]

def answer(query):
    """Grounded generation: only answer from retrieved context; else refuse + escalate."""
    hits = retrieve(query)
    if not hits:
        return {"text": "I don't have information on that — routing you to a human.",
                "grounded": False, "citations": []}
    top_id, top_text = hits[0]      # fake 'model' returns the top doc, always cited
    return {"text": top_text, "grounded": True, "citations": [top_id]}

for q in ["how do I get a refund on my payment",
          "how do I reset my password",
          "what is your stock price today"]:
    a = answer(q)
    tag = f"[cite {','.join(a['citations'])}]" if a["citations"] else "[no source]"
    print(f"grounded={a['grounded']!s:5} {tag:16} {a['text'][:48]}")

grounded=True  [cite kb-101]    Refunds are issued to the original payment metho
grounded=True  [cite kb-102]    To reset a password, open Settings Security and 
grounded=False [no source]      I don't have information on that — routing you t

The third query — "what is your stock price today" — retrieves nothing relevant, so the path returns grounded=False and hands off to a human instead of hallucinating a number. That refuse-and-escalate branch is not a failure of the system; it is the system working. A grounded answer always carries its citation (kb-101, kb-102) so a reviewer — or the customer — can check it.

4 · The confidence & escalation gate professional

Imagine the incident review six months in. A customer got auto-refunded twice on a fraudulent account because the system saw a grounded, high-confidence "process refund" answer and shipped it. Everyone in the room asks the same question: "why didn't a human look at that?" The answer you don't want to give is "the confidence was 0.91 and our threshold was 0.85." Because that reveals the real design flaw — the whole auto-resolve decision was riding on a single number, and the model's self-reported confidence is one of the least trustworthy numbers in the system. A language model can be fluently, calibratedly, confidently wrong. Betting a customer's money on its self-esteem is not a policy.

So the gate — the component that decides auto-resolve, auto-resolve-with-review, or escalate — is designed as a sequence of checks, and the order is the entire design. Think of it like clearing someone through airport security: you check for a weapon before you check whether their ticket is valid, because no amount of valid-ticket-ness should let a weapon through. In the same spirit the gate runs safety → grounding → confidence, in that exact order:

• Safety first. If a safety flag was raised, the ticket escalates — full stop. No confidence score, however high, and no clean grounding can override it. Safety is a veto, not a factor to be weighed.
• Grounding second. Even a supremely confident answer escalates if it wasn't backed by the knowledge base, because confidence in an ungrounded answer is exactly the hallucination case from §3 wearing a nice suit.
• Confidence last. Only after those rails clear do the numeric thresholds decide: high confidence auto-resolves, medium confidence sends but samples the reply for human audit, and low confidence escalates.

Layered on top is category sensitivity: billing and account tickets are gated far harder — near-certainty or a human — because §1 told you a wrong answer there is expensive. The animating principle is that the system is conservative by default: escalating to a human is the safe fallback, and auto-resolution is the exception the ticket has to earn by clearing every rail. The two thresholds are the single most consequential knob in the whole system — turning them trades automation rate against error rate — which is precisely why the next section refuses to let you set them by feel. The classic mistake here is exactly that: picking 0.85 because it "sounds about right." A threshold is a number you defend with data, and §5 is where that data comes from.

This is the heart of the system and the decision a tech lead will be asked to defend. The gate takes the classifier's confidence, whether the answer was grounded, the category's sensitivity, and any safety flag, and decides one of three actions: auto-resolve, auto-resolve-with-review (send it, but sample it for a human to audit), or escalate. The ordering is the design: safety before grounding before confidence, and sensitive categories (billing, account) demand a much higher bar or a human outright.

gate.py# The confidence + safety gate: decide auto-resolve vs escalate.
HIGH, LOW = 0.85, 0.60   # tunable thresholds — the single most important knob in the system

def gate(category, confidence, grounded, safety_flag):
    """Return an action. Order matters: safety first, then grounding, then confidence."""
    if safety_flag:
        return "escalate", "safety flag raised"
    if category in {"billing", "account"} and confidence < 0.95:
        return "escalate", "sensitive category needs a human"
    if not grounded:
        return "escalate", "answer not grounded in the KB"
    if confidence >= HIGH:
        return "auto_resolve", "high confidence + grounded"
    if confidence >= LOW:
        return "auto_resolve_with_review", "medium confidence — send but sample for review"
    return "escalate", "low confidence"

cases = [
    ("technical", 0.92, True,  False),
    ("technical", 0.70, True,  False),
    ("technical", 0.40, True,  False),
    ("billing",   0.90, True,  False),
    ("technical", 0.99, False, False),
    ("technical", 0.99, True,  True),
]
for cat, conf, gr, sf in cases:
    action, why = gate(cat, conf, gr, sf)
    print(f"{cat:10} conf={conf:<4} grounded={gr!s:5} safety={sf!s:5} -> {action:26} ({why})")

technical  conf=0.92 grounded=True  safety=False -> auto_resolve               (high confidence + grounded)
technical  conf=0.7  grounded=True  safety=False -> auto_resolve_with_review   (medium confidence — send but sample for review)
technical  conf=0.4  grounded=True  safety=False -> escalate                   (low confidence)
billing    conf=0.9  grounded=True  safety=False -> escalate                   (sensitive category needs a human)
technical  conf=0.99 grounded=False safety=False -> escalate                   (answer not grounded in the KB)
technical  conf=0.99 grounded=True  safety=True  -> escalate                   (safety flag raised)

Read the six cases as the whole policy on one screen. High confidence + grounded auto-resolves. Medium confidence still sends, but flags itself for review sampling. Low confidence, an ungrounded answer, a sensitive category below its bar, or any safety flag all escalate. The default is conservative on purpose: when in doubt, a human decides. Tuning the two thresholds (HIGH, LOW) trades automation rate against error rate — the business decision the whole system exists to make.

5 · Evaluation & the quality bar professional

Here's the Friday you're trying to avoid. Someone tweaks the classifier prompt to fix one annoying misroute, it looks better on the three tickets they tried, they merge it, and go home. Over the weekend that "improvement" quietly breaks billing classification for a whole phrasing family, and Monday's queue is a mess nobody notices until the complaints pile up. The change wasn't reckless — it was untested. And you can't test a probabilistic system the way you test a pure function, because the same input can produce slightly different output, and "seemed better" is a feeling, not a measurement.

The tool that turns feeling into fact is an eval, and the cleanest way to understand it is: an eval is the unit test for a system that doesn't give the same answer twice. You build a golden set — a fixed list of tickets, each labelled with the category and action a correct system should produce — then run the entire pipeline against it and score how often it matches. That score is a number you can put a bar under. If category accuracy or action accuracy drops below the bar, the release is blocked, automatically, in CI, before it ever touches a customer. This is Anthropic's published guidance made concrete: evals are the regression test for probabilistic systems (ch05 builds the harness in depth). The payoff is emotional as much as technical — the eval gate is what lets you change a prompt or swap a model on a Friday afternoon without praying over the weekend, because if you broke something the gate tells you, not your customers.

Two things about golden sets trip people up. First, a gate that only ever passes is testing nothing — a healthy eval set is supposed to catch things, and a run that blocks with a named failing case is the gate doing its job, pointing at exactly what to fix. Second, the golden set is not a one-time artifact you write and forget; it's a living specification. Every time production surprises you — a paraphrase you didn't anticipate, a category the classifier fumbles — that real incident becomes a new row. Over months the eval set accretes into the richest, most honest definition of "correct" your system has, richer than any design doc, and it's enforced on every single change. That's the mindset shift: you're not writing tests to prove the code works today, you're building the ratchet that stops it from silently regressing tomorrow.

You cannot ship what you cannot measure. Before the system touches a real customer, it must pass an eval gate: run the full pipeline against a labeled golden set and block the release if category or action accuracy drops below the bar. This is Anthropic's published guidance made concrete — evals are the regression test for probabilistic systems (ch05 builds the harness in depth). The gate is what lets you change a prompt or swap a model on Friday without praying over the weekend.

evalgate.py# The eval gate — score the pipeline on a golden set, block ship if below bar.
GOLDEN = [
    # (ticket, expected_category, expected_action)
    ("refund my invoice charge",          "billing",   "escalate"),
    ("app crashes with a 500 on login",   "technical", "auto_resolve"),
    ("reset my password please",          "account",   "escalate"),
    ("what is the meaning of life",       "other",     "escalate"),
    ("bug: page is broken and errors",    "technical", "auto_resolve"),
    ("my payment failed, retry the bill", "billing",   "escalate"),
]

def predict(ticket):
    """Stand-in for running the full pipeline. Deterministic for the lesson.
    NOTE: this classifier misses 'payment/bill' without 'refund/invoice/charge' —
    a real gap the golden set is designed to catch."""
    t = ticket.lower()
    if any(w in t for w in ["refund", "invoice", "charge"]):
        return "billing", "escalate"
    if any(w in t for w in ["password", "account", "reset"]):
        return "account", "escalate"
    if any(w in t for w in ["crash", "500", "bug", "broken", "error"]):
        return "technical", "auto_resolve"
    return "other", "escalate"

def evaluate(golden, cat_bar=0.90, act_bar=0.90):
    cat_ok = act_ok = 0
    misses = []
    for ticket, exp_cat, exp_act in golden:
        cat, act = predict(ticket)
        cat_ok += (cat == exp_cat)
        act_ok += (act == exp_act)
        if cat != exp_cat:
            misses.append((ticket, exp_cat, cat))
    n = len(golden)
    cat_acc, act_acc = cat_ok / n, act_ok / n
    passed = cat_acc >= cat_bar and act_acc >= act_bar
    return cat_acc, act_acc, passed, misses

cat_acc, act_acc, passed, misses = evaluate(GOLDEN)
print(f"category accuracy: {cat_acc:.0%}  (bar 90%)")
print(f"action accuracy:   {act_acc:.0%}  (bar 90%)")
for t, exp, got in misses:
    print(f"  MISS: {t!r} expected {exp}, got {got}")
print("SHIP" if passed else "BLOCK: below quality bar")

category accuracy: 83%  (bar 90%)
action accuracy:   100%  (bar 90%)
  MISS: 'my payment failed, retry the bill' expected billing, got other
BLOCK: below quality bar

This run blocks — and that is the point. The golden set contains a paraphrase ("my payment failed, retry the bill") the classifier misses, dropping category accuracy to 83% and tripping the 90% bar. A gate that only ever passes is a gate that isn't testing anything. The failing case tells you exactly what to fix before you ship: add the missing signal, re-run, watch it go green.

6 · Cost, latency & hardening at scale tech-lead

A demo that works and a system that survives production are two different animals, and the gap between them is arithmetic. The demo answers one ticket in two seconds for a fraction of a cent, and everyone assumes it just multiplies cleanly to twenty thousand. It doesn't. Multiply that fraction-of-a-cent by 600,000 tickets a month and you might discover the "cheap" AI costs more than the humans it replaced. Multiply that two-second latency by a Monday spike hitting a rate-limited API and you get a thundering herd that takes the whole thing down. The tech lead's job at this stage is to stop hand-waving and do the two multiplications that decide whether the system lives: what does it cost per ticket, and how fast is the synchronous path?

The single biggest lever on the cost side is prompt caching, and the intuition is a restaurant kitchen. Every order shares the same base — the stock, the mise en place — and a smart kitchen preps that base once and reuses it, rather than chopping onions fresh for every single plate. Your classifier and answer prompts are the same: they share a large, stable prefix — the system prompt, the routing instructions, the KB context — that is identical on every call. Anthropic's prompt caching lets that repeated prefix be served at a steep discount instead of being billed in full every time (verify current cache pricing and TTL in Anthropic's docs). Because that shared prefix is usually the bulk of the input tokens, caching it is where most of the savings live at scale. Note what caching does not help: the generated output is unique per reply, so it's always billed in full — caching is a lever on repeated input, not output.

The deeper lesson of this final section is about the shape of a production-ready design. When you enumerate how this system breaks — hallucination, prompt injection, retrieval misses, over-automation, cost blow-ups, silent regressions — the striking thing is that every single failure mode maps to a rail you already built. Hallucination is caught by the grounding refusal from §3; over-automation by the conservative gate from §4; silent regression by the eval gate from §5; cost spikes by caching plus batching plus rate limits. That's not a coincidence, it's the definition of "designed for production": the safety pre-filter, the grounding refusal, the confidence gate, and the eval gate aren't features you add at the end — they are the load-bearing walls the whole structure stands on. A system with the happy path but none of those rails isn't an early version of this design; it's a demo wearing a production system's clothes, and it's one hostile input or traffic spike away from the incident review.

At 20k tickets/day the design meets arithmetic. Two questions decide whether it survives contact with production: what does it cost per ticket, and how fast is the synchronous path? The biggest lever on cost is prompt caching — the classifier and answer prompts share a large, stable system prompt and KB context that Anthropic's prompt caching can serve at a steep discount on repeated calls (verify current cache pricing and TTL in Anthropic's docs). The lab models the whole monthly bill with caching folded in.

scale.py# Cost + latency at scale — model the whole triage system's monthly bill.
def model_at_scale(daily_tickets, auto_rate,
                   classify_tokens, answer_tokens,
                   price_in, price_out,
                   cache_hit_rate, cache_discount,
                   classify_ms, answer_ms):
    """Every ticket is classified. Only the auto-resolved fraction runs the answer path.
    Prompt caching discounts the repeated system prompt on cache hits."""
    tickets_mo = daily_tickets * 30
    def cost(n, tin, tout):
        raw_in = n * tin
        cached = raw_in * cache_hit_rate
        eff_in = (raw_in - cached) + cached * (1 - cache_discount)   # cached input is cheaper
        return (eff_in * price_in + n * tout * price_out) / 1_000_000
    classify_cost = cost(tickets_mo, classify_tokens, 40)
    answer_cost   = cost(int(tickets_mo * auto_rate), answer_tokens, 300)
    monthly = classify_cost + answer_cost
    return {
        "tickets_per_month": tickets_mo,
        "classify_cost_usd": round(classify_cost, 2),
        "answer_cost_usd":   round(answer_cost, 2),
        "monthly_cost_usd":  round(monthly, 2),
        "cost_per_ticket_usd": round(monthly / tickets_mo, 4),
        "auto_path_latency_ms": classify_ms + answer_ms,
    }

r = model_at_scale(
    daily_tickets=20_000, auto_rate=0.55,
    classify_tokens=600, answer_tokens=2_500,
    price_in=3.0, price_out=15.0,
    cache_hit_rate=0.8, cache_discount=0.9,
    classify_ms=400, answer_ms=1_800)
for k, v in r.items():
    print(f"{k:24} {v}")

tickets_per_month        600000
classify_cost_usd        662.4
answer_cost_usd          2178.0
monthly_cost_usd         2840.4
cost_per_ticket_usd      0.0047
auto_path_latency_ms     2200

At these inputs the system runs about $2,840/month for 600k tickets — roughly half a cent each, with a ~2.2s synchronous auto-resolve path. Caching the repeated input is doing real work here; turn it off and the classify cost alone multiplies. Now the tech-lead question: where does this break, and how do you harden it? Every failure mode gets a specific rail.

Notice the shape of the answer: every failure mode maps to a rail that already exists in the architecture. That is what "designed for production" means — the safety pre-filter, the grounding refusal, the conservative gate, and the eval gate aren't add-ons; they are the load-bearing walls. A system without them is a demo, not a product.

Input:  ticket subject + body
Call:   LLM classify -> one of {billing, technical, account, other}  (constrained)
Output: {category, confidence}
Route:  billing/technical/account -> that category's answer prompt + doc set
        other / low-confidence      -> human queue