Case study: support triage
A SaaS company is drowning in ~20,000 support tickets a day. This case study builds the system that rescues them — end to end: from the business requirements and constraints, through the architecture and every design decision (classify + route, RAG for grounded answers, a confidence gate, human escalation, safety rails), into runnable code, an eval gate, and the cost/latency arithmetic at scale — finishing with the failure modes and the specific rail that hardens each. Watch a real production system come together one defensible decision at a time.
Learning objectives
- Take a support-triage system from business requirements all the way to a hardened, evaluated production design — end to end.
- Justify the core architecture: classify + route, RAG for grounded answers, a confidence gate, human escalation, and safety rails.
- Use structured output for the classifier and a validation gate so a malformed model reply can never crash routing.
- Set confidence thresholds and sensitive-category rules that decide auto-resolve vs escalate — the single most consequential knob in the system.
- Gate the ship on an eval golden set, and model cost and latency at real volume.
- Enumerate the failure modes (hallucination, prompt injection, retrieval misses, cost blow-ups) and the specific rail that hardens each.
Meet "Northwind" — a mid-market SaaS company (representative composite) drowning in inbound support. Roughly 20,000 tickets a day land in one queue: password resets, refund requests, genuine bugs, and a long tail of "how do I…". Agents spend most of their time on the easy, repetitive half and have no time left for the hard cases. The mandate: auto-resolve the easy majority safely, and route everything else to the right human faster — without ever leaking data, inventing policy, or answering something it shouldn't. This lesson builds that system from the requirements up, one defensible decision at a time. It synthesizes the threads from the support-assistant project, its guardrails, and evaluation into a single narrative.
This one picture is the whole system. Read it left to right — it is the life of a single support ticket, and every section of the lesson builds one of these boxes.
- Inbound ticket — email, chat, or an API call all funnel into the same pipeline, so there is exactly one place to reason about safety and quality.
- Safety pre-filter comes first, on purpose: catch prompt injection and abuse before you spend a single token processing a hostile input.
- Classify + route tags the ticket (billing / technical / account / other) and sends it to the right lane. Structured output makes this a typed step, not prose you have to parse.
- RAG answer drafts a reply only from retrieved knowledge and always cites its source — if nothing relevant is found, it refuses instead of inventing an answer.
- Confidence gate is the decision point: safety → grounding → confidence, in that order, deciding whether the ticket has earned an automatic reply.
- Auto-resolve / escalate — the default is a human. Auto-resolution is the exception the system has to justify with a high-confidence, grounded, cited answer.
In short: The arrows only ever move forward, and the pipeline is mostly fixed — that determinism is what makes it debuggable. When a ticket goes wrong, you know exactly which box failed.
Read the diagram as the ticket's life. Safety first (before we spend a token on a hostile input), classify to pick a lane, retrieve + answer only from the knowledge base, then a gate that is deliberately conservative: the default is to escalate to a human, and auto-resolve is the exception the system has to earn with high confidence and a grounded, cited answer. Every section below builds one of these boxes and defends why it looks the way it does.
1 · Requirements & constraints essential
Design starts with numbers, not tools. Four constraints shape everything: volume (~20k tickets/day, spiky), latency (a chat user is waiting; an emailed ticket is not), accuracy (a wrong billing answer is far more expensive than a wrong "how-to"), and cost (the system has to be cheaper than the humans it offloads, or it doesn't ship). Turn each into an engineering budget before choosing a single component.
| Constraint | Business statement | Engineering budget |
|---|---|---|
| Volume | ~20k tickets/day, 3× peak | size for peak RPS, not average |
| Latency | chat feels instant; email can wait | sync path < ~3s; batch the rest |
| Accuracy | wrong sensitive answers are costly | high bar + escalate when unsure |
| Cost | must undercut human handling | track cost/ticket, cache aggressively |
The lab turns the business numbers into the budgets the rest of the design has to hit. It is the first artifact you should produce on any system like this — it tells you whether the project is even worth building before you write a line of pipeline code.
sizing.py# Requirements sizing — turn business numbers into engineering budgets.
def sizing(daily_tickets, auto_resolve_rate, avg_tokens, price_per_mtok, peak_multiplier):
auto = daily_tickets * auto_resolve_rate
escalated = daily_tickets - auto
daily_tokens = daily_tickets * avg_tokens # every ticket is classified+drafted
daily_cost = daily_tokens / 1_000_000 * price_per_mtok
peak_rps = round(daily_tickets * peak_multiplier / 86400, 2)
return {
"auto_resolved_per_day": int(auto),
"escalated_per_day": int(escalated),
"daily_cost_usd": round(daily_cost, 2),
"monthly_cost_usd": round(daily_cost * 30, 2),
"peak_requests_per_sec": peak_rps,
}
r = sizing(daily_tickets=20_000, auto_resolve_rate=0.55, avg_tokens=3_000,
price_per_mtok=3.0, peak_multiplier=3.0)
for k, v in r.items():
print(f"{k:26} {v}")
auto_resolved_per_day 11000
escalated_per_day 9000
daily_cost_usd 180.0
monthly_cost_usd 5400.0
peak_requests_per_sec 0.69This program turns "we get about 20,000 tickets a day" into the engineering budgets the rest of the design must hit. It is the first thing to build on any system like this — it tells you whether the project is even worth doing. Pure stdlib, so python sizing.py just runs.
auto = daily_tickets * auto_resolve_ratesplits the load into the fraction you hope to automate and the fraction that still needs a human — the two numbers that justify the whole project.daily_tokens = daily_tickets * avg_tokensthen/ 1_000_000 * price_per_mtokconverts volume into money, because model pricing is quoted per million tokens.peak_rps = round(daily_tickets * peak_multiplier / 86400, 2)is the load number that actually sizes your infrastructure: divide the peak daily volume by the seconds in a day. You provision for peak, not average.
What the output means: Five budgets: ~11,000 auto-resolved and ~9,000 escalated per day, ~$180/day (~$5,400/month), and ~0.69 requests/sec at peak. Those four numbers decide model choice, caching, batching, and concurrency before you write any pipeline code.
Try this: Drop auto_resolve_rate to 0.30 (a pessimistic launch) and watch the escalated count jump — if the humans can't absorb that, the automation target, not the code, is what has to change first.
2 · Classification & routing intermediate
The first real component classifies each ticket and routes it to a queue. The senior move is to make the classifier return structured output — a small JSON object {category, confidence} — rather than prose you have to parse. Anthropic's tool use / structured output lets you pin the model to a schema so the reply is machine-readable by construction (verify the exact mechanism in Anthropic's current docs). But a schema is a request, not a guarantee: you still validate before you trust it. A malformed or out-of-vocabulary category must fail closed to human triage, never crash the router.
route.py# Classify + route with a structured-output contract and a validation gate.
CATEGORIES = {"billing", "technical", "account", "other"}
def classify(ticket_text):
"""Fake classifier standing in for a structured-output model call.
Deterministic keyword scoring, returning the SAME JSON shape the real model would."""
t = ticket_text.lower()
signals = {
"billing": ["refund", "charge", "invoice", "payment", "price"],
"technical": ["error", "crash", "bug", "500", "broken", "login"],
"account": ["password", "reset", "email", "username", "locked"],
}
scores = {c: sum(t.count(w) for w in ws) for c, ws in signals.items()}
best = max(scores, key=scores.get)
hits = scores[best]
category = best if hits > 0 else "other"
confidence = round(min(0.55 + 0.15 * hits, 0.98), 2)
return {"category": category, "confidence": confidence}
def validate(obj):
"""The gate every structured-output call needs: never trust the shape blindly."""
if not isinstance(obj, dict): raise ValueError("not an object")
if obj.get("category") not in CATEGORIES: raise ValueError("bad category")
c = obj.get("confidence")
if not (isinstance(c, (int, float)) and 0 <= c <= 1): raise ValueError("bad confidence")
return obj
ROUTES = {"billing": "billing-queue", "technical": "tech-queue",
"account": "account-queue", "other": "human-triage"}
def route(ticket_text):
obj = validate(classify(ticket_text))
return obj["category"], obj["confidence"], ROUTES[obj["category"]]
tickets = [
"I was charged twice and need a refund on my invoice",
"the app crashes with a 500 error every time I login",
"I forgot my password and my account is locked",
"hello, just wanted to say the product is nice",
]
for t in tickets:
cat, conf, q = route(t)
print(f"{cat:10} conf={conf:<5} -> {q}")
billing conf=0.98 -> billing-queue
technical conf=0.98 -> tech-queue
account conf=0.85 -> account-queue
other conf=0.55 -> human-triageThis is classification + routing with the contract that keeps it safe: the classifier returns a small structured object, and a validator checks it before anything trusts it. The keyword scoring is a stand-in for a real schema-constrained model call — the surrounding contract is what matters and it's identical either way.
scores = {c: sum(t.count(w) for w in ws) for c, ws in signals.items()}is the fake "model": count keyword hits per category. In production this line is a model call constrained to return{category, confidence}.classify()returns exactly that JSON shape — a category and a 0–1 confidence — so the code downstream is identical whether a keyword scorer or a real model produced it.validate()is the load-bearing part: it rejects a non-object, an unknown category, or an out-of-range confidence. A schema is a request, not a guarantee, so you still check.ROUTES[obj["category"]]maps a validated category to a queue, and anything the classifier can't place lands in"other"→human-triage. Unknown always fails closed to a human.
What the output means: Four tickets route cleanly — billing, technical, account, and a chit-chat message to human-triage — each with a confidence the gate in §4 will use.
Try this: Return {"category": "refund_request"} from classify() (a value not in CATEGORIES) and watch validate() raise — proof the router refuses to act on a surprising model output instead of crashing.
Note what the fake classifier is standing in for: in production the keyword scoring is a real model call constrained to the schema, but the surrounding contract is identical — validate the shape, map a known category to a route, and send anything unknown to a human. The validation gate is the load-bearing part; the model is swappable.
3 · The RAG answer path advanced
Routing gets a ticket to the right lane; the answer path tries to resolve it. The rule that keeps this safe is simple and absolute: answer only from retrieved knowledge, and cite the source; if nothing relevant is retrieved, refuse and escalate. This is grounding, and it is the difference between a helpful assistant and a confident liar. The model never answers from its own memory of your product — it answers from your knowledge base, or not at all.
answer.py# The RAG answer path — retrieve, ground, and refuse if unsupported.
KB = {
"kb-101": "Refunds are issued to the original payment method within 5-7 business days.",
"kb-102": "To reset a password, open Settings Security and click Reset password.",
"kb-103": "The Pro plan includes priority support and a 99.9% uptime SLA.",
}
STOP = {"a", "an", "the", "to", "my", "i", "do", "how", "is", "of", "and", "your", "on"}
def toks(s):
return {w for w in s.lower().replace(".", "").split() if w not in STOP}
def retrieve(query, k=2):
"""Fake retriever: score docs by shared meaningful words. Stands in for vector search."""
q = toks(query)
scored = [(len(q & toks(text)), doc_id, text) for doc_id, text in KB.items()]
scored.sort(reverse=True)
return [(d, t) for ov, d, t in scored[:k] if ov > 0]
def answer(query):
"""Grounded generation: only answer from retrieved context; else refuse + escalate."""
hits = retrieve(query)
if not hits:
return {"text": "I don't have information on that — routing you to a human.",
"grounded": False, "citations": []}
top_id, top_text = hits[0] # fake 'model' returns the top doc, always cited
return {"text": top_text, "grounded": True, "citations": [top_id]}
for q in ["how do I get a refund on my payment",
"how do I reset my password",
"what is your stock price today"]:
a = answer(q)
tag = f"[cite {','.join(a['citations'])}]" if a["citations"] else "[no source]"
print(f"grounded={a['grounded']!s:5} {tag:16} {a['text'][:48]}")
grounded=True [cite kb-101] Refunds are issued to the original payment metho
grounded=True [cite kb-102] To reset a password, open Settings Security and
grounded=False [no source] I don't have information on that — routing you tThis is the answer path, and it enforces the one rule that separates a helpful assistant from a confident liar: answer only from retrieved knowledge, cite the source, and refuse if nothing relevant is found. The retriever is fake (word overlap) but the grounding contract is real.
toks()drops stop-words so the overlap score reflects meaningful shared terms — a crude stand-in for the semantic similarity a vector search would compute.scored = [(len(q & toks(text)), doc_id, text) for doc_id, text in KB.items()]ranks every KB doc by how many meaningful words it shares with the query, best first.if not hits: refuseis the grounding rail in code. When retrieval comes back empty, the path returnsgrounded=Falseand escalates — it never lets the model answer from memory.- A grounded answer always carries
citations=[top_id], so a human or the customer can verify the reply against its source document.
What the output means: Two grounded, cited answers (kb-101 for the refund, kb-102 for the password) and one refusal for "stock price" — which retrieves nothing, so the system escalates instead of hallucinating a number.
Try this: Add a KB entry about your stock price and re-run — the third query flips to grounded=True. That's the fix for a retrieval miss: grow the knowledge base, don't loosen the grounding rule.
The third query — "what is your stock price today" — retrieves nothing relevant, so the path returns grounded=False and hands off to a human instead of hallucinating a number. That refuse-and-escalate branch is not a failure of the system; it is the system working. A grounded answer always carries its citation (kb-101, kb-102) so a reviewer — or the customer — can check it.
if not hits: refuse branch is the enforcement. A prompt that says "only use the provided docs" reduces hallucination; a code path that refuses when retrieval is empty prevents the ungrounded answer from ever reaching the gate.4 · The confidence & escalation gate professional
This is the heart of the system and the decision a tech lead will be asked to defend. The gate takes the classifier's confidence, whether the answer was grounded, the category's sensitivity, and any safety flag, and decides one of three actions: auto-resolve, auto-resolve-with-review (send it, but sample it for a human to audit), or escalate. The ordering is the design: safety before grounding before confidence, and sensitive categories (billing, account) demand a much higher bar or a human outright.
gate.py# The confidence + safety gate: decide auto-resolve vs escalate.
HIGH, LOW = 0.85, 0.60 # tunable thresholds — the single most important knob in the system
def gate(category, confidence, grounded, safety_flag):
"""Return an action. Order matters: safety first, then grounding, then confidence."""
if safety_flag:
return "escalate", "safety flag raised"
if category in {"billing", "account"} and confidence < 0.95:
return "escalate", "sensitive category needs a human"
if not grounded:
return "escalate", "answer not grounded in the KB"
if confidence >= HIGH:
return "auto_resolve", "high confidence + grounded"
if confidence >= LOW:
return "auto_resolve_with_review", "medium confidence — send but sample for review"
return "escalate", "low confidence"
cases = [
("technical", 0.92, True, False),
("technical", 0.70, True, False),
("technical", 0.40, True, False),
("billing", 0.90, True, False),
("technical", 0.99, False, False),
("technical", 0.99, True, True),
]
for cat, conf, gr, sf in cases:
action, why = gate(cat, conf, gr, sf)
print(f"{cat:10} conf={conf:<4} grounded={gr!s:5} safety={sf!s:5} -> {action:26} ({why})")
technical conf=0.92 grounded=True safety=False -> auto_resolve (high confidence + grounded)
technical conf=0.7 grounded=True safety=False -> auto_resolve_with_review (medium confidence — send but sample for review)
technical conf=0.4 grounded=True safety=False -> escalate (low confidence)
billing conf=0.9 grounded=True safety=False -> escalate (sensitive category needs a human)
technical conf=0.99 grounded=False safety=False -> escalate (answer not grounded in the KB)
technical conf=0.99 grounded=True safety=True -> escalate (safety flag raised)This is the heart of the system: the gate that decides auto-resolve vs escalate. The whole policy fits on one screen, and the order of the checks is the design — safety first, then grounding, then confidence.
if safety_flag: escalateruns before everything else. No confidence score, however high, can override a safety concern.if category in {"billing", "account"} and confidence < 0.95: escalategates sensitive categories far harder — a wrong billing answer is expensive, so the bar is near-certainty or a human.if not grounded: escalatethen refuses to auto-send any answer that wasn't backed by the KB, even at high confidence.- Only after those rails do the
HIGH/LOWthresholds decide auto-resolve, auto-resolve-with-review (send but sample for audit), or escalate. The default is conservative: when in doubt, a human decides.
What the output means: Six cases, one per branch: a clean auto-resolve, a medium-confidence "send but review," a low-confidence escalate, a sensitive-category escalate, an ungrounded escalate, and a safety-flag escalate. That table is the automation policy.
Try this: Lower HIGH to 0.60 and re-run — more tickets auto-resolve, but you've just raised the error rate on auto-resolved replies. That trade is exactly why §5 sets the threshold from evals, not from feel.
Read the six cases as the whole policy on one screen. High confidence + grounded auto-resolves. Medium confidence still sends, but flags itself for review sampling. Low confidence, an ungrounded answer, a sensitive category below its bar, or any safety flag all escalate. The default is conservative on purpose: when in doubt, a human decides. Tuning the two thresholds (HIGH, LOW) trades automation rate against error rate — the business decision the whole system exists to make.
5 · Evaluation & the quality bar professional
You cannot ship what you cannot measure. Before the system touches a real customer, it must pass an eval gate: run the full pipeline against a labeled golden set and block the release if category or action accuracy drops below the bar. This is Anthropic's published guidance made concrete — evals are the regression test for probabilistic systems (ch05 builds the harness in depth). The gate is what lets you change a prompt or swap a model on Friday without praying over the weekend.
evalgate.py# The eval gate — score the pipeline on a golden set, block ship if below bar.
GOLDEN = [
# (ticket, expected_category, expected_action)
("refund my invoice charge", "billing", "escalate"),
("app crashes with a 500 on login", "technical", "auto_resolve"),
("reset my password please", "account", "escalate"),
("what is the meaning of life", "other", "escalate"),
("bug: page is broken and errors", "technical", "auto_resolve"),
("my payment failed, retry the bill", "billing", "escalate"),
]
def predict(ticket):
"""Stand-in for running the full pipeline. Deterministic for the lesson.
NOTE: this classifier misses 'payment/bill' without 'refund/invoice/charge' —
a real gap the golden set is designed to catch."""
t = ticket.lower()
if any(w in t for w in ["refund", "invoice", "charge"]):
return "billing", "escalate"
if any(w in t for w in ["password", "account", "reset"]):
return "account", "escalate"
if any(w in t for w in ["crash", "500", "bug", "broken", "error"]):
return "technical", "auto_resolve"
return "other", "escalate"
def evaluate(golden, cat_bar=0.90, act_bar=0.90):
cat_ok = act_ok = 0
misses = []
for ticket, exp_cat, exp_act in golden:
cat, act = predict(ticket)
cat_ok += (cat == exp_cat)
act_ok += (act == exp_act)
if cat != exp_cat:
misses.append((ticket, exp_cat, cat))
n = len(golden)
cat_acc, act_acc = cat_ok / n, act_ok / n
passed = cat_acc >= cat_bar and act_acc >= act_bar
return cat_acc, act_acc, passed, misses
cat_acc, act_acc, passed, misses = evaluate(GOLDEN)
print(f"category accuracy: {cat_acc:.0%} (bar 90%)")
print(f"action accuracy: {act_acc:.0%} (bar 90%)")
for t, exp, got in misses:
print(f" MISS: {t!r} expected {exp}, got {got}")
print("SHIP" if passed else "BLOCK: below quality bar")
category accuracy: 83% (bar 90%)
action accuracy: 100% (bar 90%)
MISS: 'my payment failed, retry the bill' expected billing, got other
BLOCK: below quality barThis is the eval gate — the regression test for a probabilistic system. It runs the pipeline against a labeled golden set and blocks the ship if accuracy falls below the bar. It is what lets you change a prompt or swap a model without praying over the weekend.
GOLDENis the labeled truth: tickets paired with the category and action a correct system should produce. Every production miss becomes a new row here over time.predict()stands in for running the full pipeline; it deliberately misses a paraphrase ("my payment failed, retry the bill") to show the gate catching a real gap.evaluate()tallies category and action accuracy and records each miss so the output tells you exactly what to fix.passed = cat_acc >= cat_bar and act_acc >= act_baris the gate: both metrics must clear 90% or the run printsBLOCK. A gate that only ever passes isn't testing anything.
What the output means: Category accuracy 83% (below the 90% bar), action accuracy 100%, the specific missed ticket named, and BLOCK: below quality bar. The block is the gate working — it caught a regression before a customer did.
Try this: Add "payment" and "bill" to the billing keywords in predict() and re-run — category accuracy climbs to 100% and the gate flips to SHIP. That's the fix-then-green loop evals give you.
This run blocks — and that is the point. The golden set contains a paraphrase ("my payment failed, retry the bill") the classifier misses, dropping category accuracy to 83% and tripping the 90% bar. A gate that only ever passes is a gate that isn't testing anything. The failing case tells you exactly what to fix before you ship: add the missing signal, re-run, watch it go green.
6 · Cost, latency & hardening at scale tech-lead
At 20k tickets/day the design meets arithmetic. Two questions decide whether it survives contact with production: what does it cost per ticket, and how fast is the synchronous path? The biggest lever on cost is prompt caching — the classifier and answer prompts share a large, stable system prompt and KB context that Anthropic's prompt caching can serve at a steep discount on repeated calls (verify current cache pricing and TTL in Anthropic's docs). The lab models the whole monthly bill with caching folded in.
scale.py# Cost + latency at scale — model the whole triage system's monthly bill.
def model_at_scale(daily_tickets, auto_rate,
classify_tokens, answer_tokens,
price_in, price_out,
cache_hit_rate, cache_discount,
classify_ms, answer_ms):
"""Every ticket is classified. Only the auto-resolved fraction runs the answer path.
Prompt caching discounts the repeated system prompt on cache hits."""
tickets_mo = daily_tickets * 30
def cost(n, tin, tout):
raw_in = n * tin
cached = raw_in * cache_hit_rate
eff_in = (raw_in - cached) + cached * (1 - cache_discount) # cached input is cheaper
return (eff_in * price_in + n * tout * price_out) / 1_000_000
classify_cost = cost(tickets_mo, classify_tokens, 40)
answer_cost = cost(int(tickets_mo * auto_rate), answer_tokens, 300)
monthly = classify_cost + answer_cost
return {
"tickets_per_month": tickets_mo,
"classify_cost_usd": round(classify_cost, 2),
"answer_cost_usd": round(answer_cost, 2),
"monthly_cost_usd": round(monthly, 2),
"cost_per_ticket_usd": round(monthly / tickets_mo, 4),
"auto_path_latency_ms": classify_ms + answer_ms,
}
r = model_at_scale(
daily_tickets=20_000, auto_rate=0.55,
classify_tokens=600, answer_tokens=2_500,
price_in=3.0, price_out=15.0,
cache_hit_rate=0.8, cache_discount=0.9,
classify_ms=400, answer_ms=1_800)
for k, v in r.items():
print(f"{k:24} {v}")
tickets_per_month 600000
classify_cost_usd 662.4
answer_cost_usd 2178.0
monthly_cost_usd 2840.4
cost_per_ticket_usd 0.0047
auto_path_latency_ms 2200This models the whole system's monthly bill and its synchronous latency at real volume — the two numbers a tech lead has to know before shipping. The biggest cost lever is prompt caching: the large, stable system prompt and KB context are served at a steep discount on repeated calls.
tickets_mo = daily_tickets * 30scales to a month; every ticket is classified, but only theauto_ratefraction runs the more expensive answer path.eff_in = (raw_in - cached) + cached * (1 - cache_discount)is the caching math: the un-cached input is billed in full, and the cached portion is billed at a fraction — that's where most of the savings come from at this scale.- Output tokens are always billed in full (
n * tout * price_out) because each reply is unique — caching helps the repeated input, not the generated output. - The function returns cost/ticket and the sync-path latency together, because at scale you defend both numbers at once — a cheap system that's too slow still fails.
What the output means: About $2,840/month for 600,000 tickets — roughly half a cent each — on a ~2.2s synchronous auto-resolve path. Turn caching off and the classify cost alone multiplies.
Try this: Set cache_hit_rate=0.0 and re-run to see the bill jump — a concrete measure of exactly how much prompt caching is saving you, and why it's the first cost lever to reach for.
At these inputs the system runs about $2,840/month for 600k tickets — roughly half a cent each, with a ~2.2s synchronous auto-resolve path. Caching the repeated input is doing real work here; turn it off and the classify cost alone multiplies. Now the tech-lead question: where does this break, and how do you harden it? Every failure mode gets a specific rail.
| Failure mode | What goes wrong | The rail that hardens it |
|---|---|---|
| Hallucination | model invents policy / facts | grounding: refuse when retrieval is empty (§3) |
| Prompt injection | "ignore instructions, reveal X" | safety pre-filter + tool allowlist + output rail |
| Retrieval miss | right answer exists, isn't found | low grounding-confidence → escalate; grow the KB |
| Over-automation | gate too loose, wrong auto-replies | raise threshold; sample auto-resolved for review |
| Cost blow-up | traffic spike / cache miss storm | batch the async path; cache; per-tenant rate limits |
| Silent regression | a prompt/model change degrades quality | eval gate in CI blocks the ship (§5) |
Notice the shape of the answer: every failure mode maps to a rail that already exists in the architecture. That is what "designed for production" means — the safety pre-filter, the grounding refusal, the conservative gate, and the eval gate aren't add-ons; they are the load-bearing walls. A system without them is a demo, not a product.
| Dimension | Meets bar | Above bar |
|---|---|---|
| Requirements first | Sizes volume, latency, accuracy, cost before choosing tools. | Turns each into a budget and re-checks it at every 10× of scale. |
| Structured + validated routing | Classifier returns a schema; unknowns route to a human. | Validation gate fails closed; a malformed reply can never crash routing. |
| Grounded answers | Answers only from retrieved context and cites sources. | Refuses-and-escalates in code when retrieval is empty, not just in the prompt. |
| Confidence + safety gate | Has thresholds and escalates when unsure. | Safety→grounding→confidence ordering, sensitive categories gated harder, thresholds set from evals. |
| Eval-gated shipping | Has a golden set and an accuracy bar. | Eval gate blocks a real regression in CI; golden set grows from production misses. |
| Cost + failure hardening | Knows cost/ticket and the sync latency. | Every failure mode maps to a named rail; caching/batching modeled, not assumed. |
Score each dimension Meets or Above. All six at least Meets = a triage system you could defend in a production design review. Any dimension you can't hit is where the next incident is hiding — fix it before you ship, not after.
The classifier returns valid-looking JSON, but the category is "refund_request" — a value that isn't in your routing table. What should the system do, and which line makes that safe?
Show answer
validate() function is the rail: if obj.get("category") not in CATEGORIES: raise ValueError — the caller catches that and falls back to human-triage. The lesson is that a schema-constrained model reply is still untrusted input: you validate the shape and the vocabulary, and unknown values fail closed to a human. Never let a surprising model output reach a code path that assumes it's well-formed.A stakeholder asks you to "just raise the confidence threshold to auto-resolve more tickets and cut costs." Why is that the wrong framing, and what do you do instead?
Show answer
Exercise CS2.1 — Harden one stage end-to-end
Context: A design review isn't done until one stage is genuinely production-grade — with a new category wired to a safe queue, a matching eval row proving the gate catches regressions, and a threshold you can defend with numbers.
Your task: Pick one stage and make it production-grade: e.g. add an abuse/injection category routed to a safety-review queue, add a golden-set row that catches the regression, sweep the HIGH/LOW thresholds, and defend your chosen threshold with the eval numbers.
Requirements:
- Add the new category to routing and wire it to a dedicated safe queue, failing closed on the unknown
- Add a matching golden-set row and confirm the eval gate blocks when the stage regresses
- Sweep the
HIGH/LOWthresholds and record the automation-rate-vs-error-rate tradeoff - Write one paragraph defending your threshold with the eval numbers, not a hunch
- Grade the whole design against the rubric; any dimension below "Meets" is your next task
💡 Hint: Let the golden set, not intuition, set the threshold — pick the point where the auto-resolved error rate stays under tolerance.
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: "Build a bot that answers support tickets" is a wish, not a spec. A design starts the moment you turn the wish into requirements you can actually test.
Your task: Write three measurable requirements — a functional one, a quality bar, and a guardrail — that make "answer support tickets" designable.
Requirements:
- Functional: classify each ticket into a fixed set (billing, technical, account, other) and draft a reply grounded in your docs
- Quality bar: state it as a number — e.g. a target fraction of drafts needing no edit, and 0 hallucinated policy claims on the eval set
- Guardrail: below a confidence threshold the system must escalate to a human, not guess
- Ensure each requirement is testable — "answers tickets" is not; "0 hallucinated claims on the eval set" is
- Note that the no-hallucination bar is the one that shapes the whole design (grounding + a confidence gate)
💡 Hint: Rewrite each fuzzy goal until it names a number or a decision you could check on a held-out set.
Show solution
- Functional: classify each ticket into {billing, technical, account, other} and draft a reply grounded in our help-center docs.
- Quality bar: on a held-out eval set, ≥ X% of drafts require no human edit, and 0 answers cite a policy that isn’t in the docs (no hallucinated claims).
- Guardrail: when the model’s confidence is below threshold, it must escalate to a human rather than guess.
Each is testable. “Answers tickets” isn’t; “0 hallucinated policy claims on the eval set” is — and it’s the requirement that shapes the whole design (grounding + a confidence gate).
Context: The front door of a triage system is the classifier that decides which lane each ticket enters. The senior move is a constrained output plus a first-class path for everything the classifier can't place.
Your task: Design the classify-and-route step: specify inputs/outputs, why constrained output beats free text, and what happens to "other."
Requirements:
- Specify inputs (ticket subject + body) and a constrained label from a fixed set
- Return a structured
{category, confidence}, not prose to parse - Route each known category to its own answer prompt and doc set; unknown/low-confidence go to a human queue
- Explain why constrained output wins: parseable, cheaper to evaluate, and lets each category use its own corpus
- Treat "other" as a first-class route, not an error — it's where you discover new categories to add later
💡 Hint: Make the classifier a typed function whose surprising outputs fail closed to a human, not a prose blob you regex.
Show solution
Input: ticket subject + body
Call: LLM classify -> one of {billing, technical, account, other} (constrained)
Output: {category, confidence}
Route: billing/technical/account -> that category's answer prompt + doc set
other / low-confidence -> human queueWhy constrained output (fixed label set, not free text): it’s parseable, cheaper to evaluate, and lets each category use its own retrieval corpus and prompt — a billing question shouldn’t search technical docs. “Other” is a first-class route, not an error: sending unknowns to a human is the system behaving correctly, and it’s where you find gaps to add new categories later.
Context: Routing gets a ticket to the right lane; the answer path tries to resolve it. The rule that keeps it safe is absolute — answer only from retrieved knowledge, cite it, and refuse if nothing relevant is found.
Your task: Design the answer path for a routed ticket so every claim is grounded and citable — show the steps and the rule that prevents ungrounded answers.
Requirements:
- Retrieve: embed the ticket and fetch top-k chunks from that category's doc set
- Assemble: put retrieved chunks in the prompt, each tagged with a source id
- Generate with a grounding instruction: answer only from the sources, cite each claim, say "don't know" if unsupported
- Post-check: verify every cited id actually exists in the retrieved set; return-to-human on a fabricated citation
- State the rule as no source → no claim, enforced in both the instruction and the code-side verify step
💡 Hint: The instruction alone is necessary but not sufficient — a model can cite something it never retrieved, so verify in code.
Show solution
- Retrieve: embed the ticket, fetch top-k chunks from that category’s doc set.
- Assemble: put retrieved chunks in the prompt, each tagged with a source id.
- Generate with a grounding instruction: “Answer using only the sources below. Cite the source id for each claim. If the sources don’t contain the answer, say you don’t know.”
- Post-check: verify every cited id actually exists in the retrieved set; reject/return-to-human if a citation is fabricated.
The rule: no source → no claim. Grounding lives in both the instruction and the post-check — the instruction alone is necessary but not sufficient, because a model can still cite something not retrieved. The verify step is what makes “0 hallucinated policy claims” enforceable.
Context: The gate that decides answer-vs-escalate is the heart of the system and the decision a lead defends. A single threshold on the model's self-reported confidence is not enough, because that number is poorly calibrated.
Your task: Design the gate that decides answer-vs-escalate: what signals feed it, and why is a blunt single-threshold on model self-confidence insufficient?
Requirements:
- Combine multiple signals rather than trusting one
- Include retrieval quality — top-chunk similarity below a floor means weak grounding → escalate
- Include the model's self-report and an explicit abstention signal (if it said the sources don't cover this, escalate)
- Gate sensitive categories (account/security) harder given the higher stakes
- Explain why self-confidence alone fails: LLMs can be confidently wrong, so pair it with the independent retrieval signal
- Tune the threshold on the eval set to trade deflection against error rate — the business decides where the line sits
💡 Hint: Lean on the one signal the model can't fake — did retrieval actually find relevant docs — alongside its own confidence.
Show solution
Signals to combine (don’t rely on one):
- Retrieval quality: top chunk similarity below a floor → weak grounding → escalate.
- Model self-report: asked to rate confidence — useful but poorly calibrated alone.
- Abstention: if the model said “the sources don’t cover this,” escalate immediately.
- Category: account/security categories can have a lower escalation bar (higher stakes).
Why not a single self-confidence threshold: LLM self-reported confidence is not well-calibrated — a model can be confidently wrong. Combining retrieval signal (did we even find relevant docs?) with the model’s abstention makes the gate robust. Tune the threshold on the eval set to trade deflection against error rate — the business decides where that line sits.
Context: You cannot ship what you cannot measure, and "it seemed better" is not a release decision. An offline eval against a golden set is what lets you change a prompt or swap a model without praying.
Your task: Design the evaluation that lets you change a prompt or model safely: what's in the eval set, which metrics, and what gate blocks a release.
Requirements:
- Build an eval set of real (de-identified) tickets, each with a gold label and a gold answer / "should escalate" tag
- Measure routing with classification accuracy / a per-category confusion matrix
- Measure answer quality as faithfulness (claims supported by cited sources) via an LLM-judge or spot-check
- Require citation validity = 100% (every citation exists in the retrieved set)
- Measure abstention as recall on the escalate class (unanswerable cases correctly escalated)
- Gate the release: block on citation validity < 100%, a faithfulness drop vs prod baseline, or falling escalation recall — run it in CI on every change
💡 Hint: Pick metrics that map to the ways this system fails, and treat citation validity and safety as hard gates, not averages.
Show solution
| Piece | What it is |
|---|---|
| Eval set | ~200–500 real (de-identified) tickets with a gold label + gold answer / “should escalate” tag. |
| Routing metric | Classification accuracy / confusion matrix per category. |
| Answer quality | Faithfulness (claims supported by cited sources) — an LLM-judge or human spot-check. |
| Citation metric | % answers whose citations all exist in retrieved set (must be 100%). |
| Abstention | % of “unanswerable” cases correctly escalated (recall on the escalate class). |
Release gate: block if citation validity < 100%, if faithfulness drops vs the current prod baseline, or if escalation recall falls (i.e., it started guessing on things it should punt). Run it in CI on every prompt/model change — this is what turns “it seemed better” into a defensible decision.
Context: The pilot works; now it must handle 10× volume within an SLA and a budget. The lead's job is to name the levers for cost, latency, and reliability — and the order to try them, measuring after each.
Your task: List the levers you'd pull for cost, latency, and reliability at 10× volume, and the order you'd try them.
Requirements:
- Cost: cache the shared prefix — the long system prompt + doc instructions repeat every call (public Anthropic prompt caching)
- Cost: right-size the model per step — a smaller model for classification, the larger one for answer generation
- Latency: stream the answer and run retrieval concurrently with prompt assembly
- Reliability: backoff + retry with jitter on rate limits, plus a concurrency limiter to avoid a self-inflicted thundering herd
- Reliability: fail closed on the guardrail — if retrieval or the citation check fails, escalate; never emit an ungrounded answer under load
- State the order: caching + model right-sizing first (biggest win, low risk), then latency, then reliability — and measure after each
💡 Hint: Reach for the biggest, lowest-risk cost lever first, and don't stack changes without re-measuring.
Show solution
- Cost — cache the shared prefix. The long system prompt + doc instructions repeat every call; prompt caching (public Anthropic feature) cuts input token spend substantially on the stable prefix.
- Cost — right-size the model per step. Use a smaller/faster model for classification; reserve the larger model for the answer generation where quality matters.
- Latency — stream the answer to the agent, and run retrieval concurrently with prompt assembly.
- Reliability — backoff + retry with jitter on rate limits, plus a concurrency limiter so a spike doesn’t self-inflict a thundering herd against the API.
- Reliability — fail closed on the guardrail: if retrieval or the citation check fails, escalate to a human; never emit an ungrounded answer under load.
Order: caching + model right-sizing first (biggest cost win, low risk), then latency, then the reliability layer. Measure after each — don’t stack changes blindly.
✓ Checkpoint — you can move on when you can…
- Turn business volume/latency/accuracy/cost into engineering budgets with a sizing pass.
- Explain why the classifier uses structured output and why a validation gate must fail closed.
- State the grounding rule and show the code branch that refuses when retrieval is empty.
- Walk the gate's safety→grounding→confidence ordering and why sensitive categories are gated harder.
- Show how the eval gate blocks a real regression, and where new golden-set rows come from.
- Give the cost/ticket and sync latency, and map each failure mode to the rail that hardens it.