Part IV · Chapter 5

Evaluation & Testing

LLM outputs are non-deterministic, so you can't test them with assertEqual alone. Evals are the CI of LLM systems: a repeatable way to know whether a prompt tweak or model swap made things better or worse. This chapter turns "it seems fine" into a number you can gate on.

⏱️ ~90 min🧪 5 labs🎯 Intermediate

⚙️ To run this for realThe code here is complete and correct as written. To actually execute it you'll need:

an Anthropic API key (ANTHROPIC_API_KEY) + pip install anthropic

Reading and learning works without any of this — run when you're ready. IDs/ARNs in examples are placeholders; swap in your own.

Learning objectives

Build a golden dataset and grow it from real failures.
Write deterministic checks and LLM-as-judge evals.
Write gradeable rubrics that produce stable scores.
Measure RAG retrieval and generation separately.
Wire a regression gate so quality can't silently drop on a change.

Why you can't ship without evals essential

Change a prompt to fix one bad case and you might break five others you never see. Swap to a cheaper model and quality might drop 8% — invisible until customers complain. Without evals you're flying blind, making changes on vibes. With them, every change gets a score, and regressions fail your build like any other test.

The shift in mindsetTraditional tests ask "is the output exactly X?" LLM evals ask "is the output good enough on these dimensions, often enough?" You measure aggregate quality over a dataset, not exact equality on one input.

The evaluation pyramid essential

🗺️ How to read this diagram

This picture answers one question: if I can't just check the answer is exactly right, how do I know an LLM is any good? The answer is layers of checks, drawn as a pyramid — cheap-and-automatic at the bottom, expensive-and-human at the top.

Read it bottom to top. The wide green base (Deterministic) is plain code checks — is it valid JSON? is a required word present? These are free and run on every commit, which is why the layer is widest.
Reference-based (blue) compares the output to a known-correct answer you wrote down in advance — good when there is a right answer to compare to.
LLM-as-judge (amber) uses a strong model to grade open-ended answers where no simple rule works ("is this helpful and on-topic?"). It scales, but costs money.
Human (red, the tiny tip) is the gold standard but slow, so you do it rarely, on small samples. The labels on the right — fast, every commit up to slow, gold standard — are the trade-off each layer makes.

In short: Build bottom-up. Most of your checks should be the cheap automatic kind at the base; save the expensive human review for the few things machines can't judge.

Cheap deterministic checks run on every commit; expensive human review runs rarely on samples. Build bottom-up.

Lab 5.1 · Build a golden set essential

A golden set is your test dataset: representative inputs, each with what a good output looks like. Start small (15–30 cases) and grow it from every production failure.

Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.

Lab 5.1

goldens.pyGOLDEN_SET = [
    {
        "id": "pw-reset",
        "input": "How do I reset my password?",
        "expected": "Settings > Security > Reset password; link emailed.",
        "expected_chunk": "faq.md#2",      # for RAG recall (Lab 5.4)
        "rubric": "Mentions Settings > Security and that a link is emailed.",
    },
    {
        "id": "refund-window",
        "input": "What is the refund window?",
        "expected": "30 days from purchase.",
        "expected_chunk": "policy.md#0",
        "rubric": "States the window is 30 days.",
    },
    # ... grow this from real misses
]

▶ How this works

A golden set is just a list of test cases you trust: some realistic inputs, and for each one, a note about what a good answer looks like. It's the ruler you'll measure every version of your system against. Here it's an ordinary Python list of dictionaries — no magic.

Each case is a { } dictionary with named fields. "id" is a short label so you can talk about a specific case (e.g. "pw-reset").
"input" is the question you'll feed the system; "expected" is roughly the right answer — what you'd accept as correct.
"expected_chunk" names the document piece that should be retrieved to answer this — used later in Lab 5.4 to grade retrieval on its own.
"rubric" is a plain-English pass/fail rule ("Mentions Settings > Security and that a link is emailed"). The LLM judge in Lab 5.3 grades against exactly this sentence, so make it specific and checkable.

What the output means: Nothing runs yet — this is just data. But it's the backbone of the whole chapter: every later lab loops over GOLDEN_SET to produce a score.

Try this: Start with 15–30 cases, not hundreds. And whenever the system flubs a real question in production, add that exact question here with the correct answer — the # grow this from real misses comment is the habit that makes the set valuable.

Grow it from incidentsEvery time the system gives a bad answer in production, add that input to the golden set with the correct expected output. Your eval set becomes a living record of "mistakes we must never make again."

Lab 5.2 · Deterministic checks intermediate

The fastest, most reliable evals need no model at all — they check hard properties. Run these on every commit.

Lab 5.2

eval_deterministic.pydef check_schema(output_obj) -> bool:
    return output_obj is not None            # parsed & validated?

def check_no_pii(text) -> bool:
    import re
    return not re.search(r"\b\d{3}-\d{2}-\d{4}\b", text)  # no SSNs leaked

def check_contains(text, must_have) -> bool:
    return must_have.lower() in text.lower()

def check_length(text, max_words=150) -> bool:
    return len(text.split()) <= max_words

▶ How this works

These are the cheapest evals of all: plain Python functions that return True or False. No AI model, no cost, no waiting — they check hard facts about an output, the kind of thing that is simply right or wrong. Run them on every commit.

check_schema returns True when the parsed output isn't None — i.e. the model gave back data in the shape you required (see the Pydantic pattern from Chapter 2), rather than garbage you can't use.
check_no_pii uses a regular expression (a text-pattern search) to look for a US Social Security number like 123-45-6789. re.search finds that pattern; not flips it, so the check passes only when no SSN is present — a safety guard against leaking private data.
check_contains asks "does the answer include this required word?", lower-casing both sides first so capitalization doesn't matter.
check_length counts words with text.split() and passes only if the answer is at most max_words — a simple guard against rambling.

What the output means: Each function is a yes/no gate. You'll call them across your golden set and turn the results into a percentage (e.g. "100% of outputs were valid JSON").

Try this: Use these for anything with a crisp right/wrong: valid JSON, a required field, an allowed category, a banned phrase, a length cap. They're instant and never flaky — lean on them before reaching for a model-based judge.

Use these for anything with a crisp right/wrong: valid JSON, required fields present, enum membership, no forbidden content, length limits, format rules. They're free and instant — lean on them hard.

Lab 5.3 · LLM-as-judge intermediate

For open-ended quality ("is this answer helpful and grounded?") no rule works — so use a strong model as a grader against an explicit rubric.

Lab 5.3

eval_judge.pyfrom anthropic import Anthropic
from pydantic import BaseModel
client = Anthropic()

class Judgment(BaseModel):
    passed: bool
    score: int          # 1-5
    reason: str

def judge(question, answer, rubric) -> Judgment:
    r = client.messages.parse(
        model="claude-opus-4-8", max_tokens=400,
        system=("You are a strict grader. Score the answer against the "
                "rubric only. Be specific about any failure."),
        messages=[{"role":"user","content":
            f"Question: {question}\nAnswer: {answer}\n"
            f"Rubric (must satisfy): {rubric}"}],
        output_format=Judgment,
    )
    return r.parsed_output

▶ How this works

Some qualities can't be checked with a rule — "is this answer actually helpful and grounded in the docs?" There's no if for that. So you hand the answer to a strong model acting as a grader and make it score against an explicit rubric. This is LLM-as-judge.

class Judgment(BaseModel) declares the exact shape the grade must come back in: passed (yes/no), score (a 1–5 number), and reason (why). Forcing structure means you get a clean verdict, not a paragraph you'd have to interpret.
The system message tells the grader its job: be a strict grader, score against the rubric only, and name any failure. Strictness keeps scores consistent run to run.
The user message stitches together the question, the answer being judged, and the rubric it must satisfy — everything the grader needs in one prompt.
client.messages.parse(..., output_format=Judgment) makes the model return data matching your Judgment shape; r.parsed_output hands you back a ready-to-use Python object with .passed, .score, .reason.

What the output means: For one answer you get a structured verdict, e.g. passed=True, score=4, reason="States the 30-day window" — a grade you can count and average.

Try this: Notice the judge is a big model (claude-opus-4-8) grading work that may come from a smaller one — you want your grader to be at least as capable as the thing it grades. For high-stakes evals, run three judges and take the majority vote.

Write gradeable rubrics"Is the answer good?" produces noisy, inconsistent scores. "States the refund window is 30 days" is checkable and stable. The judge scores each explicit criterion independently — vague rubric in, noisy score out. For high-stakes evals, run a panel of 3 judges and take the majority vote.

Lab 5.4 · Evaluate your RAG system intermediate

From Chapter 3: measure retrieval and generation separately, or you won't know which to fix.

Lab 5.4

Illustrative fragment — defines demo values / files are needed before this runs standalone.

eval_rag.pyfrom goldens import GOLDEN_SET
from rag import answer, store

def eval_retrieval(k=5):
    """Recall@k — did we fetch the answer-bearing chunk?"""
    hits = 0
    for case in GOLDEN_SET:
        ids = [c["id"] for c,_ in store.search(case["input"], k=k)]
        hits += case["expected_chunk"] in ids
    return hits / len(GOLDEN_SET)

def eval_generation():
    """Groundedness/relevance via judge."""
    passed = 0
    for case in GOLDEN_SET:
        ans, _ = answer(case["input"])
        j = judge(case["input"], ans, case["rubric"])
        passed += j.passed
    return passed / len(GOLDEN_SET)

print("recall@5:", eval_retrieval())
print("generation pass rate:", eval_generation())

recall@5: 0.90
generation pass rate: 0.85

▶ How this works

A RAG system has two jobs — find the right document, then write a good answer from it. If you only score the final answer, a low score can't tell you which job failed. This lab measures the two separately, so a bad number points straight at the fix.

eval_retrieval measures recall@k: for each case it searches the store and collects the ids of the top k chunks, then checks whether the case's expected_chunk is among them. hits += (... in ids) adds 1 when the right chunk was found.
It returns hits / len(GOLDEN_SET) — the fraction of cases where the answer-bearing chunk was retrieved. 0.90 means "we fetched the right source 90% of the time".
eval_generation measures answer quality: it generates a real answer with answer(...), then calls the judge from Lab 5.3 against that case's rubric, and counts j.passed.
It too returns a fraction — the pass rate of the judged answers. The two print lines report both numbers side by side.

What the output means: recall@5: 0.90 = the retriever found the right chunk 90% of the time; generation pass rate: 0.85 = the judge passed 85% of the written answers.

Try this: Read the two numbers together. Low recall → fix retrieval (chunking, reranking). High recall but low generation → the docs were found but the answer was weak, so fix the prompt. That's the whole point of splitting them.

Reading the two numbersLow recall + high generation-on-retrieved = fix retrieval (chunking, hybrid, rerank). High recall + low generation = fix the prompt (grounding rules, format). Separating them turns vague "it's bad" into a clear next action.

Lab 5.5 · The regression gate advanced

Tie it together: run the whole suite, compare to a saved baseline, and fail if any metric regresses. This is what runs in CI.

The gate turns "seems fine" into a hard check. CI runs the golden set on every change and compares to a committed baseline; a regression fails the build and blocks the deploy — the mechanism that lets you move fast without silently dropping quality (see also A8).

🗺️ How to read this diagram

This is the payoff of the whole chapter, drawn as a flow: it shows how your evals become an automatic quality gate that stands between a code change and shipping to users — the same role unit tests play in normal software.

Follow the arrows left to right. A change (a new prompt, model, or retrieval tweak) triggers run evals — the golden-set suite from the earlier labs.
The diamond is the decision: is pass_rate ≥ baseline − tol? — did quality stay at or above the committed baseline, allowing a tiny tolerance for normal noise?
The yes path (green) leads to deploy ✓ — quality held, so ship it. The no path (red) leads to block ✗ — a regression, so the build fails and nothing ships.
The caption at the bottom adds two more gates real systems use: safety evals (a hard fail, never negotiable) and cost per case (a cheaper answer isn't a win if quality dropped).

In short: This is CI for LLMs: every change is scored automatically, and a drop below baseline blocks the deploy. It's what lets you change prompts fast without silently getting worse.

Lab 5.5

Illustrative fragment — defines demo values / files are needed before this runs standalone.

run_evals.pyimport json, sys

BASELINE = {"recall@5": 0.85, "gen_pass": 0.80, "schema_ok": 1.0}
TOLERANCE = 0.02                     # allow tiny noise

def main():
    results = {
        "recall@5": eval_retrieval(),
        "gen_pass": eval_generation(),
        "schema_ok": eval_schema_rate(),
    }
    print(json.dumps(results, indent=2))

    regressed = [m for m,v in results.items()
                 if v < BASELINE[m] - TOLERANCE]
    if regressed:
        print("❌ REGRESSED:", regressed); sys.exit(1)   # fail the build
    print("✅ all metrics within tolerance")

if __name__ == "__main__":
    main()

▶ How this works

This is the gate from the diagram, in code — the script CI actually runs. It gathers every metric, compares each to a saved baseline, and exits with an error if anything slipped. An error exit is how CI knows to fail the pull request.

BASELINE is a dictionary of the scores you've already achieved and want to protect. TOLERANCE = 0.02 allows a tiny wobble, since LLM scores jitter a little between runs even with no real change.
results re-runs the actual evals (retrieval, generation, schema rate) to get this version's numbers, then prints them as readable JSON.
The regressed line is the heart of it: it keeps any metric m whose new value v dropped below its baseline minus the tolerance. If that list has anything in it, quality regressed.
if regressed: prints what broke and calls sys.exit(1) — a non-zero exit code, which tells CI "this build failed." Otherwise it prints the all-clear.

What the output means: On a healthy change: the metrics JSON, then ✅ all metrics within tolerance. On a regression: ❌ REGRESSED: ['gen_pass'] and the build stops.

Try this: When you intentionally improve a metric, raise its number in BASELINE and commit that — otherwise the new, higher quality becomes the floor future changes must clear. The baseline is a living record of "how good we are right now."

Run it in CI on any change to prompts, retrieval, or model. A prompt edit that drops gen_pass below baseline now fails the pull request — exactly like a broken unit test. When you intentionally improve a metric, update the baseline.

Offline then onlineEvals prove gains before you ship. After shipping, confirm with an online A/B test on real traffic and your feedback signal (Chapter 6). Offline evals catch regressions; online tells you if the win is real.

Common pitfalls advanced

Pitfall	Fix
Vibes-based "seems better"	Score against a golden set; make it a number
Vague rubrics → noisy judge scores	Write explicit, checkable criteria
Testing only the happy path	Add edge cases & every past failure to the golden set
Only measuring quality	Track cost & latency too — a 2% gain at 3× cost may not ship
Judge model = production model, graded on its own output	Prefer a strong independent judge; use a panel for high stakes
Evaluating RAG end-to-end only	Split retrieval vs generation metrics

Exercises advanced

Exercise 5.1 — Grow the golden set

Context: A golden set is only as good as the failures it remembers. Feeding real weak answers back into it is the habit that makes evals a living record of mistakes you must never repeat.

Your task: Find 5 inputs where your Chapter 3 RAG system gives a weak answer, add each to the golden set with the correct expected output and rubric, and re-run the suite.

Requirements:

At least 5 new cases drawn from actual weak answers, not invented ones
Each case has an input, an expected, and a checkable rubric
The suite is re-run over the enlarged set
The pass rate drops, revealing the real gaps you just captured

💡 Hint: Write the rubric as a specific, checkable sentence ("States the window is 30 days") so the Lab 5.3 judge can grade it consistently.

Exercise 5.2 — Judge stability

Context: An LLM judge's reliability lives in its rubric. The fastest way to feel that is to run the same judgment repeatedly and watch how rubric quality controls the variance.

Your task: Run the judge on the same (answer, rubric) pair 5 times and check whether you get the same verdict; then make the rubric vaguer and repeat.

Requirements:

The identical (answer, rubric) pair is judged 5 times
Verdict stability across the 5 runs is recorded
The rubric is then made deliberately vaguer and the runs repeated
You observe that a vague rubric yields noisier, less consistent scores

💡 Hint: "Is the answer good?" versus "States the refund window is 30 days" — the gap between their variances is the whole lesson.

Exercise 5.3 — Prove a change

Context: "It feels better" is not evidence. Toggling one retrieval knob and reading the recall number on both sides turns a tuning decision into a fact.

Your task: Turn hybrid search off in Chapter 3 and run eval_retrieval() to record recall; turn it back on and re-run, producing a number that proves whether hybrid helped on your corpus.

Requirements:

Recall is measured with hybrid search off, then on, on the same golden set
Only the hybrid toggle changes between the two runs
Both recall numbers are reported side by side
You state which setting wins on your corpus, backed by the numbers

💡 Hint: Reuse eval_retrieval() unchanged from Lab 5.4 — the only variable is the retriever config, so the comparison is clean.

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · Deterministic checks on the golden setBeginner

Context: The widest, cheapest layer of the eval pyramid is plain code: yes/no checks that run on every commit with no model and no cost. Turning a bunch of them into one number is the foundation everything else builds on.

Your task: Given the chapter's check_length and check_contains, write a pass_rate(cases, check) function that runs one check across a golden set and returns the fraction that pass.

Requirements:

pass_rate applies a single check function to every case
It returns a fraction (mean of the booleans), not a count
An empty case list returns 0.0 rather than dividing by zero
Demonstrated with both check_length and check_contains over a small golden list
Runs offline — the checks are pure Python, no API

💡 Hint: A boolean is 0/1 when summed, so the pass rate is just sum(...)/len(cases) — guard the empty-list case first.

Show solution

Deterministic checks are cheap yes/no gates; a pass rate is just the mean of the booleans.

def check_contains(text, must_have):
    return must_have.lower() in text.lower()

def check_length(text, max_words=150):
    return len(text.split()) <= max_words

def pass_rate(cases, check):
    if not cases:
        return 0.0
    hits = sum(1 for c in cases if check(c))
    return hits / len(cases)

GOLDEN = [
    {"answer": "Reset it in Settings > Security."},
    {"answer": "The refund window is 30 days."},
    {"answer": "word " * 200},
]
print(round(pass_rate(GOLDEN, lambda c: check_length(c["answer"])), 2))    # 0.67
print(round(pass_rate(GOLDEN, lambda c: check_contains(c["answer"], "30 days")), 2))  # 0.33

Exercise 2 · Recall@k + generation, read togetherIntermediate

Context: A RAG system has two jobs — find the chunk, then write the answer. Scoring only the final answer can't tell you which job failed; reading recall@k and generation together names the fix.

Your task: Combine the two Lab 5.4 metrics: compute recall@k from retrieved chunk ids and a generation pass rate, then return the one-line diagnosis the chapter prescribes (fix retrieval vs fix the prompt).

Requirements:

recall@k counts cases whose expected_chunk is in the retrieved ids
gen_pass is the fraction of cases the judge passed
Low recall → diagnosis says fix RETRIEVAL (chunking/hybrid/rerank)
High recall but low generation → diagnosis says fix the PROMPT
Both healthy → a passing diagnosis; all runnable on plain dict cases

💡 Hint: Check the recall floor first, then the generation floor — the order encodes the callout's decision tree exactly.

Show solution

The whole point of splitting the metrics is that the pair of numbers names the fix.

def recall_at_k(cases):
    hits = sum(1 for c in cases if c["expected_chunk"] in c["retrieved_ids"])
    return hits / len(cases)

def gen_pass(cases):
    return sum(1 for c in cases if c["judge_passed"]) / len(cases)

def diagnose(cases, k_recall_floor=0.85, gen_floor=0.80):
    r, g = recall_at_k(cases), gen_pass(cases)
    if r < k_recall_floor:
        return f"recall {r:.2f} low -> fix RETRIEVAL (chunking/hybrid/rerank)"
    if g < gen_floor:
        return f"recall {r:.2f} ok but gen {g:.2f} low -> fix the PROMPT"
    return f"recall {r:.2f}, gen {g:.2f} -> both healthy"

cases = [
    {"expected_chunk": "faq.md#2", "retrieved_ids": ["faq.md#2"], "judge_passed": True},
    {"expected_chunk": "policy.md#0", "retrieved_ids": ["policy.md#0"], "judge_passed": False},
    {"expected_chunk": "faq.md#9", "retrieved_ids": ["faq.md#9"], "judge_passed": False},
]
print(diagnose(cases))  # recall 1.00 ok but gen 0.33 low -> fix the PROMPT

Exercise 3 · A 3-judge panel with majority voteAdvanced

Context: A single LLM judge is noisy. For high-stakes evals the chapter recommends a panel of three and a majority vote — and the cases where the judges disagree are themselves a signal worth surfacing.

Your task: Simulate a 3-judge panel: given a list of per-judge passed booleans per case, return the majority verdict and flag 2-1 splits as low-confidence.

Requirements:

panel_verdict passes a case when a majority of the votes are True
It flags a case as confident only when the judges are unanimous
eval_panel returns the overall pass rate plus the indices of contested cases
A 2-1 split is reported as contested (low-confidence), not silently averaged
Runs offline over hard-coded vote lists

💡 Hint: Count the True votes once: majority is yes > len(votes)/2, and unanimity is yes == 0 or yes == len(votes).

Show solution

Majority vote reduces single-judge noise; disagreement is itself a signal worth surfacing.

def panel_verdict(judge_votes):
    """judge_votes: list[bool] from 3 judges for ONE case."""
    yes = sum(1 for v in judge_votes if v)
    passed = yes > len(judge_votes) / 2
    unanimous = yes == 0 or yes == len(judge_votes)
    return {"passed": passed, "confident": unanimous, "yes": yes}

def eval_panel(cases):
    verdicts = [panel_verdict(c["votes"]) for c in cases]
    pass_rate = sum(1 for v in verdicts if v["passed"]) / len(verdicts)
    contested = [i for i, v in enumerate(verdicts) if not v["confident"]]
    return pass_rate, contested

cases = [
    {"votes": [True, True, True]},    # unanimous pass
    {"votes": [True, False, True]},   # 2-1 pass, contested
    {"votes": [False, False, True]},  # 2-1 fail, contested
]
rate, contested = eval_panel(cases)
print(round(rate, 2), contested)  # 0.67 [1, 2]

Exercise 4 · The regression gate — correctness of the comparisonExpert

Context: The regression gate is the payoff of the chapter — CI for LLMs. The subtlety is all in the comparison: an improvement must never trip the gate, tolerance absorbs noise, and safety metrics bypass tolerance entirely.

Your task: Reimplement Lab 5.5's gate carefully: a metric regresses only if it drops below baseline minus tolerance, safety metrics are a hard fail with no tolerance, and the gate returns an exit code (0 pass, 1 regressed) plus the offender list.

Requirements:

Each metric is compared against baseline[m] - tolerance
Metrics in a HARD_FAIL set use zero tolerance
An improved metric (above baseline) never appears in the offenders
Returns 1 with the offender list when anything regressed, else 0 with an empty list
Demonstrated on a noise-absorbed pass and on a real-regression + safety-drop fail

💡 Hint: Compute the per-metric floor as base - (0.0 if m in HARD_FAIL else tol), then collect every metric whose value falls below its floor.

Show solution

The subtlety: an improvement must never trip the gate, tolerance absorbs noise, and safety metrics bypass tolerance entirely.

BASELINE = {"recall@5": 0.85, "gen_pass": 0.80, "schema_ok": 1.0, "safety": 1.0}
TOLERANCE = 0.02
HARD_FAIL = {"safety", "schema_ok"}   # never allow ANY drop

def gate(results, baseline=BASELINE, tol=TOLERANCE):
    regressed = []
    for m, v in results.items():
        floor = baseline[m] - (0.0 if m in HARD_FAIL else tol)
        if v < floor:
            regressed.append(m)
    return (1 if regressed else 0), regressed

# noise on gen_pass is absorbed; an improvement never trips it
print(gate({"recall@5": 0.85, "gen_pass": 0.79, "schema_ok": 1.0, "safety": 1.0}))  # (0, [])
# a real regression AND a hard-fail safety drop
print(gate({"recall@5": 0.70, "gen_pass": 0.90, "schema_ok": 1.0, "safety": 0.99}))
# -> (1, ['recall@5', 'safety'])

Exercise 5 · Track cost & latency alongside qualityProfessional

Context: Quality is necessary but not sufficient: a 2% quality gain at 3× cost may not ship. Production gates treat cost and latency as first-class metrics, and a failed build must explain itself in the CI log.

Your task: Extend the gate to also veto a change whose cost_per_case or p95_latency_ms regresses beyond budget, and print a per-metric report so the block is self-explaining.

Requirements:

Higher-is-better metrics (quality) use a floor; lower-is-better metrics (cost, latency) use a ceiling
Each metric carries its own tolerance (e.g. 10% latency headroom, zero cost slack)
A per-metric OK/FAIL report line is printed for every metric
A quality-up-but-cost-3× change is BLOCKED, naming cost_per_case as the offender
The direction of the comparison is chosen by which metric-set the name is in

💡 Hint: Split the check by direction: new <= base*(1+tol) for lower-is-better, new >= base-tol for higher-is-better; keep a set of the lower-is-better names.

Show solution

Quality is necessary but not sufficient — production gates cost and latency as first-class metrics, and the report makes a failed build self-explaining.

BASELINE = {"gen_pass": 0.80, "cost_per_case": 0.010, "p95_latency_ms": 1500}
# higher-is-better vs lower-is-better matters for the direction of the check
LOWER_IS_BETTER = {"cost_per_case", "p95_latency_ms"}
TOL = {"gen_pass": 0.02, "cost_per_case": 0.0, "p95_latency_ms": 0.10}  # 10% latency headroom

def check_metric(name, new, base):
    if name in LOWER_IS_BETTER:
        ceiling = base * (1 + TOL[name])
        return new <= ceiling, f"{name}: {new} (ceiling {ceiling:.4f})"
    floor = base - TOL[name]
    return new >= floor, f"{name}: {new} (floor {floor:.4f})"

def gate(results):
    offenders, report = [], []
    for name, new in results.items():
        ok, line = check_metric(name, new, BASELINE[name])
        report.append(("OK  " if ok else "FAIL") + " " + line)
        if not ok:
            offenders.append(name)
    return offenders, report

offenders, report = gate({"gen_pass": 0.83, "cost_per_case": 0.030, "p95_latency_ms": 1550})
print("\n".join(report))
print("BLOCK" if offenders else "SHIP", offenders)  # quality up but 3x cost -> BLOCK ['cost_per_case']

Exercise 6 · Your team ships a model swap under an eval SLAIndustry scenario

Context: Your team wants to swap the production model to cut cost, under a hard constraint: quality must not regress and the swap must be revertible in seconds. A model swap changes behavior and invalidates prompt caches, so it is a risky change.

Your task: Design the offline+online eval flow (golden-set gate → canary → instant rollback) and write the CI gate authorize_canary(control, candidate) that decides whether the swap even reaches the canary.

Requirements:

Offline first: run the full golden-set suite (deterministic + recall@k + judge panel) on the candidate model
The gate hard-fails on any safety drop
It blocks if gen_pass regresses beyond a small margin vs the control
It blocks if there is no cost win (candidate cost must drop)
The model id is a flag so rollback is a seconds-long flip, not a redeploy
The gate logic runs offline over returned metric dicts; only the real suite and canary need a key

💡 Hint: Accumulate failure reasons into a list and pass only when it is empty — the same offenders-list pattern as the plain regression gate, applied to a control-vs-candidate comparison.

Show solution

Design. A model swap changes behavior and invalidates prompt caches (Ch 6), so treat it as a risky change:

Offline first (Ch 5): run the full golden-set suite (deterministic + recall@k + LLM-judge panel) on the candidate model. Gate on gen_pass not regressing, safety hard-fail, and cost_per_case within budget.
Canary online (Ch 6): if the offline gate passes, route a small % of traffic to the candidate behind a feature flag; compare live quality-feedback, p95 latency, and cost vs the control arm.
Instant rollback: the model id is a flag, so a dip flips it back in seconds — no redeploy. Re-baseline (token counts, prompt tuning) only after the win is confirmed.

The offline gate that authorizes the canary:

def eval_suite(model):
    # in real life these call the golden-set harness against `model`;
    # returned here as a dict so the gate logic is testable offline.
    return {"claude-opus-4-8":  {"gen_pass": 0.86, "safety": 1.0, "cost_per_case": 0.012},
            "claude-haiku-4-5": {"gen_pass": 0.85, "safety": 1.0, "cost_per_case": 0.003}}[model]

def authorize_canary(control, candidate, min_gen=0.02, cost_must_drop=True):
    c0, c1 = eval_suite(control), eval_suite(candidate)
    reasons = []
    if c1["safety"] < 1.0:
        reasons.append("safety hard-fail")
    if c1["gen_pass"] < c0["gen_pass"] - min_gen:
        reasons.append(f"quality regressed {c0['gen_pass']}->{c1['gen_pass']}")
    if cost_must_drop and c1["cost_per_case"] >= c0["cost_per_case"]:
        reasons.append("no cost win")
    return (not reasons), reasons

ok, reasons = authorize_canary("claude-opus-4-8", "claude-haiku-4-5")
print("CANARY" if ok else "BLOCK", reasons)  # quality held within 0.02, cost dropped -> CANARY

This is offline gating logic and runs as-is. The real eval_suite and the online canary comparison need an API key and live traffic.

✓ Checkpoint — you can move on when you can…

Explain the eval pyramid and what runs at each level.
Build a golden set and grow it from failures.
Write deterministic checks and an LLM-judge with a gradeable rubric.
Measure RAG retrieval and generation separately and act on each number.
Wire a regression gate that fails CI when a metric drops.

🏗️ Toward the capstoneEvals are how the DevOps agent earns the right to act. It can't graduate from "diagnose" to "execute a rollback" until evals prove its diagnoses are correct on real past incidents. And the most important eval in the whole system is the capstone's hard-fail safety check: "did the agent ever attempt an action above its permission rung?" — the regression test that keeps it from ever touching prod uninvited. See infra evals in the capstone →

Knowledge check check yourself

✓ Knowledge check

A RAG eval reports recall@5 = 0.90 but generation pass rate = 0.60. What does that combination tell you to fix, and why is measuring the two metrics separately the whole point?

Show answer

High recall with low generation means the right chunks were retrieved but the answer was weak — so fix the prompt (grounding rules, format), not retrieval. Splitting the metrics turns a vague 'it's bad' into a clear next action: low recall points to chunking/hybrid/rerank, low generation points to the prompt.

✓ Knowledge check

The chapter warns against using your production model as its own judge, graded on its own output. Why is that a problem, and what does it recommend for high-stakes evals?

Show answer

A model grading its own output can be biased toward its own answers, giving an unreliable score. The lesson recommends a strong, independent judge at least as capable as the thing it grades, and for high-stakes evals a panel of three judges with a majority vote to reduce noise.

Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.

Learning objectives

Why you can't ship without evals essential

The evaluation pyramid essential

Lab 5.1 · Build a golden set essential

Lab 5.2 · Deterministic checks intermediate

Lab 5.3 · LLM-as-judge intermediate

Lab 5.4 · Evaluate your RAG system intermediate

Lab 5.5 · The regression gate advanced

Common pitfalls advanced

Exercises advanced

Exercise 5.1 — Grow the golden set

Exercise 5.2 — Judge stability

Exercise 5.3 — Prove a change

🪜 Practice ladder beginner → industry

✓ Checkpoint — you can move on when you can…

Knowledge check check yourself

More in Core curriculum