Evaluation & Testing

Why you can't ship without evals essential

Here's the pattern that humbles almost everyone who builds with LLMs. You wire up a demo, throw it a handful of questions, and it dazzles — fluent, confident, right on the money. You show it around, everyone nods, you ship. Then real users arrive with the phrasings you never imagined, and the same "working" system starts hedging, hallucinating, and contradicting itself. Nothing in your code changed; the inputs did. That gap — between a demo that impresses and a product that holds up — is exactly the gap evals exist to close. The industry rule of thumb is blunt: most demos work, most products fail, and the difference is usually whether anyone bothered to measure.

The reason ordinary tests don't rescue you here is that LLM output is non-deterministic. Ask the same question twice and you can get two differently-worded answers, both perfectly acceptable. So assertEqual(output, "30 days") is useless — the model might reply "You have thirty days" and be entirely correct while your test screams failure. You're no longer checking for one exact string; you're asking a fuzzier, more honest question: is this good enough, on the dimensions I care about, often enough?

Think of an eval harness as the dashboard on a car. Without instruments you can still drive — for a while, on a clear road, by feel. But the moment conditions change you have no idea you're overheating or nearly out of fuel until the engine dies on the highway. The speedometer doesn't make the car faster; it makes your decisions informed. Evals are the same: they don't improve your prompt, they tell you — before your users do — whether the last change helped or quietly broke something.

The common mistake is "vibes-based development": you tweak a prompt to fix one annoying case, eyeball two or three outputs, decide it "seems better," and move on. What you can't see is the five other cases that tweak just broke. Swap to a cheaper model to save money and quality might sag 8% across the board — invisible in a spot-check, painfully visible in a support queue three weeks later. Without a scored dataset, every change is a coin flip you're not even watching land. With one, every change produces a number, and a regression fails your build like any other broken test. That is the whole promise of this chapter: turning "it seems fine" into a measurement you can gate a deploy on.

The evaluation pyramid essential

The instinct of every beginner is to skip straight to the fanciest check. "I'll just have GPT grade everything!" — and now every commit costs money, takes minutes, and gives slightly different scores each run. The mistake isn't using an LLM judge; it's using it for things a single line of code could have decided for free. The evaluation pyramid is the map that stops you from making that mistake, and it borrows its shape from an idea you may already know from software testing.

If you've seen the classic test pyramid — lots of fast unit tests at the base, fewer integration tests in the middle, a handful of slow end-to-end tests at the top — this is the same wisdom pointed at LLM output. The principle is identical: the cheapest check that can catch a given bug should be the one that catches it. You don't hire a human reviewer to notice the JSON didn't parse; a two-line function does that instantly. You save the expensive, scarce judgment for the questions only expensive, scarce judgment can answer.

Read the four layers as a ladder of cost and coverage. At the wide base sit deterministic checks — plain code asking crisp yes/no questions (valid schema? banned word absent? under the length cap?). They cost nothing, never flake, and run on every commit, so they're the widest layer. Above them, reference-based checks compare the output to a known-correct answer you wrote down in advance — useful whenever there genuinely is a right answer. Higher still, LLM-as-judge grades open-ended quality where no rule fits ("is this helpful and grounded?"); it scales to thousands of cases but costs tokens and jitters slightly. At the narrow tip is the human — the gold standard, and also the slowest and priciest, so you spend it rarely, on small samples, on the things machines still can't judge.

The direction that matters is bottom-up. Build the base first and lean on it hard; only climb a layer when the one below genuinely can't answer your question. Teams that invert the pyramid — mostly human review, a little automation — end up with evals so slow and expensive that nobody runs them, which is the same as having none. A fat, fast base is what lets you check quality on every change instead of once a quarter.

Cheap deterministic checks run on every commit; expensive human review runs rarely on samples. Build bottom-up.

Lab 5.1 · Build a golden set essential

Every eval number you'll ever report is only as trustworthy as the dataset behind it. You can build the most elegant judge and the slickest CI gate in the world, but if you measure them against ten questions you happened to think of on a Tuesday afternoon, your "90% pass rate" means almost nothing — it measures how well you do on your imagination, not on reality. The golden set is the foundation the whole harness stands on, which is why it comes first.

A golden set is simply your test dataset: a list of representative inputs, each paired with a note about what a good output looks like. Think of it as the answer key a teacher writes before an exam. The teacher doesn't grade forty papers by re-deciding what "correct" means each time; she settles the answers once, in advance, then grades consistently against them. Your golden set does the same job — it fixes "what good looks like" once, so every future run is scored against the same ruler instead of your shifting mood.

The mistake that quietly poisons everything is an unrepresentative set. If your cases are all the tidy, well-phrased questions you'd ask yourself, your evals will glow green while real users — who type half-sentences, typos, and edge cases you never dreamed of — hit the failures your dataset can't see. A representative set deliberately includes the awkward phrasings, the ambiguous asks, the out-of-scope questions, and above all the specific cases your system has already gotten wrong. That last source is gold: a golden set grown from real production misses becomes a living record of "mistakes we must never make again," and it's the single highest-leverage habit in this chapter.

One more piece of hard-won advice on size: start small and start now. Fifteen to thirty carefully chosen cases beat a thousand hastily scraped ones, because every case is something you'll look at, debug against, and trust. A giant set you never curate is just noise with a large denominator. Grow it deliberately, one real failure at a time, and it stays sharp.

goldens.pyGOLDEN_SET = [
    {
        "id": "pw-reset",
        "input": "How do I reset my password?",
        "expected": "Settings > Security > Reset password; link emailed.",
        "expected_chunk": "faq.md#2",      # for RAG recall (Lab 5.4)
        "rubric": "Mentions Settings > Security and that a link is emailed.",
    },
    {
        "id": "refund-window",
        "input": "What is the refund window?",
        "expected": "30 days from purchase.",
        "expected_chunk": "policy.md#0",
        "rubric": "States the window is 30 days.",
    },
    # ... grow this from real misses
]

Lab 5.2 · Deterministic checks intermediate

Before you spend a cent on an AI grader, ask a question that costs nothing: is the answer even the right shape? A shocking fraction of "the LLM gave a bad answer" bugs aren't subtle quality problems at all — they're the model returning malformed JSON, forgetting a required field, blowing past a length cap, or leaking a phone number it was told to redact. None of that needs judgment to catch. It needs a plain if. Deterministic checks are that base layer of the pyramid, and they earn their keep on every single commit.

A deterministic check is just a function that looks at an output and returns True or False for a property that is objectively right or wrong. Think of them as the metal detector at the airport: fast, automatic, unglamorous, and non-negotiable. The detector doesn't decide whether you're a nice person — that's a judgment for later — it only answers one crisp question ("carrying metal? yes/no") and answers it the same way every time, for everyone, in a fraction of a second. That determinism is the whole point: run the check a thousand times and you get the identical verdict, so it can never flake and never surprise you at 2am.

These checks split naturally into two jobs. Some verify structure — did the output parse into the schema you demanded, is the required field present, is it under the word limit? Others verify safety and policy — no Social Security numbers, no banned phrases, no disallowed category. Both kinds share the same virtue: they're free, instant, and utterly reliable, so there's no reason not to run all of them on every change.

The mistake to avoid runs in both directions. One camp reaches for an LLM judge to check things a regex could settle — burning money and adding noise to answer "is this valid JSON?" The other camp overreaches, trying to cram fuzzy quality into a rule ("the answer must contain the word 'sorry' to count as empathetic") and ends up with a brittle check that fails good answers and passes bad ones. The discipline is knowing the boundary: if a property is crisply true or false, a deterministic check owns it; the moment it requires taste or context, that's the judge's job (next lab). Lean on this layer as hard as you can — everything it catches is a case you never have to pay a model to look at.

eval_deterministic.pydef check_schema(output_obj) -> bool:
    return output_obj is not None            # parsed & validated?

def check_no_pii(text) -> bool:
    import re
    return not re.search(r"\b\d{3}-\d{2}-\d{4}\b", text)  # no SSNs leaked

def check_contains(text, must_have) -> bool:
    return must_have.lower() in text.lower()

def check_length(text, max_words=150) -> bool:
    return len(text.split()) <= max_words

Use these for anything with a crisp right/wrong: valid JSON, required fields present, enum membership, no forbidden content, length limits, format rules. They're free and instant — lean on them hard.

Lab 5.3 · LLM-as-judge intermediate

Sooner or later you hit a question no rule can answer. "Is this support reply actually helpful?" "Is this summary faithful to the source, or did it invent a detail?" "Does this answer stay grounded in the retrieved docs?" There's no regex for helpful, no if for faithful. You could read every output by hand — and for ten cases you should — but that doesn't scale to the thousands of cases a real eval set needs. This is the exact problem LLM-as-judge solves: use a strong model to grade open-ended quality, cheaply and at scale, against a rubric you write down.

The mental model is a grading rubric handed to a teaching assistant. A good professor doesn't tell the TA "grade these however you feel"; she hands over an explicit key — "1 point for stating the 30-day window, 1 point for citing the policy, 0 if it invents a number" — so that whoever grades, and whenever they grade, the scores land in the same place. The LLM is your tireless TA: it will read a thousand answers without getting bored, but it's only as consistent as the rubric you give it. A crisp rubric produces crisp, repeatable grades; a vague one produces mood-lighting.

Be honest about the biases, because they're real and they'll bite you. LLM judges are known to favor longer, more verbose answers over tight correct ones; they can prefer the first option shown in a comparison (position bias); and — the trap that catches everyone — a model asked to grade its own output tends to like what it produced. So the cardinal rule is independence: your judge should be a separate, strong model, at least as capable as the thing it's grading. Never let your production model be sole judge of its own answers, and for anything high-stakes, don't trust a single verdict at all.

The fix for a noisy judge is the same one democracies use for noisy individuals: vote. Run a panel of three judges and take the majority, and the random jitter of any one grader washes out — while the cases where the three disagree become a valuable signal, flagging the genuinely ambiguous answers that deserve a human's eye. The single most important thing you control, though, isn't the panel size; it's the rubric. Write it as a specific, checkable sentence — "States the refund window is 30 days" beats "Is the answer good?" every time — because vague rubric in means noisy score out.

eval_judge.pyfrom anthropic import Anthropic
from pydantic import BaseModel
client = Anthropic()

class Judgment(BaseModel):
    passed: bool
    score: int          # 1-5
    reason: str

def judge(question, answer, rubric) -> Judgment:
    r = client.messages.parse(
        model="claude-opus-4-8", max_tokens=400,
        system=("You are a strict grader. Score the answer against the "
                "rubric only. Be specific about any failure."),
        messages=[{"role":"user","content":
            f"Question: {question}\nAnswer: {answer}\n"
            f"Rubric (must satisfy): {rubric}"}],
        output_format=Judgment,
    )
    return r.parsed_output

eval_judge.pyfrom openai import OpenAI
from pydantic import BaseModel
client = OpenAI()

class Judgment(BaseModel):
    passed: bool
    score: int          # 1-5
    reason: str

def judge(question, answer, rubric) -> Judgment:
    r = client.responses.parse(
        model="gpt-5.5", max_output_tokens=400,
        instructions=("You are a strict grader. Score the answer against the "
                      "rubric only. Be specific about any failure."),
        input=(
            f"Question: {question}\nAnswer: {answer}\n"
            f"Rubric (must satisfy): {rubric}"),
        text_format=Judgment,
    )
    return r.output_parsed

Lab 5.4 · Evaluate your RAG system intermediate

Picture the most frustrating kind of bug report: "the answers are just... bad." Bad how? Your RAG system does two entirely different jobs in sequence — first it finds the relevant chunk of your documents, then it writes an answer from what it found. A single end-to-end score smears those two jobs together into one number, and when that number is low you're left guessing which half broke. Guessing is expensive. This lab's whole idea is to refuse to guess: measure the two stages separately, so a bad score points a finger at exactly the thing to fix.

The analogy that makes this click is a restaurant kitchen. A dish can arrive disappointing for two completely different reasons: the shopper bought the wrong ingredients, or the chef cooked good ingredients badly. If you only taste the final plate you can't tell which happened — and you might retrain the chef when the real problem was the shopping list. RAG is the same two-stage pipeline: retrieval is the shopper (did it fetch the right source?) and generation is the chef (did it cook a good answer from that source?). Score each station on its own and the diagnosis is instant.

The two metrics that do this are worth naming precisely. Recall@k asks the retrieval question: of the top k chunks we fetched, was the answer-bearing one among them? It's a fraction — 0.90 means we grabbed the right source 90% of the time — and crucially it says nothing about the wording of the final answer. The generation pass rate asks the writing question: given the answer the system produced, did the judge (from Lab 5.3) rule it acceptable against the rubric? One measures the shopper, the other the chef.

Now the payoff, and the mistake it prevents. Read the two numbers together, not separately. High recall but low generation means the right documents were found and the answer was still weak — so the retriever is fine and the fix is in the prompt (grounding rules, format, instructions to actually use the context). Low recall means the answer never had a chance because the source wasn't even fetched — so leave the prompt alone and fix retrieval (chunking, hybrid search, reranking). The beginner error is to see a bad answer and immediately rewrite the prompt, when half the time the prompt was innocent and the retriever was starving it. Splitting the metrics turns a vague "it's bad" into a precise, one-line next action — that is the entire reason this lab exists.

eval_rag.pyfrom goldens import GOLDEN_SET
from rag import answer, store

def eval_retrieval(k=5):
    """Recall@k — did we fetch the answer-bearing chunk?"""
    hits = 0
    for case in GOLDEN_SET:
        ids = [c["id"] for c,_ in store.search(case["input"], k=k)]
        hits += case["expected_chunk"] in ids
    return hits / len(GOLDEN_SET)

def eval_generation():
    """Groundedness/relevance via judge."""
    passed = 0
    for case in GOLDEN_SET:
        ans, _ = answer(case["input"])
        j = judge(case["input"], ans, case["rubric"])
        passed += j.passed
    return passed / len(GOLDEN_SET)

print("recall@5:", eval_retrieval())
print("generation pass rate:", eval_generation())

recall@5: 0.90
generation pass rate: 0.85

Lab 5.5 · The regression gate advanced

Here's the uncomfortable truth about evals nobody runs: they're worthless. You can build a beautiful golden set, sharp deterministic checks, and a well-calibrated judge panel — and if firing them off is a manual chore someone does "when they remember," quality will still rot. People forget. Deadlines loom. The one prompt tweak that quietly drops your pass rate 6% ships on a Friday because no one thought to re-run the suite. The regression gate is the fix: it takes everything you've built and makes it automatic and unskippable, wiring your evals into CI so a change literally cannot ship if quality drops.

This is the moment your evals stop being a report and become a gate — and the analogy is exactly the unit-test suite you already trust in normal software. When a code change breaks a unit test, the pull request goes red and the merge button locks. You don't rely on the developer to "remember to check"; the machine refuses. The regression gate gives LLM systems that same reflex: every change to a prompt, a retrieval setting, or a model runs the golden set, and a drop below your committed baseline fails the build like any other broken test.

Two design details make this work in practice, and both come from the fact that LLM scores wobble. First, a baseline: a committed record of the scores you've already achieved and now refuse to fall below — the high-water mark future changes must clear. Second, a tolerance: a small slack (a couple of percent) that absorbs the natural run-to-run jitter of model outputs, so you fail on real regressions, not on noise. Set tolerance too tight and the gate cries wolf on every commit; too loose and a genuine slide slips through. And the direction matters — when you intentionally raise a metric, you must bump the baseline to lock in the win, or the old, lower bar keeps letting quality drift back down.

The mistake here is gating on quality alone. A change that lifts your pass rate 2% but triples the cost per case, or doubles latency, is often not a win at all — and a version that's a hair cheaper but leaks PII is a catastrophe, not a trade-off. So real gates carry more than a quality floor: cost and latency get ceilings, and safety checks get a hard fail with zero tolerance — no amount of quality buys back a safety regression. Get this right and you've built the thing that lets a team move fast without fear: change boldly, ship often, and trust that the gate will catch you before your users do.

run_evals.pyimport json, sys

BASELINE = {"recall@5": 0.85, "gen_pass": 0.80, "schema_ok": 1.0}
TOLERANCE = 0.02                     # allow tiny noise

def main():
    results = {
        "recall@5": eval_retrieval(),
        "gen_pass": eval_generation(),
        "schema_ok": eval_schema_rate(),
    }
    print(json.dumps(results, indent=2))

    regressed = [m for m,v in results.items()
                 if v < BASELINE[m] - TOLERANCE]
    if regressed:
        print("❌ REGRESSED:", regressed); sys.exit(1)   # fail the build
    print("✅ all metrics within tolerance")

if __name__ == "__main__":
    main()

Run it in CI on any change to prompts, retrieval, or model. A prompt edit that drops gen_pass below baseline now fails the pull request — exactly like a broken unit test. When you intentionally improve a metric, update the baseline.

Common pitfalls advanced

Exercises advanced

def check_contains(text, must_have):
    return must_have.lower() in text.lower()

def check_length(text, max_words=150):
    return len(text.split()) <= max_words

def pass_rate(cases, check):
    if not cases:
        return 0.0
    hits = sum(1 for c in cases if check(c))
    return hits / len(cases)

GOLDEN = [
    {"answer": "Reset it in Settings > Security."},
    {"answer": "The refund window is 30 days."},
    {"answer": "word " * 200},
]
print(round(pass_rate(GOLDEN, lambda c: check_length(c["answer"])), 2))    # 0.67
print(round(pass_rate(GOLDEN, lambda c: check_contains(c["answer"], "30 days")), 2))  # 0.33

def recall_at_k(cases):
    hits = sum(1 for c in cases if c["expected_chunk"] in c["retrieved_ids"])
    return hits / len(cases)

def gen_pass(cases):
    return sum(1 for c in cases if c["judge_passed"]) / len(cases)

def diagnose(cases, k_recall_floor=0.85, gen_floor=0.80):
    r, g = recall_at_k(cases), gen_pass(cases)
    if r < k_recall_floor:
        return f"recall {r:.2f} low -> fix RETRIEVAL (chunking/hybrid/rerank)"
    if g < gen_floor:
        return f"recall {r:.2f} ok but gen {g:.2f} low -> fix the PROMPT"
    return f"recall {r:.2f}, gen {g:.2f} -> both healthy"

cases = [
    {"expected_chunk": "faq.md#2", "retrieved_ids": ["faq.md#2"], "judge_passed": True},
    {"expected_chunk": "policy.md#0", "retrieved_ids": ["policy.md#0"], "judge_passed": False},
    {"expected_chunk": "faq.md#9", "retrieved_ids": ["faq.md#9"], "judge_passed": False},
]
print(diagnose(cases))  # recall 1.00 ok but gen 0.33 low -> fix the PROMPT

def panel_verdict(judge_votes):
    """judge_votes: list[bool] from 3 judges for ONE case."""
    yes = sum(1 for v in judge_votes if v)
    passed = yes > len(judge_votes) / 2
    unanimous = yes == 0 or yes == len(judge_votes)
    return {"passed": passed, "confident": unanimous, "yes": yes}

def eval_panel(cases):
    verdicts = [panel_verdict(c["votes"]) for c in cases]
    pass_rate = sum(1 for v in verdicts if v["passed"]) / len(verdicts)
    contested = [i for i, v in enumerate(verdicts) if not v["confident"]]
    return pass_rate, contested

cases = [
    {"votes": [True, True, True]},    # unanimous pass
    {"votes": [True, False, True]},   # 2-1 pass, contested
    {"votes": [False, False, True]},  # 2-1 fail, contested
]
rate, contested = eval_panel(cases)
print(round(rate, 2), contested)  # 0.67 [1, 2]

BASELINE = {"recall@5": 0.85, "gen_pass": 0.80, "schema_ok": 1.0, "safety": 1.0}
TOLERANCE = 0.02
HARD_FAIL = {"safety", "schema_ok"}   # never allow ANY drop

def gate(results, baseline=BASELINE, tol=TOLERANCE):
    regressed = []
    for m, v in results.items():
        floor = baseline[m] - (0.0 if m in HARD_FAIL else tol)
        if v < floor:
            regressed.append(m)
    return (1 if regressed else 0), regressed

# noise on gen_pass is absorbed; an improvement never trips it
print(gate({"recall@5": 0.85, "gen_pass": 0.79, "schema_ok": 1.0, "safety": 1.0}))  # (0, [])
# a real regression AND a hard-fail safety drop
print(gate({"recall@5": 0.70, "gen_pass": 0.90, "schema_ok": 1.0, "safety": 0.99}))
# -> (1, ['recall@5', 'safety'])

BASELINE = {"gen_pass": 0.80, "cost_per_case": 0.010, "p95_latency_ms": 1500}
# higher-is-better vs lower-is-better matters for the direction of the check
LOWER_IS_BETTER = {"cost_per_case", "p95_latency_ms"}
TOL = {"gen_pass": 0.02, "cost_per_case": 0.0, "p95_latency_ms": 0.10}  # 10% latency headroom

def check_metric(name, new, base):
    if name in LOWER_IS_BETTER:
        ceiling = base * (1 + TOL[name])
        return new <= ceiling, f"{name}: {new} (ceiling {ceiling:.4f})"
    floor = base - TOL[name]
    return new >= floor, f"{name}: {new} (floor {floor:.4f})"

def gate(results):
    offenders, report = [], []
    for name, new in results.items():
        ok, line = check_metric(name, new, BASELINE[name])
        report.append(("OK  " if ok else "FAIL") + " " + line)
        if not ok:
            offenders.append(name)
    return offenders, report

offenders, report = gate({"gen_pass": 0.83, "cost_per_case": 0.030, "p95_latency_ms": 1550})
print("\n".join(report))
print("BLOCK" if offenders else "SHIP", offenders)  # quality up but 3x cost -> BLOCK ['cost_per_case']

def eval_suite(model):
    # in real life these call the golden-set harness against `model`;
    # returned here as a dict so the gate logic is testable offline.
    return {"claude-opus-4-8":  {"gen_pass": 0.86, "safety": 1.0, "cost_per_case": 0.012},
            "claude-haiku-4-5": {"gen_pass": 0.85, "safety": 1.0, "cost_per_case": 0.003}}[model]

def authorize_canary(control, candidate, min_gen=0.02, cost_must_drop=True):
    c0, c1 = eval_suite(control), eval_suite(candidate)
    reasons = []
    if c1["safety"] < 1.0:
        reasons.append("safety hard-fail")
    if c1["gen_pass"] < c0["gen_pass"] - min_gen:
        reasons.append(f"quality regressed {c0['gen_pass']}->{c1['gen_pass']}")
    if cost_must_drop and c1["cost_per_case"] >= c0["cost_per_case"]:
        reasons.append("no cost win")
    return (not reasons), reasons

ok, reasons = authorize_canary("claude-opus-4-8", "claude-haiku-4-5")
print("CANARY" if ok else "BLOCK", reasons)  # quality held within 0.02, cost dropped -> CANARY

Knowledge check check yourself