Case study: compliance doc agent

1 · The scenario & its requirements advanced

Picture the meeting where this project gets killed. An engineer demos a slick "chat with your contracts" bot. It answers a question about a termination clause instantly and confidently. Then someone from legal asks the only question that matters: "Where in the document does it say that?" — and the bot has no answer, because it summarized from a blur of retrieved text. In a consumer app that's a shrug. Here it's a showstopper: the bank cannot file, act on, or defend a compliance decision it can't trace to a specific line of a specific document. The demo dies in the room, and the lesson is that in a regulated domain the usual default flips — a confident wrong answer is worse than no answer at all.

That flip is why the first step of this case study is not architecture or code — it's requirements, and they read differently than the feature lists you're used to. When you build a normal product, requirements describe what the system does ("summarize a document", "answer questions"). Here they describe what the system is not allowed to do: never assert an obligation it can't cite, never treat a machine output as authoritative, never lose the paper trail. Think of it like the difference between a helpful research assistant and a court reporter. The assistant is judged on being useful; the court reporter is judged on being verifiable — every word attributable to a source, on the record, reviewable later. This agent must behave like the court reporter.

Before the table below, notice how the pieces hang together, because they aren't independent wishes — they form a single chain of trust:

• Accuracy without auditability is worthless — a correct obligation you can't trace back to a clause can't be defended when challenged, so it can't be relied on.
• No-hallucination is the load-bearing rule — it's what lets a human trust the pipeline enough to review rather than re-read every source from scratch.
• Human sign-off is the release valve — it converts "the machine proposed this" into "a person with authority approved this", which is the only status that counts as authoritative.

The common beginner mistake is to treat these as nice-to-haves bolted on after the model works. They are the opposite: they are the reason the model is allowed to run at all. Build the controls first and the model becomes a fast draft-generator inside a safe frame; build the model first and you get the demo that dies in the meeting.

A mid-size bank has a shelf of long documents: vendor master-services agreements, regulatory rulebooks, and internal policies — each tens to hundreds of pages. A compliance analyst's job is to read them and answer three questions: What are we obligated to do? What's the risk if we don't? Where exactly does the document say so? Doing this by hand is slow, and a missed clause can become a regulatory finding. The bank wants an agent that extracts obligations, flags risk, and answers questions over these documents — but only if it can be trusted the way a human analyst is trusted: with a citation for every claim and a human sign-off before anything is filed.

That trust requirement is the whole design. This is not a chatbot where a wrong answer is embarrassing; it is a system where a wrong obligation is a legal liability. So the requirements read like a controls document, not a feature list:

2 · The architecture, end to end advanced

Here's the trap that catches most teams building their first document agent. They wire the model straight to the document — "here's the contract, extract the obligations" — and it works beautifully in the demo. Then it meets a real 200-page rulebook and three things go wrong at once: the cost balloons because every question re-reads the whole document, the model quietly invents a clause that sounds right, and when an auditor asks "how did you arrive at this?" there's nothing to show. Each of those is a separate leak, and the architecture in this section is best understood as a pipeline built specifically to plug all three, in the right order.

The mental model that makes the whole diagram click: this is not one clever AI call, it's an assembly line with a quality-control station and a human inspector at the end. Think of how a pharmacy fills a prescription. The doctor's order (the document) comes in; a technician counts out the pills (extraction); a pharmacist checks the count against the label before it goes out the door (the confidence gate + citation check); and every fill is logged with who dispensed it (the audit trail). No single step is trusted on its own — the safety comes from the sequence, and specifically from a checkpoint that can reject work and a human who signs before anything leaves the building.

Why does the order of the stations matter so much? Because each one exists to protect the next:

• Chunk + cache comes first so you pay to read the document once, not once per obligation — the economics that make everything downstream affordable enough to run at all.
• Extract-with-citation comes before the gate because the gate has nothing to check unless every proposed obligation already carries a claimed source span.
• The confidence gate comes before the human so the human's scarce attention is spent only on the uncertain and the ungrounded, not on rubber-stamping obvious accepts.
• The audit log wraps everything because a decision you can't reconstruct later is, for compliance purposes, a decision that never happened defensibly.

The precise term for what this design achieves is grounding by construction: rather than hoping the model stays truthful, the pipeline makes an ungrounded obligation structurally impossible to accept. Keep that phrase in mind as you read left to right — every box is there to turn a "we hope it won't" into a "it can't".

Read the pipeline left to right. A large document is chunked and cached so we don't re-pay to read it on every question; each chunk is run through a schema-constrained extraction that must return a source span (a citation) for every obligation; a confidence gate decides what is trustworthy enough to auto-accept; everything uncertain goes to human sign-off; and every step writes to an append-only audit log.

Three design commitments make this safe rather than merely clever. First, extraction is grounded: the schema requires a citation, so an obligation with no supporting span is invalid by construction. Second, the gate is conservative: uncertainty routes to a human, never to auto-accept. Third, the log is immutable: you can reconstruct exactly what the agent claimed, from which source, at which prompt version, and who approved it — which is what an auditor will ask for.

3 · Ingesting & caching big documents advanced

Imagine hiring a lawyer to answer twenty questions about one contract — and re-reading the entire contract from page one before every single answer. You'd be billed for twenty full read-throughs of a document that never changed. That's absurd for a human, and it's exactly what a naive document agent does: each question ships the whole 200-page document back to the model, and you pay full input price to "read" it every time. The lab in this section makes the number concrete, but the intuition is enough to feel the problem — the document is the expensive part, and you're paying for it over and over.

Prompt caching is the fix, and the analogy that makes it stick is a bookmark. Instead of re-reading, the lawyer reads the contract once, then keeps it open on the desk with the relevant sections tabbed. Each new question just consults the already-open book. Anthropic's prompt caching does the same thing at the API level: you mark a large, stable chunk of the prompt (the document) as cacheable; the first request pays a small premium to "read and bookmark" it (the cache write), and every later request that reuses that same prefix pays a steeply discounted cache read instead of full price. The document sits ready; only your small, changing question is billed at normal rates.

Two subtleties separate people who use caching correctly from people who get burned by it:

• The cache is a prefix, so ordering is everything. The stable stuff — document, system prompt, schema — must come first, and the volatile stuff — the specific question — must come after the cache breakpoint. Put the question before the document and you've changed the prefix every time, so nothing ever hits the cache and you quietly pay full price while believing you're saving.
• A cache is a snapshot, and snapshots go stale. This is the operational trap you'll meet again in the hardening section: if the contract is amended but the cache still holds the old text, the agent will confidently answer against a version that no longer exists — and cite it. The discipline is to key the cache by the document's version or hash and invalidate it the moment the document changes.

There's also a genuine trade-off worth naming, not hiding. Caching is a bet: you pay the write premium up front gambling that you'll reuse the document enough times to earn it back. Ask one question and walk away and caching actually cost you a little more. Ask twenty and the write premium is dwarfed by the cheap reads. That's why this pattern fits a compliance agent so well — analysts don't ask one question of a contract, they interrogate it — and why you'll see the savings climb as reuse climbs.

A 200-page rulebook is far too big to re-send on every question. Anthropic's prompt caching is built for exactly this: you mark a large, stable prefix (the document) with a cache breakpoint, pay a one-time cache-write premium, and then every subsequent request that reuses that prefix pays a deeply discounted cache-read instead of full input price. That is what makes "ask twenty questions of one big contract" economically sane.

Two ingestion decisions matter. Chunking: long documents are split so that each obligation lives whole inside one chunk (split on clause/section boundaries, not blindly every N characters), because a citation that straddles a chunk boundary is useless. Cache placement: the document (and the stable system prompt + schema) go in the cached prefix; only the small, changing question goes after the breakpoint. The lab below models the cost of that arrangement with pure arithmetic — no network.

caching_cost.py# stdlib only, no network. Models the economics of caching one big document
# across many questions. Prices are $ per 1M tokens; multipliers reflect Anthropic's
# PUBLIC caching guidance (cache write ~1.25x input, cache read ~0.1x input).
# Verify the exact numbers in Anthropic's current docs before budgeting.

def session_cost(doc_tokens, n_questions, q_tokens, out_tokens,
                 in_price, out_price, write_mult=1.25, read_mult=0.10):
    """Compare re-sending the whole doc every question vs caching it once."""
    # No cache: every question re-pays full input price for the whole document.
    no_cache_in = (doc_tokens + q_tokens) * n_questions
    no_cache = (no_cache_in * in_price + out_tokens * n_questions * out_price) / 1_000_000

    # With cache: pay the write premium once for the doc, then cheap reads thereafter.
    write = doc_tokens * in_price * write_mult
    reads = doc_tokens * in_price * read_mult * (n_questions - 1)
    questions_in = q_tokens * in_price * n_questions          # the small changing part
    outputs = out_tokens * out_price * n_questions
    cached = (write + reads + questions_in + outputs) / 1_000_000

    return {
        "no_cache_usd": round(no_cache, 2),
        "cached_usd": round(cached, 2),
        "saved_usd": round(no_cache - cached, 2),
        "saved_pct": round(100 * (no_cache - cached) / no_cache),
    }

# A 200-page rulebook (~150k tokens), 20 questions, on a mid model ($3 in / $15 out).
r = session_cost(doc_tokens=150_000, n_questions=20, q_tokens=80, out_tokens=350,
                 in_price=3.00, out_price=15.00)
for k, v in r.items():
    print(f"{k}: {v}")

no_cache_usd: 9.11
cached_usd: 1.53
saved_usd: 7.58
saved_pct: 83

4 · Schema-constrained extraction with citations expert

Ask a model to "extract the obligations" as free text and you'll get prose you then have to parse, second-guess, and babysit. Worse, nothing stops it from writing a beautifully phrased obligation that isn't actually in the document — it has no obligation (pun intended) to point at a source, so on a hard clause it will smooth over the gap with something plausible. In a chatbot that's a mild annoyance. Extracting obligations from a bank's contracts, a single smoothed-over clause can become a regulatory finding. So the design does two things at this step that turn hope into guarantee: it constrains the shape of the output, and it makes a source citation a mandatory part of that shape.

Start with the shape. Schema-constrained extraction means you hand the model a rigid template — think of a paper form with labelled boxes — and require the answer to come back as that form filled in, not as an essay. Instead of "tell me about the obligations", it's "for each obligation, fill in: party, text, risk level, citation, confidence". A filled form is machine-checkable; an essay is not. This is the same schema-constrained-extraction pattern the Document Intelligence project uses, and its quiet superpower is that it moves correctness checks from "read the answer and judge it" to "check that box 4 is present and well-formed" — something code can do perfectly, every time.

Now the load-bearing move: make the citation box a required field of the form. This is a genuinely different kind of safety than a polite instruction like "please cite your sources". A requested citation is something the model can forget under pressure; a required field is something the reply is structurally invalid without. Put plainly:

• With an optional citation, "an obligation with no source" is a bad answer you have to notice and catch after the fact.
• With a required citation, "an obligation with no source" is an impossible shape — it can't be emitted as a valid reply in the first place.

That is the difference between hoping and enforcing. You've promoted "every claim must be grounded" from a wish written in the prompt to a type guarantee written in the schema. Be honest about the limit, though — and the next section is built entirely on this limit: a required-citation schema forces the model to attach a quote and offsets, but it cannot force that quote to be true. The model can still fill the box with a fabricated or misaligned citation. The schema guarantees the citation is present; only code (§5) can guarantee it is real. The lab below is deliberately an offline, deterministic stand-in for the model call, precisely so you focus on that data contract — the shape and the required span — rather than the network mechanics.

The heart of the agent is one disciplined call per chunk: structured output with a JSON schema forces the model to return obligations in an exact shape, and the schema requires a citation on every obligation. Anthropic's citations feature is designed to return grounded spans that point back into the source document; combined with a required-citation schema, an obligation the model cannot ground simply cannot be emitted as valid.

Below is the deterministic, offline stand-in for that call. A real system would send the chunk to Claude with a required-citation schema; here a fake extractor returns obligations, each with a claimed quote and character offsets, plus a confidence. The point of the lab is the shape of the data and the contract it must satisfy — not the model call, which is why it runs with no key.

extract.py# stdlib only, no network. A deterministic fake extractor that returns the SAME
# shape a schema-constrained Claude call would: obligations, each REQUIRED to carry a
# citation (quote + character span into the source) and a confidence score.
from dataclasses import dataclass, field

DOC = (
    "Section 4.1 The Bank shall notify the Regulator within 30 days of any material "
    "breach. Section 7.2 The Vendor shall maintain insurance of at least $5,000,000. "
    "Section 9.0 Either party may terminate for convenience upon 90 days written notice."
)

@dataclass
class Citation:
    quote: str
    start: int
    end: int

@dataclass
class Obligation:
    party: str
    text: str
    risk: str                      # low | medium | high
    citation: Citation
    confidence: float

def fake_extract(doc):
    """Pretend-model output. In production this is one schema-constrained Claude call
    per chunk, with the citations feature returning the grounded spans."""
    spans = [
        ("Bank",   "notify the Regulator within 30 days of any material breach",
         "high",   0.94),
        ("Vendor", "maintain insurance of at least $5,000,000",
         "medium", 0.88),
        ("Either party", "terminate for convenience upon 90 days written notice",
         "low",    0.61),
    ]
    obligations = []
    for party, quote, risk, conf in spans:
        start = doc.find(quote)
        end = start + len(quote)
        obligations.append(Obligation(
            party=party, text=quote, risk=risk,
            citation=Citation(quote=quote, start=start, end=end),
            confidence=conf))
    return obligations

for o in fake_extract(DOC):
    print(f"[{o.risk:6}] {o.party:12} conf={o.confidence:.2f} "
          f"@({o.citation.start},{o.citation.end})")

[high  ] Bank         conf=0.94 @(27,85)
[medium] Vendor       conf=0.88 @(116,157)
[low   ] Either party conf=0.61 @(188,241)

5 · The no-hallucination discipline — abstain & cite expert

Here is the failure that keeps compliance leads awake, and it looks nothing like an obvious bug. The model returns an obligation, in perfect schema, with a citation filled in, and a confidence of 0.97. Everything looks green. But the quoted text isn't actually in the contract — the model paraphrased a clause from a different agreement it saw in training, or subtly shifted the numbers, and dressed it in a citation that points at plausible-looking offsets. It feels more trustworthy than the shaky 0.6 answers precisely because it's confident. If you trust it, a clause the bank never agreed to flows into a compliance filing. This is the hallucination that hurts, and this section is the single control that stops it.

The core insight is a distinction people conflate all the time: confidence is an opinion; verification is a fact. The confidence score is the model's own assessment of itself — and a model that hallucinates is, by definition, wrong about how sure it should be, so its confidence on a fabrication is worthless. You cannot fix a self-report by reading more of the self-report. You need an external, objective check, and here the check is beautifully simple and needs no AI at all: take the quote the model claims, take the offsets it claims, and ask a plain string comparison — does the source document, at exactly those character positions, contain exactly that text? If yes, the citation is grounded. If no, it's a fabrication, no matter how confident the model was.

Think of it like a bank teller checking a signature. It doesn't matter how confidently the customer signs or how genuine the signature looks in isolation — the teller compares it against the card on file. Match or no match. The confidence of the person signing is irrelevant to whether the signature checks out. Citation verification is that comparison, in code.

Two rules fall out of this, and their order is the whole discipline:

• Verify before you consider confidence. The check runs first and can veto anything. A citation that fails to verify is dropped and escalated even at confidence 0.99 — because a confident claim you cannot ground is exactly the dangerous case, not the safe one.
• When you can't ground it, abstain — don't guess. Abstaining routes the chunk to a human. That's a visible gap someone can fill. Guessing produces an invisible wrong answer that flows silently into a filing. In this domain a flagged gap is cheap and recoverable; a silent wrong obligation is neither.

That last line is the value inversion at the heart of the whole case study: being right or abstaining always beats being confidently wrong. A normal product optimizes for coverage — answer as much as you can. A compliance agent optimizes for the opposite — never assert what you can't back up. Hold that thought, because §7 turns it into a hard, measurable gate.

A required-citation schema stops the model from omitting a citation, but it cannot stop the model from fabricating one — quoting text that isn't actually in the document, or pointing at the wrong span. So the citation is not trusted; it is verified by code. This is the single most important control in the whole system, and it needs no model: you check that the claimed quote appears verbatim at the claimed offsets in the source. If it doesn't, the obligation is dropped and the chunk is escalated — the agent abstains rather than assert something it cannot ground.

gate.py# stdlib only, no network. Verifies each claimed citation against the source text,
# then GATES: verified + high-confidence -> AUTO-ACCEPT; verified + low-confidence ->
# REVIEW; a citation that does NOT verify is a hallucination -> DROP + ESCALATE.
CONFIDENCE_THRESHOLD = 0.80

def verify_citation(doc, cite):
    """A citation is valid only if the exact quote sits at the claimed offsets.
    This catches fabricated quotes and misaligned spans — with no model."""
    if not (0 <= cite["start"] < cite["end"] <= len(doc)):
        return False
    return doc[cite["start"]:cite["end"]] == cite["quote"]

def gate(doc, obligations):
    results = []
    for o in obligations:
        if not verify_citation(doc, o["citation"]):
            results.append((o["party"], "DROP+ESCALATE (citation failed to verify)"))
        elif o["confidence"] >= CONFIDENCE_THRESHOLD:
            results.append((o["party"], "AUTO-ACCEPT"))
        else:
            results.append((o["party"], "REVIEW (low confidence)"))
    return results

DOC = "The Bank shall notify the Regulator within 30 days of any material breach."
obligations = [
    # verified + confident -> auto-accept
    {"party": "Bank", "confidence": 0.94,
     "citation": {"quote": "notify the Regulator within 30 days", "start": 15, "end": 50}},
    # verified but low confidence -> review
    {"party": "Bank-B", "confidence": 0.55,
     "citation": {"quote": "material breach", "start": 58, "end": 73}},
    # FABRICATED quote (not in doc) -> drop + escalate
    {"party": "Ghost", "confidence": 0.99,
     "citation": {"quote": "pay a penalty of $1,000,000", "start": 15, "end": 42}},
]
for party, decision in gate(DOC, obligations):
    print(f"{party:8} -> {decision}")

Bank     -> AUTO-ACCEPT
Bank-B   -> REVIEW (low confidence)
Ghost    -> DROP+ESCALATE (citation failed to verify)

6 · Human-in-the-loop & the audit trail professional

Fast-forward eighteen months. A regulator disputes one obligation the bank acted on, and asks a single question: "On what basis was this set, and who approved it?" If the answer is "our AI extracted it, and we think a analyst reviewed it around then," the bank is in serious trouble — not because the obligation was necessarily wrong, but because it can't be defended. Everything in this section exists to make that dreaded question boring to answer. The two pieces are a human in the loop and an audit trail, and it's worth being precise about what each one buys you, because beginners tend to add a login screen, call it "human review", and miss the actual point.

First, the status distinction that the whole workflow turns on. The confidence gate's "AUTO-ACCEPT" bucket does not mean "authoritative" — it means proposed. The agent is always a proposer; authority comes only when a person with compliance responsibility signs off. This is the "the agent proposes, a human disposes" rule, and it's not bureaucratic theater — it's what places accountability on a named human who can be asked to explain the decision. A machine can't be held responsible in a regulatory sense; a person who reviewed the citation and approved it can.

Second, the audit trail — and the mental model here is a courtroom evidence log or a hospital chart, not an application log file. An ordinary debug log exists to help engineers; an audit trail exists to reconstruct the truth for an outsider who wasn't there and doesn't trust you. That reframing dictates two properties most logging gets wrong:

• It must be append-only (immutable). A record you can quietly edit later proves nothing — an auditor's first suspicion is that inconvenient history was rewritten. So approval must append a new record, never overwrite the original proposal. Both the machine's proposal and the human's approval survive, side by side. You can see the agent proposed and the human disposed, in order.
• It must carry full provenance — enough context to replay the decision cold. That's the five fields the code makes concrete: what (the obligation), where (document + exact span), how (model + prompt/schema version), how sure (confidence + whether the citation verified), and who (signer + timestamp). Drop any one and the trail has a hole an auditor will find.

Why bother hashing the whole log into a digest, as the lab does? Because immutability enforced only by convention is a promise, not a guarantee. A running hash over every record means any silent tampering with earlier history changes the digest — a cheap, tamper-evident seal on top of the append-only discipline. It turns "we promise we didn't change it" into "here's the proof nothing changed."

The gate produces three buckets — accept, review, escalate — but "accept" still means proposed, not authoritative. A person with compliance authority reviews the proposed obligations and signs off. What makes the sign-off defensible is the audit trail: an append-only log where every event — extraction, verification result, gate decision, and human approval — is recorded with enough context to reconstruct exactly what happened and why. An auditor's question is always the same: "show me where this came from and who approved it." The log is the answer.

Good audit records share five fields: what (the obligation), where (the document + span), how (model + prompt/schema version), how sure (confidence + verification result), and who (the signer + timestamp). Make the log append-only so history cannot be quietly rewritten — that immutability is the point.

audit.py# stdlib only, no network. An append-only audit trail: every decision is recorded
# with provenance (document, span, model+prompt version, confidence) and, on sign-off,
# who approved it and when. Records are frozen so history cannot be rewritten in place.
from dataclasses import dataclass, asdict
import json, hashlib

@dataclass(frozen=True)                 # frozen -> a written record can't be mutated
class AuditRecord:
    obligation: str
    doc_id: str
    span: tuple
    model_version: str
    prompt_version: str
    confidence: float
    citation_verified: bool
    decision: str
    signed_off_by: str = ""             # empty until a human approves
    signed_at: str = ""

class AuditLog:
    def __init__(self):
        self._records = []              # append-only; never edited in place

    def append(self, rec):
        self._records.append(rec)
        return len(self._records) - 1   # the record's immutable index

    def sign_off(self, idx, approver, at):
        """Approval does not edit the original record — it appends a NEW one."""
        old = self._records[idx]
        approved = AuditRecord(**{**asdict(old),
                                  "decision": "APPROVED",
                                  "signed_off_by": approver, "signed_at": at})
        return self.append(approved)

    def digest(self):
        """A hash over the whole log — any tampering changes the digest."""
        blob = json.dumps([asdict(r) for r in self._records], sort_keys=True)
        return hashlib.sha256(blob.encode()).hexdigest()[:12]

log = AuditLog()
i = log.append(AuditRecord(
    obligation="Bank shall notify the Regulator within 30 days",
    doc_id="MSA-2026-0142", span=(15, 73),
    model_version="claude-x", prompt_version="oblig-extract-v3",
    confidence=0.94, citation_verified=True, decision="AUTO-ACCEPT"))
log.sign_off(i, approver="a.nguyen@bank.example", at="2026-09-06T10:22:00Z")

for n, r in enumerate(log._records):
    who = r.signed_off_by or "(unsigned)"
    print(f"#{n} {r.decision:11} verified={r.citation_verified} by={who}")
print("log digest:", log.digest())

#0 AUTO-ACCEPT verified=True by=(unsigned)
#1 APPROVED    verified=True by=a.nguyen@bank.example
log digest: 8d79848114bc   # varies per run — timestamp is hashed

7 · Evaluating the agent — accuracy, citations, abstention professional

"It looked great in the demo" is how unsafe systems reach production. A compliance agent that answered a dozen questions impressively in a meeting tells you almost nothing about the thousandth contract it hasn't seen. Faith doesn't scale and it isn't defensible; the only thing that lets you say "this is safe to trust" is a measurement against a labeled gold set — a stack of documents where a human has already recorded, per field, what the true answer is (and, importantly, where the true answer is absent). Evaluation is just grading the agent's homework against that answer key, automatically, before you let it near real filings.

The subtle part is what to measure, because a single accuracy score is dangerously misleading here. Picture a student who aces most questions but occasionally invents a confident, wrong answer on the ones they don't know. Their average might look fine — and in compliance that student is a catastrophe, because the invented answers are exactly the ones that become legal liabilities. So this agent is graded on three separate axes that map one-to-one onto the three risks from the earlier sections:

• Extraction accuracy — did it find the obligations that are really there without inventing ones that aren't? This is the familiar precision/recall pair: precision asks "of what it claimed, how much was true", recall asks "of what was true, how much did it find".
• Citation integrity — of everything it asserted, what fraction had a citation that actually verifies against the source? This is grounding, measured. In this domain it should be 1.0; anything less means ungrounded claims are leaking through.
• Abstention correctness — when it genuinely didn't know, did it abstain (flag the gap for a human) rather than silently drop the obligation or paper over it? And did it abstain only when it should, not on everything?

The last one deserves special weight, and it's why the lab treats it as a hard gate rather than just another number. A missed obligation is tolerable if and only if the agent flagged the miss — a visible gap a human can close. A silent miss (the obligation is just gone, no flag) is the dangerous failure: nobody knows to look for it. So the eval encodes the value inversion from §5 directly into the release rule — a run can have decent precision and recall and still fail outright if it ever missed something silently. This is a deliberate reversal of the usual instinct to chase the highest average score. Here, "never pretend you found everything" outranks "find everything", and the gate enforces that ranking so no future model or prompt change can quietly trade it away.

You cannot ship this on faith. Three evals, run against a labeled gold set of documents, tell you whether it's safe to trust — and they map directly onto the three risks. Extraction accuracy: did it find the obligations that are actually there, without inventing ones that aren't? Citation integrity: do the citations it returns actually verify against the source? Abstention correctness: when it's genuinely uncertain or ungrounded, does it abstain — and does it abstain only then, not on everything?

Anthropic's published guidance on evals is the frame here: build a representative test set, score deterministically where you can, and treat the safety metric as a hard gate. The lab computes all three from a tiny gold set with pure Python.

evals.py# stdlib only, no network. Scores a run against a labeled gold set on the three
# axes that matter for a compliance agent. All deterministic — no model, no judge.
GOLD = {                          # doc_id -> set of true obligation ids present
    "A": {"o1", "o2"},
    "B": {"o3"},
}

# What the agent produced: (predicted ids, ids whose citation verified, ids abstained on)
RUN = {
    "A": {"predicted": {"o1", "o2"}, "verified": {"o1", "o2"}, "abstained": set()},
    # doc B: found o3 but its citation did NOT verify, and it correctly abstained on it
    "B": {"predicted": set(),        "verified": set(),        "abstained": {"o3"}},
}

def extraction_accuracy(gold, run):
    tp = fp = fn = 0
    for doc, truth in gold.items():
        pred = run[doc]["predicted"]
        tp += len(truth & pred)
        fp += len(pred - truth)
        fn += len(truth - pred)
    precision = tp / (tp + fp) if (tp + fp) else 1.0
    recall = tp / (tp + fn) if (tp + fn) else 1.0
    return round(precision, 2), round(recall, 2)

def citation_integrity(run):
    """Of everything the agent PREDICTED, what fraction had a verifiable citation?"""
    pred = sum(len(r["predicted"]) for r in run.values())
    ok = sum(len(r["verified"]) for r in run.values())
    return round(ok / pred, 2) if pred else 1.0

def abstention_correct(gold, run):
    """A missed true obligation is SAFE only if the agent abstained on it (didn't
    silently drop it). Silent misses are the dangerous failure."""
    for doc, truth in gold.items():
        missed = truth - run[doc]["predicted"]
        if missed - run[doc]["abstained"]:      # missed but NOT abstained = silent
            return False
    return True

prec, rec = extraction_accuracy(GOLD, RUN)
print(f"precision: {prec}  recall: {rec}")
print(f"citation_integrity: {citation_integrity(RUN)}")
print(f"abstention_correct (hard gate): {abstention_correct(GOLD, RUN)}")

precision: 1.0  recall: 0.67
citation_integrity: 1.0
abstention_correct (hard gate): True

8 · Tech-lead — failure modes & hardening tech-lead

An engineer asks "does it work?" A tech lead asks "how will it hurt us, and will we see it coming?" That shift is the whole job at this level. By now the happy path is solid — extract, cite, verify, gate, sign off, log. But a compliance system doesn't fail loudly with a stack trace; it fails quietly, producing a confident, wrong obligation that looks exactly like a correct one until a regulator disagrees months later. Your responsibility as a lead is to enumerate those silent-failure paths in advance and, for each, install a control that converts the silent error into a visible one — a dropped claim, an escalation, an alert. In this domain a visible gap is recoverable; a silent wrong answer is a liability. The table below is that enumeration; here's the way to think about it so the individual rows aren't just a checklist to memorize.

Notice that every failure mode is a variation on one theme: something the system believes is trustworthy silently isn't. Group them and the pattern is easier to hold in your head:

• The model lies, plausibly — a fabricated citation (a confident quote that isn't in the document). The control is code-side citation verification from §5, which drops and escalates it. This is the failure the earlier sections were built to catch.
• The pipeline drops something on the floor — an obligation split across a chunk boundary is seen whole by neither chunk and just vanishes. The control is structural: chunk on clause boundaries with overlap, reconcile by span, and let the recall eval catch what slips.
• A trusted input goes stale — a cached document outlives an amendment, so answers cite a version that no longer exists. The control is to key the cache by document version/hash and record that version in every audit record (the caching trap from §3, now with teeth).
• The controls themselves erode over time — a new model shifts confidence and the gate quietly lets weaker claims through (threshold drift), or reviewers rubber-stamp a flooded queue until sign-off means nothing (sign-off theater). The controls: re-run the eval set on every model/prompt version, pin and log the prompt version, keep the queue small by gating conservatively, and sample-audit approvals.

The single most useful mental model to carry out of this whole case study is this: the agent is a proposal engine wrapped in controls. Its job is to draft cited obligations quickly; the controls' job is to guarantee nothing untrustworthy is ever treated as authoritative. Design it that way and the system degrades safely — under stress, load, or a model regression, it abstains and escalates more, it does not start confidently asserting nonsense. That property — failing toward "I'm not sure, ask a human" rather than toward a wrong answer — is the definition of a hardened compliance system, and it's the bar the table and rubric below hold you to.

A lead owns the ways this system can hurt the business. Each failure mode below is a silent-error path, and each control turns the silent error into a visible one — a dropped claim, an escalation, an alert — because in compliance a visible gap is recoverable and a silent wrong answer is not.

The mental model for a lead: the agent is a proposal engine wrapped in controls. Its job is to draft cited obligations fast; the controls' job is to guarantee that nothing untrustworthy is ever treated as authoritative. Get citation verification, conservative gating, the immutable audit trail, and versioned evals right, and the system degrades safely: under stress it abstains and escalates more, it does not start asserting confident nonsense.

Target schema (one object per field):
{
  "field": "liability_cap",
  "value": "12 months of fees" | null,
  "citation": {"doc": "MSA.pdf", "loc": "§8.2"} | null,
  "found": true | false
}

Instruction to the model:
"Extract each field ONLY from the provided sources. For each, return value,
 the citation (doc + location) it came from, and found=true. If a field is
 not present in the sources, return value=null, citation=null, found=false.
 Never infer a value that is not written in the sources."