Case study: clinical-knowledge RAG

This is a full lifecycle walkthrough — requirements → architecture → design → code → evaluation → hardening — for one representative system: a clinical-knowledge RAG assistant. A clinician types a question ("is metformin safe at this patient's eGFR?"); the assistant retrieves from a vetted corpus of internal protocols and approved reference material, and returns a grounded, cited answer — or abstains when it isn't sure. Every design choice here is dominated by two forces you don't feel as sharply in a consumer chatbot: patient privacy and the fact that a wrong answer can hurt someone.

1 · Requirements — privacy, residency, and the scope line advanced

Picture the demo that goes beautifully and ships nothing. An engineer wires Claude to a folder of clinical PDFs over a weekend, asks "is metformin safe at a low eGFR?", gets a fluent cited paragraph, and the room nods. Then legal asks three questions — Where did the patient's name go? Which region did that request touch? What happens when the model is wrong? — and the project stops for six months. The retrieval was never the hard part. In a clinical system the constraints that decide whether you can ship at all arrive before any feature, and they are the thing beginners skip.

Think of these requirements like the load-bearing walls of a building versus the paint. Features — search, summarization, a nice UI — are paint; you can change them late and cheaply. The walls here are non-functional: privacy, data residency, grounding, abstention, decision-support scope. They never show up as a button, but move one and the whole floor plan changes. That is why we write them down first and let them veto architectures, rather than designing a system we like and bolting compliance on afterward — bolted-on privacy is how PHI ends up in a log nobody audited.

The precise idea to hold onto: a requirement that dominates design is one that eliminates otherwise-reasonable options. "Answers should be fast" is a goal you tune. "Un-redacted patient data must never cross the trust boundary" is a veto — it deletes entire deployment choices from the table before you compare them. The common mistake is treating all requirements as a wish-list to be balanced; in a safety-critical domain some of them are gates, and a gate you fail means the design is wrong no matter how good the rest is. Read the table below as that gate list — each row is a wall, and the third column is which architectures it knocks down.

Before a single line of retrieval code, write down the constraints that will veto architectures. For a clinical assistant they are unusually sharp, and most of them are non-functional — they never appear as a feature, but they decide whether you can ship at all.

Two of these deserve care because people get them wrong. Data residency is a legal/contractual property of where bytes live and get processed, and it is distinct from encryption — verify the specifics with your compliance/legal team, and confirm current region availability and eligibility in Anthropic's current docs rather than assuming. And abstention is not a failure mode to be minimized to zero; in a safety-critical setting, a calibrated "I can't ground this — ask the on-call pharmacist" is a success.

2 · Architecture — the request path end to end advanced

Here's a mistake that looks harmless in a diagram and is catastrophic in production: drawing the boxes in whatever order they occurred to you. In most systems the order of components is a matter of taste. In this one, the order is the safety property. Put "log the request" before "redact PHI" and you have quietly built a system that writes patient names to disk on every single query — and it will pass every functional test, because logging and redaction both "work." The bug isn't in a box; it's in the sequence.

The mental model that makes this click is a one-way airlock. Think of the airlock on a spacecraft or a biosafety lab: things may only move through it in one direction, and each door must be sealed before the next opens. Our pipeline is the same. Raw patient text starts inside the trust boundary; the redactor is the first door and it must close (PHI stripped) before the next door (any model call, any log write) can open. Retrieval is scoped to a vetted shelf so the model can only speak from approved material. The citation-and-gate stage is the door between a fluent guess and a human — it converts model output into either a reviewable evidence bundle or an honest "I don't know." And the clinician review is the final door before anything reaches care.

So when you read the pipeline below, don't read it as "steps that happen." Read each arrow as a promise the previous box kept: nothing reaches a model until PHI is gone, nothing reaches a clinician until it is grounded and gated, and nothing at all happens without an audit record. The common failure is to see this as an ordinary data-flow diagram and feel free to reorder for convenience or performance. You are not free to. Deleting the redact box or the review box doesn't give you a faster version of this system — it gives you a different, unsafe one. The architecture's real content is the constraints between the boxes, not the boxes.

Every clinician query flows through the same guarded pipeline. Read it left to right: nothing reaches a model until PHI is stripped, nothing reaches a clinician until it is grounded and gated, and nothing happens without an audit record.

The ordering is the security design. Redaction is first so raw PHI never crosses your trust boundary. Retrieval is scoped to a vetted corpus so the model can only speak from approved material. Citations and the confidence gate sit between generation and the human, converting a fluent guess into either a reviewable evidence bundle or an honest abstention. The clinician is in the loop by construction, and the audit log records what happened without storing what it must not.

3 · Redaction — strip PHI before it leaves the boundary advanced

Imagine the incident review a year from now. Someone greps the trace store while debugging retrieval and finds a real patient's name, MRN, and date of birth sitting in plain text, copied there on every query for eleven months. Nobody decided to do that. It happened because the redactor was treated as a nice-to-have that ran "somewhere in the flow" instead of a hard gate that runs first and fails closed. This section is where that airlock door actually gets built, so it's worth being precise about what the door promises.

The redactor's job is narrower and stranger than "remove sensitive words." Its contract has three parts. First, take free text that may contain protected health information and return text safe to send onward. Second — and this is the part people miss — report what type of PHI it removed (a name, an MRN, a date) so the audit log can record that redaction happened, without ever storing the raw value it stripped. You want to be able to prove "we redacted a name here" without the name itself living anywhere. Third, when it can't be confident the text is clean, it must block rather than let a maybe-dirty string through. A redactor that guesses is worse than none, because it gives you false confidence.

The analogy that keeps engineers honest here is a metal detector at an airport, not a spellchecker. A spellchecker's failure is cosmetic and you'd rather it be permissive. A metal detector's failure can be lethal, so it is deliberately tuned to false-alarm — better to wand a hundred harmless belt buckles than miss one weapon. Redaction lives in the metal-detector world: the cost of leaking one identifier dwarfs the cost of blocking one harmless question, so the safe default when uncertain is stop, the opposite of the UX-first "when in doubt, let it through" instinct most software is built on. Keep that asymmetry in mind as you read the demo below — and note loudly that the regex approach it uses is a teaching shape, not a shippable control, precisely because hand-written patterns miss the messy real cases a metal detector must catch.

The first executable component is the redactor. Its contract: take free-text that may contain PHI, return text safe to send onward, and report what type of PHI was removed so the audit log can record it — without ever storing the raw value. The lab below is a deterministic, stdlib-only demonstration; production uses a vetted NER redactor (e.g. Presidio/spaCy) whose recall you measure against your own PHI test corpus.

redact.pyimport re

# A DEMONSTRATION PII/PHI redactor. Deterministic, stdlib-only, no ML, no network.
# In production use a vetted NER redactor (e.g. Presidio/spaCy); verify recall
# against your own PHI test corpus and confirm scope with your compliance/legal team.
PATTERNS = [
    ("MRN",   re.compile(r"\bMRN[:#]?\s*\d{6,}\b", re.I)),
    ("DOB",   re.compile(r"\bDOB[:#]?\s*\d{1,2}/\d{1,2}/\d{4}\b", re.I)),
    ("SSN",   re.compile(r"\b\d{3}-\d{2}-\d{4}\b")),
    ("PHONE", re.compile(r"\b\d{3}[-.]\d{3}[-.]\d{4}\b")),
    ("EMAIL", re.compile(r"\b[\w.]+@[\w.]+\.\w+\b")),
    ("NAME",  re.compile(r"\b(?:Mr|Ms|Mrs|Dr)\.?\s+[A-Z][a-z]+")),
]

def redact(text):
    """Replace recognised PHI spans with typed placeholders. Returns
    (clean_text, spans_removed) so an audit log can record WHAT was stripped
    by type without ever storing the raw PHI value."""
    removed = []
    clean = text
    for label, rx in PATTERNS:
        clean, n = rx.subn(f"[{label}]", clean)
        removed += [label] * n
    return clean, sorted(removed)

def has_raw_phi(text):
    """Belt-and-suspenders: would ANY pattern still fire on the cleaned text?"""
    return any(rx.search(text) for _, rx in PATTERNS)

q = ("Mr. Alvarez (MRN: 40021991, DOB 03/14/1958) asks whether metformin "
     "is safe with his eGFR; reach him at 415-555-0198 or a.alvarez@example.com.")
clean, removed = redact(q)
print("CLEAN  :", clean)
print("REMOVED:", removed)
print("SAFE_TO_SEND:", not has_raw_phi(clean))

CLEAN  : [NAME] ([MRN], [DOB]) asks whether metformin is safe with his eGFR; reach him at [PHONE] or [EMAIL].
REMOVED: ['DOB', 'EMAIL', 'MRN', 'NAME', 'PHONE']
SAFE_TO_SEND: True

4 · Model sourcing — API vs region-locked vs self-hosted professional

You have probably watched this argument happen, maybe been in it. The team debates for two weeks: hosted API is easiest, self-hosting is "more secure," someone read that a competitor uses a managed cloud. Slides get made. Everyone has a preference. And the whole debate is a category error — because in a PHI workload under data-residency rules, deployment is not a preference you argue, it's an answer you derive from the constraints you already wrote down in §1. The uncomfortable, useful truth is that if two engineers reach different deployment choices for the same constraints, at least one of them made an aesthetic call where a logical one was required.

Think of it like a decision made by a flowchart, not a vote. The constraints are the inputs; the deployment is the output; your job is to run the flowchart honestly, not to lobby for the box you like. And the very first branch isn't a choice at all — it's a veto: if un-redacted PHI would reach any external endpoint and redaction isn't guaranteed, the correct output is BLOCK. Not "pick the safest-looking vendor." Block. A sourcing function that can't return "don't do this" is a broken function, because it will always launder a hard privacy failure into a soft endpoint preference.

Only after that veto passes do the real trade-offs open up, and they are genuine ones with no free lunch. A region-locked managed cloud keeps data in-jurisdiction while still giving you a frontier-quality model, at some extra setup. A self-hosted open-weight model is the answer when egress is forbidden entirely (a truly air-gapped network) — but you pay for it in capability and in the ops burden of running, patching, and scaling inference yourself. The standard hosted API is the simplest path when residency isn't mandated. The mistake to avoid is reaching for self-hosting because it feels safest: it maximizes control but also maximizes what you have to operate correctly, and "we run our own model" is not a compliance strategy on its own. The code below encodes exactly this priority order — veto first, then residency, then air-gap, then default — so you can see the logic instead of debating the vibes. Treat its option names as a general sketch of a real deployment surface and confirm current region and eligibility details in the provider's own docs.

With residency and PHI constraints written down, the deployment choice is largely decided for you. This is a compliance-and-cost decision, not a preference. Anthropic's public deployment surface — at a general level — includes the Claude API and Claude on managed clouds (Amazon Bedrock, Google Vertex AI) that offer region selection and enterprise data controls; open-weight models are an option when egress is forbidden entirely. Exact region availability, HIPAA/BAA eligibility, and data-retention terms change — verify in Anthropic's current docs and confirm with your compliance/legal team.

sourcing.py# Model-sourcing decision for a PHI workload. Deterministic, stdlib only.
# The options below reflect Anthropic's PUBLIC deployment surface at a general
# level: the Claude API, and Claude via managed clouds (Amazon Bedrock, Google
# Vertex AI) that offer region selection and enterprise data controls. Exact
# region availability, BAA/HIPAA eligibility, and zero-retention terms change --
# verify in Anthropic's current docs and confirm with your compliance/legal team.

def choose_deployment(req):
    """req: dict of requirements. Returns (deployment, rationale)."""
    if req["phi_in_prompt"] and not req["redaction_guaranteed"]:
        return ("BLOCK", "Do not send un-redacted PHI to any external endpoint.")
    if req["data_residency"] and req["needs_frontier_quality"]:
        return ("Claude via region-locked managed cloud (Bedrock/Vertex)",
                "Keeps data in-region under an enterprise agreement while using a frontier model.")
    if req["air_gapped"]:
        return ("Self-hosted open-weight model",
                "No egress permitted; accept a capability trade-off and own the ops burden.")
    return ("Claude API with a signed enterprise agreement",
            "Simplest path when region-locking is not mandated.")

scenarios = [
    {"name": "region-locked frontier", "phi_in_prompt": True, "redaction_guaranteed": True,
     "data_residency": True, "needs_frontier_quality": True, "air_gapped": False},
    {"name": "raw PHI, no redaction", "phi_in_prompt": True, "redaction_guaranteed": False,
     "data_residency": True, "needs_frontier_quality": True, "air_gapped": False},
    {"name": "air-gapped hospital net", "phi_in_prompt": True, "redaction_guaranteed": True,
     "data_residency": True, "needs_frontier_quality": False, "air_gapped": True},
]
for s in scenarios:
    dep, why = choose_deployment(s)
    print(f"{s['name']:<24} -> {dep}")

region-locked frontier   -> Claude via region-locked managed cloud (Bedrock/Vertex)
raw PHI, no redaction    -> BLOCK
air-gapped hospital net  -> Self-hosted open-weight model

5 · Grounded RAG with citations — the retriever professional

Ask a bare language model a clinical question and it will answer — confidently, fluently, and from nobody-knows-where. It might paraphrase a real guideline, or a guideline from a different country, or a plausible sentence it assembled from the statistical soup of its training. You can't tell which, and neither can the clinician, and that ambiguity is intolerable when the answer touches a patient. Retrieval exists to remove the ambiguity: instead of asking the model "what do you know about metformin?", you hand it the exact approved paragraphs and ask "answer only from these, and tell me which one you used." The model stops being an oracle and becomes a librarian — it fetches from a known shelf and points at the spine.

Here's the mechanism, stripped to its core so the demo makes sense. To find "the most relevant chunks," the retriever needs a way to measure relevance as a number. The classic trick is to turn both the question and each stored chunk into a vector — a list of numbers summarizing its content — and then measure the angle between them with cosine similarity: point in nearly the same direction, high score; point apart, low score. The toy below uses word-frequency counts as the vectors so it runs offline with no model; a production system swaps in learned embeddings that capture meaning, and usually blends dense (embedding) scores with a keyword method like BM25. But the shape — vectorize, score, sort, take the top few — is identical, and that's what you're meant to learn from the code.

Two things about this step matter more than the ranking math, and both are easy to under-weight. First, the score isn't just for ordering — it's a confidence signal the next stage depends on. When the top chunk scores high, the shelf has something to say; when every score collapses toward zero, that's the system honestly reporting "nothing here answers this," which is exactly the input the abstention gate in §6 needs. Second, and this is the mistake that quietly sinks clinical RAG: a perfect retriever over a bad shelf is worthless. Relevance ranking only surfaces what's in the corpus, so what goes into the corpus is a clinical-governance decision, not an engineering one — only vetted, versioned, approved material, with revisions tracked so a citation points at an exact document a human can audit. Watch for the version marker (v7) in the demo; that's not decoration, it's the audit trail.

Retrieval is the heart of the system, and its job description is narrow: pull the most relevant chunks from a vetted corpus so the model can answer only from approved material. The demo below is a deterministic bag-of-words cosine over a tiny corpus of internal protocols plus reference snippets — stdlib only, no embeddings, no network. Production uses real embeddings and hybrid (dense + BM25) retrieval, and Anthropic publicly describes contextual retrieval (prepending a situating line to each chunk) to lift recall — verify in Anthropic's current docs.

retriever.pyimport re, math
from collections import Counter

# A DEMONSTRATION retriever over a VETTED corpus: internal protocols + a few
# clinical-reference snippets. Deterministic bag-of-words cosine, stdlib only,
# no embeddings, no network. Production uses real embeddings + hybrid BM25 and
# (per Anthropic's public "contextual retrieval" guidance) a situating context
# line per chunk -- verify in Anthropic's current docs.
CORPUS = [
    {"id": "PROTO-CKD-07", "src": "Renal Dosing Protocol v7",
     "text": "Metformin is contraindicated when eGFR is below 30 mL/min/1.73m2. "
             "Between 30 and 45, do not initiate; review dose if already prescribed."},
    {"id": "PROTO-SEPSIS-03", "src": "Sepsis Bundle v3",
     "text": "Begin broad-spectrum antibiotics within one hour of recognising "
             "sepsis and obtain blood cultures before the first dose."},
    {"id": "REF-METFORMIN-1", "src": "Formulary Reference",
     "text": "Metformin lowers hepatic glucose output. Renal function should be "
             "assessed before starting and at least annually thereafter."},
]

def toks(s): return re.findall(r"[a-z0-9]+", s.lower())

def vec(s):
    c = Counter(toks(s))
    n = math.sqrt(sum(v*v for v in c.values())) or 1.0
    return {k: v/n for k, v in c.items()}

def cosine(a, b):
    return sum(a.get(k, 0.0)*b.get(k, 0.0) for k in a)

INDEX = [(d, vec(d["text"])) for d in CORPUS]

def retrieve(query, k=2):
    qv = vec(query)
    scored = [(round(cosine(qv, dv), 3), d) for d, dv in INDEX]
    scored.sort(key=lambda x: x[0], reverse=True)
    return scored[:k]

hits = retrieve("Is metformin safe if eGFR is low?", k=2)
for score, d in hits:
    print(f"{score:>5}  {d['id']:<16} {d['src']}")

0.441  PROTO-CKD-07     Renal Dosing Protocol v7
0.081  REF-METFORMIN-1  Formulary Reference

6 · Abstention — the confidence gate professional

Every instinct you built writing normal software is wrong here, and it's worth feeling that before the code. For years the goal has been: always return something useful, never leave the user staring at an error, degrade gracefully. So the natural move, when retrieval comes back weak, is to stitch together the best of the mediocre chunks and produce an answer — because an answer feels helpful and a blank feels like failure. In a clinical assistant that instinct is a hazard. A confidently wrong answer about a drug dose is not a degraded success; it is the single worst thing the system can do, because it can travel through a busy clinician into a patient. The whole point of this section is to install a gate that says "no" on purpose.

That reframing has a name worth internalizing: abstention is a first-class, correct answer, not an error path. Compare the two failure directions honestly. If the assistant abstains when it actually could have helped, the cost is a small annoyance — the clinician goes to another trusted source they'd have used anyway. If the assistant answers when it shouldn't have, the cost is a fabricated clinical claim wearing the costume of a cited fact. Those costs are wildly asymmetric, so the gate is deliberately tuned toward the cheap failure. "I don't have a grounded answer in the approved sources; consult the on-call pharmacist" is the system succeeding, not giving up.

Mechanically the gate is almost embarrassingly simple, and its simplicity is a feature — a safety control you can't explain in one sentence is a safety control you can't trust. It rests on the confidence signal from §5: if the top retrieval score sits below a threshold (or nothing was retrieved), abstain; otherwise assemble the answer using only retrieved chunk text, with a citation appended to every sentence so each claim is traceable. That threshold is the one real dial in the whole system, and here is the trap: it is not an engineering parameter you tune for accuracy on a dashboard. It's a clinical risk decision. Set it too low and the demo below will happily "answer" an off-topic question from an irrelevant chunk — you'll see exactly that when you drop it to 0.05. So the final value is tuned against a labelled evaluation set and signed off with the clinical owners, not chosen by whoever wrote the code. The engineer builds the gate; the clinicians decide where it sits.

Now assemble a grounded answer and gate it. Two rules make this safe: every sentence carries a citation to the chunk it came from, and if the top retrieval score is below a threshold, the assistant abstains rather than stitching together a plausible-sounding guess. The generation here is faked with deterministic logic so it runs offline; a real system calls Claude with the retrieved chunks in context, instructs it to answer only from them and to say it doesn't know otherwise, and returns the result as structured output — verify in Anthropic's current docs.

ground.py# Grounded-answer assembly with mandatory citations + a confidence gate.
# DEMONSTRATION ONLY: a real system calls Claude with the retrieved chunks in
# context and instructs it to answer ONLY from them and cite chunk ids. Anthropic
# publicly documents grounding-in-context, structured/JSON output via tool use,
# and a `refusal` stop reason -- verify in Anthropic's current docs. Here we fake
# the generation with deterministic logic so it runs offline.

CHUNKS = {
    "PROTO-CKD-07": ("Metformin is contraindicated when eGFR is below 30. "
                     "Between 30 and 45, do not initiate."),
    "REF-METFORMIN-1": ("Renal function should be assessed before starting "
                        "metformin and at least annually."),
}

CONF_THRESHOLD = 0.35   # tune against your eval set; confirm with clinical owners

def generate(question, retrieved):
    """retrieved: list of (score, chunk_id). Returns a dict the UI/audit consumes.
    The 'answer' is assembled ONLY from retrieved chunk text; every claim carries
    a citation. If top score is below the gate, we ABSTAIN instead of guessing."""
    if not retrieved or retrieved[0][0] < CONF_THRESHOLD:
        return {
            "answer": ("I don't have a grounded answer in the approved sources. "
                       "Please consult the on-call pharmacist."),
            "citations": [],
            "abstained": True,
            "top_score": retrieved[0][0] if retrieved else 0.0,
        }
    cites = [cid for _, cid in retrieved if cid in CHUNKS]
    sentences = [CHUNKS[c] + f" [{c}]" for c in cites]
    return {
        "answer": " ".join(sentences),
        "citations": cites,
        "abstained": False,
        "top_score": retrieved[0][0],
    }

on_topic = generate("metformin and low eGFR", [(0.44, "PROTO-CKD-07"), (0.08, "REF-METFORMIN-1")])
off_topic = generate("what wine pairs with salmon", [(0.02, "PROTO-SEPSIS-03")])

print("ON-TOPIC abstained:", on_topic["abstained"], "| cites:", on_topic["citations"])
print("ANSWER:", on_topic["answer"])
print("OFF-TOPIC abstained:", off_topic["abstained"], "| cites:", off_topic["citations"])
print("ANSWER:", off_topic["answer"])

ON-TOPIC abstained: False | cites: ['PROTO-CKD-07', 'REF-METFORMIN-1']
ANSWER: Metformin is contraindicated when eGFR is below 30. Between 30 and 45, do not initiate. [PROTO-CKD-07] Renal function should be assessed before starting metformin and at least annually. [REF-METFORMIN-1]
OFF-TOPIC abstained: True | cites: []
ANSWER: I don't have a grounded answer in the approved sources. Please consult the on-call pharmacist.

7 · Evaluation — faithfulness, citation, abstention production

"It looked great when I tried it" is how unsafe systems get shipped. You type ten questions, the answers are cited and sensible, everyone's impressed, and you go live — never noticing that on the eleventh kind of question it invents a dose. The problem with judging a probabilistic system by hand is that your test set is tiny, biased toward questions you already know it handles, and gone the moment the demo ends. You cannot tell "usually right" from "right on the things I happened to ask." For a system where a rare wrong answer can reach a patient, "usually right" is not a safety claim — it's a hope.

Evaluation replaces the hope with a repeatable, automated exam. The mental shift is from tasting the soup once to running the same graded test on every change — every new prompt, model version, or corpus edit re-sits the exam, and a regression shows up as a lower score instead of a production incident weeks later. And the exam here has an unusual property most software tests lack: some of its questions are designed to be failed by bad behavior. You don't only check "does the good answer pass?"; you feed it a hallucinated answer and confirm the harness catches it. A test suite that only ever goes green on good input can't tell you it would catch a bad one.

Three gates cover the failure modes that matter, and it's worth understanding what each actually measures — because the common mistake is collapsing them into one fuzzy "quality" number. Faithfulness asks: does the answer stay inside its cited sources, or did it smuggle in a claim from nowhere? (The demo proxies this with token overlap; production uses an LLM-as-judge on a held-out labelled set.) Citation-presence asks the cheaper, non-negotiable question: does every in-scope answer actually cite a source a human could open? Abstention asks the inverted question: when handed something out-of-scope, does the system correctly decline? These are separate because they fail separately — an answer can be perfectly cited and still unfaithful, or faithful but silently uncited. One last discipline the code hints at: keep a held-out set you never tune against, so a rising score reflects genuine quality rather than you slowly overfitting to the questions you happened to write.

You cannot ship a safety-critical assistant on vibes. Build an offline eval harness with three gates: faithfulness (does the answer stay within its sources?), citation-presence (does every in-scope answer cite?), and abstention (does it correctly decline out-of-scope questions?). The scoring below uses token-overlap and simple rules for a deterministic offline run; in production, use an LLM-as-judge for faithfulness on a held-out labelled set — verify metric choices against Anthropic's current eval guidance. Notice the harness is designed to catch a hallucination, so one case fails on purpose.

evals.py# Offline eval harness: faithfulness, citation-presence, and abstention on
# out-of-scope. DEMONSTRATION scoring uses token-overlap and simple rules; in
# production use an LLM-as-judge for faithfulness and a held-out labelled set.
# Verify metric choices against Anthropic's current eval guidance.
import re

def toks(s): return set(re.findall(r"[a-z0-9]+", s.lower()))

def faithfulness(answer, context):
    """Fraction of answer content-tokens supported by the retrieved context.
    A proxy for 'no claims beyond the sources'. 1.0 = fully grounded."""
    a, c = toks(answer), toks(context)
    a -= {"the","a","and","is","of","to","in","if","at","be"}  # stopwords
    if not a: return 1.0
    return round(len(a & c) / len(a), 2)

def has_citation(answer):
    return bool(re.search(r"\[[A-Z0-9\-]+\]", answer))

def abstained(answer):
    return "don't have a grounded answer" in answer.lower()

CASES = [
    {"q": "metformin low eGFR", "in_scope": True,
     "answer": "Metformin is contraindicated when eGFR is below 30. [PROTO-CKD-07]",
     "context": "Metformin is contraindicated when eGFR is below 30 do not initiate"},
    {"q": "wine with salmon", "in_scope": False,
     "answer": "I don't have a grounded answer in the approved sources.",
     "context": ""},
    {"q": "hallucinated dose", "in_scope": True,
     "answer": "Give 2000mg metformin nightly regardless of kidney function.",
     "context": "Metformin is contraindicated when eGFR is below 30"},
]

passed = 0
for c in CASES:
    if c["in_scope"]:
        f = faithfulness(c["answer"], c["context"])
        cite = has_citation(c["answer"])
        ok = f >= 0.5 and cite
        tag = "PASS" if ok else "FAIL"
        print(f"[{tag}] {c['q']:<20} faith={f} cite={cite}")
    else:
        ok = abstained(c["answer"])
        tag = "PASS" if ok else "FAIL"
        print(f"[{tag}] {c['q']:<20} abstained={ok}")
    passed += ok
print(f"SCORE: {passed}/{len(CASES)} gates passed")

[PASS] metformin low eGFR   faith=0.67 cite=True
[PASS] wine with salmon     abstained=True
[FAIL] hallucinated dose    faith=0.14 cite=False
SCORE: 2/3 gates passed

8 · Failure modes & hardening — the fail-closed chain tech-lead

The difference between an engineer and a tech lead shows up in one question: not "does it work?" but "how does it hurt someone when it breaks, and what happens then?" Every safeguard you built so far — redaction, grounding, citations, the abstention gate — is real, and every one of them can fail individually. A NER redactor misses an oddly-formatted MRN. The model cites a chunk it was never handed. A weak retrieval slips just over the threshold. Betting the patient's safety on any single guard holding is how good systems produce bad outcomes. Hardening is the discipline of assuming each guard will fail sometime and arranging things so that when one does, the system still lands safe.

The key concept has a name borrowed from physical safety engineering: fail closed. A fail-open lock springs open when the power dies — convenient, and exactly wrong for a vault. A fail-closed lock stays shut. Software mostly fails open by habit ("if the check errored, just continue so the user isn't blocked"), and that habit is a liability here. So the defense chain below is built so that every link, when it fails or is even unsure, blocks rather than passes. Its best possible outcome is not "answer" — it's SENT_FOR_REVIEW. Notice what that means: the architecture is stating, in code, that a human decides. The system is structurally incapable of issuing a verdict on its own, which is a requirement in this domain, not a nicety you could refactor away.

Two design choices in the chain reward a second look because they encode real judgment. First, ordering matters and it's deliberate: the checks run in sequence and stop at the first failure, so the audit trail names the specific gate that blocked — reorder them and the reported reason changes, which is why the sequence is a design decision, not an accident. Second, and this is the subtle one people get backwards: the audit log must be both complete and PHI-free at the same time. It feels like a contradiction — how do you record what happened without recording who it happened to? The resolution is to log a salted hash of the user plus metadata (did it abstain? which chunks? what verdict?), never the raw question or answer. That's enough to reconstruct what happened for debugging and compliance without ever having stored who or what in the clear — the same redaction discipline from §3, applied one last time on the way out. Read the failure catalogue first, then watch the chain turn each of those failure modes into a named, logged, safe block.

A lead owns the ways this system can hurt someone and designs the defenses before the incident. Below is the failure catalogue, then a runnable defense chain that runs after generation and before any clinician sees a word — every gate fails closed (block, don't guess) and the final step writes a PHI-free audit record.

harden.py# Failure-mode hardening: a defense chain that runs AFTER generation and BEFORE
# anything reaches a clinician. Deterministic, stdlib only, no network. Each gate
# is a cheap check that fails CLOSED (block, don't guess). The final step writes a
# PHI-free audit record. This SUPPORTS a clinician's review; it never replaces
# clinical judgment.
import re, hashlib, json

def check_citations(resp):
    if resp["abstained"]:
        return True, "abstention is allowed"
    return (len(resp["citations"]) > 0), "answer must cite approved sources"

def check_grounding(resp, allowed_ids):
    bad = [c for c in resp["citations"] if c not in allowed_ids]
    return (not bad), f"cited non-retrieved chunk(s): {bad}" if bad else "ok"

def check_no_advice_language(resp):
    # block imperative dosing directives -- decision-support must present evidence,
    # not issue orders. (Illustrative pattern list.)
    banned = re.compile(r"\b(give|administer|prescribe|start)\b.*\b\d+\s?mg\b", re.I)
    hit = banned.search(resp["answer"])
    return (hit is None), "contains directive dosing language" if hit else "ok"

def audit_record(user, resp, verdict):
    # NEVER log raw PHI or the raw question -- log a salted hash + metadata only.
    return {
        "user_hash": hashlib.sha256(("salt::" + user).encode()).hexdigest()[:12],
        "abstained": resp["abstained"],
        "citations": resp["citations"],
        "verdict": verdict,
    }

def review_gate(user, resp, allowed_ids):
    for name, (ok, why) in {
        "citations": check_citations(resp),
        "grounding": check_grounding(resp, allowed_ids),
        "no_advice": check_no_advice_language(resp),
    }.items():
        if not ok:
            rec = audit_record(user, resp, f"BLOCKED:{name}")
            return "BLOCKED", name, rec
    rec = audit_record(user, resp, "SENT_FOR_REVIEW")
    return "SENT_FOR_REVIEW", "-", rec

good = {"answer": "Metformin is contraindicated below eGFR 30. [PROTO-CKD-07]",
        "citations": ["PROTO-CKD-07"], "abstained": False}
bad  = {"answer": "Give 2000 mg metformin nightly.",
        "citations": [], "abstained": False}
allowed = {"PROTO-CKD-07", "REF-METFORMIN-1"}

for label, resp in [("grounded", good), ("hallucinated", bad)]:
    verdict, gate, rec = review_gate("clinician-42", resp, allowed)
    print(f"{label:<14} -> {verdict} (gate: {gate})")
    print("   audit:", json.dumps(rec, sort_keys=True))

grounded       -> SENT_FOR_REVIEW (gate: -)
   audit: {"abstained": false, "citations": ["PROTO-CKD-07"], "user_hash": "099607778fd8", "verdict": "SENT_FOR_REVIEW"}
hallucinated   -> BLOCKED (gate: citations)
   audit: {"abstained": false, "citations": [], "user_hash": "099607778fd8", "verdict": "BLOCKED:citations"}

The chain never returns "answer" — its best outcome is SENT_FOR_REVIEW. That is the architecture stating, in code, that a human decides. The hallucinated response is blocked at the first gate (no citation) before the advice-language gate even runs; a defense chain fails at the earliest gate so the audit trail names the specific reason. And every branch — allow or block — writes an audit record built from a salted hash and metadata, so you can reconstruct what happened without ever having stored who or what in the clear.

User (inside trust boundary)
   |  question
   v
[Redactor] --- strips PHI ---> [Retriever: guideline doc index]  (in-region)
   |                                   |  top-k chunks (no PHI)
   v                                   v
[Prompt assembler] ----> [LLM inference (in-region / approved)] --> grounded answer + citations
   |
   v
[Abstention gate] -> answer OR "out of scope / see a clinician"

redaction uncertain  -> block external call
weak retrieval       -> abstain
out-of-scope (advice)-> refuse + refer to clinician
citation unverifiable-> downgrade to "can't answer"