Case study: clinical-knowledge RAG
A full end-to-end case study: a RAG assistant that helps clinicians find grounded, cited answers over vetted medical protocols and literature — under strict PII/PHI and data-residency constraints, with a confidence gate that abstains when it isn't sure, and a clinician in the loop on every answer. Requirements → architecture → design → code → evaluation → hardening. Decision-support only; never medical advice.
Learning objectives
- Translate a healthcare knowledge-assistant brief into hard requirements: PHI handling, data residency, grounding, citations, abstention, and decision-support-only scope.
- Choose a deployment posture — Claude API vs a region-locked managed cloud vs self-hosted open weights — from the privacy and residency constraints.
- Redact PII/PHI before anything leaves your trust boundary, and log what was stripped without ever storing the raw value.
- Ground every answer in a vetted corpus, force citations, and gate on a confidence threshold so the assistant abstains instead of guessing.
- Evaluate the system on faithfulness, citation-presence, and abstention on out-of-scope questions — and gate deploys on those metrics.
- Enumerate the failure modes of a clinical RAG assistant and design a fail-closed defense chain with a clinician in the loop and a PHI-free audit trail.
This is a full lifecycle walkthrough — requirements → architecture → design → code → evaluation → hardening — for one representative system: a clinical-knowledge RAG assistant. A clinician types a question ("is metformin safe at this patient's eGFR?"); the assistant retrieves from a vetted corpus of internal protocols and approved reference material, and returns a grounded, cited answer — or abstains when it isn't sure. Every design choice here is dominated by two forces you don't feel as sharply in a consumer chatbot: patient privacy and the fact that a wrong answer can hurt someone.
1 · Requirements — privacy, residency, and the scope line advanced
Before a single line of retrieval code, write down the constraints that will veto architectures. For a clinical assistant they are unusually sharp, and most of them are non-functional — they never appear as a feature, but they decide whether you can ship at all.
| Requirement | What it means here | Why it dominates design |
|---|---|---|
| PHI/PII protection | Protected health information (names, MRNs, dates, contact info) must not leak into logs, prompts, or third-party systems it isn't authorized for. | Forces a redaction boundary and PHI-free logging — before retrieval, before any model call. |
| Data residency | Data may be legally required to stay in a region/jurisdiction. | Vetoes deployment options; may force a region-locked or self-hosted model. |
| Grounding | Answers come only from a vetted corpus, not the model's parametric memory. | RAG is mandatory; the corpus is a governed asset, not a scrape. |
| Citations | Every claim is traceable to a source chunk a clinician can open. | Turns the model from an oracle into a librarian — reviewable by a human. |
| Abstention | "I don't know" is a first-class, correct answer when evidence is thin. | A confident wrong answer is the worst outcome; the gate must prefer silence. |
| Decision-support only | No diagnoses, no orders, no dosing directives — evidence for a human. | Shapes the output contract and the guardrails; enforced in code, not just the prompt. |
Two of these deserve care because people get them wrong. Data residency is a legal/contractual property of where bytes live and get processed, and it is distinct from encryption — verify the specifics with your compliance/legal team, and confirm current region availability and eligibility in Anthropic's current docs rather than assuming. And abstention is not a failure mode to be minimized to zero; in a safety-critical setting, a calibrated "I can't ground this — ask the on-call pharmacist" is a success.
2 · Architecture — the request path end to end advanced
Every clinician query flows through the same guarded pipeline. Read it left to right: nothing reaches a model until PHI is stripped, nothing reaches a clinician until it is grounded and gated, and nothing happens without an audit record.
This is the whole system in one line, and the order of the boxes is the safety design — read it left to right and notice what has to happen before what.
- Clinician query → PII/PHI redact: raw free text is cleaned first, so protected data never crosses your trust boundary into a model call or a log.
- → RAG over vetted corpus: retrieval is scoped to approved protocols and references, so the model can only speak from curated material — not its memory.
- → Grounded answer + citations: every claim is tied to a source chunk a clinician can open. The model becomes a librarian, not an oracle.
- → Confidence gate: if the evidence is thin, the pipeline abstains instead of guessing. "I don't know" is a first-class, correct answer here.
- → Clinician reviews → Audit log: a qualified human decides on every answer, and a PHI-free record is written so you can reconstruct what happened.
In short: Nothing reaches a model until PHI is stripped, and nothing reaches a patient without a clinician in the path. If you deleted the redact box or the review box, you'd have a different — and unsafe — system.
The ordering is the security design. Redaction is first so raw PHI never crosses your trust boundary. Retrieval is scoped to a vetted corpus so the model can only speak from approved material. Citations and the confidence gate sit between generation and the human, converting a fluent guess into either a reviewable evidence bundle or an honest abstention. The clinician is in the loop by construction, and the audit log records what happened without storing what it must not.
refusal stop reason and explicit "say you don't know" prompting. Deployment (next section) can use the Claude API or Claude on a managed cloud. Verify the exact feature names and availability in Anthropic's current docs.3 · Redaction — strip PHI before it leaves the boundary advanced
The first executable component is the redactor. Its contract: take free-text that may contain PHI, return text safe to send onward, and report what type of PHI was removed so the audit log can record it — without ever storing the raw value. The lab below is a deterministic, stdlib-only demonstration; production uses a vetted NER redactor (e.g. Presidio/spaCy) whose recall you measure against your own PHI test corpus.
redact.pyimport re
# A DEMONSTRATION PII/PHI redactor. Deterministic, stdlib-only, no ML, no network.
# In production use a vetted NER redactor (e.g. Presidio/spaCy); verify recall
# against your own PHI test corpus and confirm scope with your compliance/legal team.
PATTERNS = [
("MRN", re.compile(r"\bMRN[:#]?\s*\d{6,}\b", re.I)),
("DOB", re.compile(r"\bDOB[:#]?\s*\d{1,2}/\d{1,2}/\d{4}\b", re.I)),
("SSN", re.compile(r"\b\d{3}-\d{2}-\d{4}\b")),
("PHONE", re.compile(r"\b\d{3}[-.]\d{3}[-.]\d{4}\b")),
("EMAIL", re.compile(r"\b[\w.]+@[\w.]+\.\w+\b")),
("NAME", re.compile(r"\b(?:Mr|Ms|Mrs|Dr)\.?\s+[A-Z][a-z]+")),
]
def redact(text):
"""Replace recognised PHI spans with typed placeholders. Returns
(clean_text, spans_removed) so an audit log can record WHAT was stripped
by type without ever storing the raw PHI value."""
removed = []
clean = text
for label, rx in PATTERNS:
clean, n = rx.subn(f"[{label}]", clean)
removed += [label] * n
return clean, sorted(removed)
def has_raw_phi(text):
"""Belt-and-suspenders: would ANY pattern still fire on the cleaned text?"""
return any(rx.search(text) for _, rx in PATTERNS)
q = ("Mr. Alvarez (MRN: 40021991, DOB 03/14/1958) asks whether metformin "
"is safe with his eGFR; reach him at 415-555-0198 or a.alvarez@example.com.")
clean, removed = redact(q)
print("CLEAN :", clean)
print("REMOVED:", removed)
print("SAFE_TO_SEND:", not has_raw_phi(clean))
CLEAN : [NAME] ([MRN], [DOB]) asks whether metformin is safe with his eGFR; reach him at [PHONE] or [EMAIL].
REMOVED: ['DOB', 'EMAIL', 'MRN', 'NAME', 'PHONE']
SAFE_TO_SEND: TrueThis is the redactor — the first executable guard. Its job is to take text that might contain PHI and return text safe to send onward, while reporting what kind of PHI it removed (never the raw value).
PATTERNSis a list of(label, regex)pairs. Each regex recognises one PHI type — an MRN, a date of birth, a phone, an email, a titled name.redact()walks the patterns and usesrx.subn(...)to replace every match with a typed placeholder like[MRN], counting how many it swapped so it can report what was stripped by type.has_raw_phi()is a second, independent check: after cleaning, would any pattern still fire? If yes, the text is not safe to send. This belt-and-suspenders style is how you fail closed.
What the output means: The cleaned sentence keeps the clinical content ("metformin", "eGFR") but swaps every identifier for a [TYPE] tag; REMOVED lists the types found; SAFE_TO_SEND is True only because no pattern matches the cleaned text.
Try this: Add a new fake MRN in a different format (say MRN 12, too short) and watch it slip through — that miss is exactly why production needs a measured NER redactor, not hand-written regexes.
4 · Model sourcing — API vs region-locked vs self-hosted professional
With residency and PHI constraints written down, the deployment choice is largely decided for you. This is a compliance-and-cost decision, not a preference. Anthropic's public deployment surface — at a general level — includes the Claude API and Claude on managed clouds (Amazon Bedrock, Google Vertex AI) that offer region selection and enterprise data controls; open-weight models are an option when egress is forbidden entirely. Exact region availability, HIPAA/BAA eligibility, and data-retention terms change — verify in Anthropic's current docs and confirm with your compliance/legal team.
sourcing.py# Model-sourcing decision for a PHI workload. Deterministic, stdlib only.
# The options below reflect Anthropic's PUBLIC deployment surface at a general
# level: the Claude API, and Claude via managed clouds (Amazon Bedrock, Google
# Vertex AI) that offer region selection and enterprise data controls. Exact
# region availability, BAA/HIPAA eligibility, and zero-retention terms change --
# verify in Anthropic's current docs and confirm with your compliance/legal team.
def choose_deployment(req):
"""req: dict of requirements. Returns (deployment, rationale)."""
if req["phi_in_prompt"] and not req["redaction_guaranteed"]:
return ("BLOCK", "Do not send un-redacted PHI to any external endpoint.")
if req["data_residency"] and req["needs_frontier_quality"]:
return ("Claude via region-locked managed cloud (Bedrock/Vertex)",
"Keeps data in-region under an enterprise agreement while using a frontier model.")
if req["air_gapped"]:
return ("Self-hosted open-weight model",
"No egress permitted; accept a capability trade-off and own the ops burden.")
return ("Claude API with a signed enterprise agreement",
"Simplest path when region-locking is not mandated.")
scenarios = [
{"name": "region-locked frontier", "phi_in_prompt": True, "redaction_guaranteed": True,
"data_residency": True, "needs_frontier_quality": True, "air_gapped": False},
{"name": "raw PHI, no redaction", "phi_in_prompt": True, "redaction_guaranteed": False,
"data_residency": True, "needs_frontier_quality": True, "air_gapped": False},
{"name": "air-gapped hospital net", "phi_in_prompt": True, "redaction_guaranteed": True,
"data_residency": True, "needs_frontier_quality": False, "air_gapped": True},
]
for s in scenarios:
dep, why = choose_deployment(s)
print(f"{s['name']:<24} -> {dep}")
region-locked frontier -> Claude via region-locked managed cloud (Bedrock/Vertex)
raw PHI, no redaction -> BLOCK
air-gapped hospital net -> Self-hosted open-weight modelThis encodes the deployment decision as logic driven by constraints, not preference. Read the branches top to bottom — the order is a priority order.
- The first check is the veto: if PHI would go in the prompt and redaction isn't guaranteed, it returns
"BLOCK". No endpoint choice can override an un-met privacy requirement. - Next, data residency + frontier quality routes to Claude on a region-locked managed cloud (Bedrock/Vertex) — keep data in-region while still using a strong model.
- Air-gapped networks (no egress at all) fall to a self-hosted open-weight model, accepting a capability trade and the ops burden.
- The final line is the simple default: the Claude API under an enterprise agreement when region-locking isn't mandated.
What the output means: Three scenarios print their routed deployment. The middle one returns BLOCK — the system refusing to proceed rather than picking a 'less bad' endpoint for un-redacted PHI.
Try this: Flip needs_frontier_quality to False on the first scenario and see how the route changes. Sourcing here is a compliance + cost function; verify the real region/BAA options in Anthropic's current docs.
5 · Grounded RAG with citations — the retriever professional
Retrieval is the heart of the system, and its job description is narrow: pull the most relevant chunks from a vetted corpus so the model can answer only from approved material. The demo below is a deterministic bag-of-words cosine over a tiny corpus of internal protocols plus reference snippets — stdlib only, no embeddings, no network. Production uses real embeddings and hybrid (dense + BM25) retrieval, and Anthropic publicly describes contextual retrieval (prepending a situating line to each chunk) to lift recall — verify in Anthropic's current docs.
retriever.pyimport re, math
from collections import Counter
# A DEMONSTRATION retriever over a VETTED corpus: internal protocols + a few
# clinical-reference snippets. Deterministic bag-of-words cosine, stdlib only,
# no embeddings, no network. Production uses real embeddings + hybrid BM25 and
# (per Anthropic's public "contextual retrieval" guidance) a situating context
# line per chunk -- verify in Anthropic's current docs.
CORPUS = [
{"id": "PROTO-CKD-07", "src": "Renal Dosing Protocol v7",
"text": "Metformin is contraindicated when eGFR is below 30 mL/min/1.73m2. "
"Between 30 and 45, do not initiate; review dose if already prescribed."},
{"id": "PROTO-SEPSIS-03", "src": "Sepsis Bundle v3",
"text": "Begin broad-spectrum antibiotics within one hour of recognising "
"sepsis and obtain blood cultures before the first dose."},
{"id": "REF-METFORMIN-1", "src": "Formulary Reference",
"text": "Metformin lowers hepatic glucose output. Renal function should be "
"assessed before starting and at least annually thereafter."},
]
def toks(s): return re.findall(r"[a-z0-9]+", s.lower())
def vec(s):
c = Counter(toks(s))
n = math.sqrt(sum(v*v for v in c.values())) or 1.0
return {k: v/n for k, v in c.items()}
def cosine(a, b):
return sum(a.get(k, 0.0)*b.get(k, 0.0) for k in a)
INDEX = [(d, vec(d["text"])) for d in CORPUS]
def retrieve(query, k=2):
qv = vec(query)
scored = [(round(cosine(qv, dv), 3), d) for d, dv in INDEX]
scored.sort(key=lambda x: x[0], reverse=True)
return scored[:k]
hits = retrieve("Is metformin safe if eGFR is low?", k=2)
for score, d in hits:
print(f"{score:>5} {d['id']:<16} {d['src']}")
0.441 PROTO-CKD-07 Renal Dosing Protocol v7
0.081 REF-METFORMIN-1 Formulary ReferenceThis is a tiny, deterministic stand-in for a real retriever: it turns the corpus and the query into word-frequency vectors and ranks chunks by cosine similarity. Real systems swap the vectors for learned embeddings, but the shape is identical.
CORPUSis the vetted shelf: a few internal protocols and references, each with anidand asrcso a citation can point at an exact, auditable source.vec()builds a normalized bag-of-words vector;cosine()scores how much a chunk's words overlap the query's.INDEXpre-computes one vector per chunk so each query only vectorizes the question.retrieve(query, k=2)scores every chunk, sorts high-to-low, and returns the topkwith their scores — the raw material the next step grounds on.
What the output means: The renal-dosing protocol scores far higher (0.441) than the generic formulary reference (0.081) for the eGFR question — the retriever surfaced the right chunk to answer from.
Try this: Change the query to something the corpus can't answer ("blood pressure targets") and watch every score collapse toward zero — that low top-score is precisely the signal the confidence gate uses to abstain.
v7) so a citation points at an exact revision a clinician can audit.6 · Abstention — the confidence gate professional
Now assemble a grounded answer and gate it. Two rules make this safe: every sentence carries a citation to the chunk it came from, and if the top retrieval score is below a threshold, the assistant abstains rather than stitching together a plausible-sounding guess. The generation here is faked with deterministic logic so it runs offline; a real system calls Claude with the retrieved chunks in context, instructs it to answer only from them and to say it doesn't know otherwise, and returns the result as structured output — verify in Anthropic's current docs.
ground.py# Grounded-answer assembly with mandatory citations + a confidence gate.
# DEMONSTRATION ONLY: a real system calls Claude with the retrieved chunks in
# context and instructs it to answer ONLY from them and cite chunk ids. Anthropic
# publicly documents grounding-in-context, structured/JSON output via tool use,
# and a `refusal` stop reason -- verify in Anthropic's current docs. Here we fake
# the generation with deterministic logic so it runs offline.
CHUNKS = {
"PROTO-CKD-07": ("Metformin is contraindicated when eGFR is below 30. "
"Between 30 and 45, do not initiate."),
"REF-METFORMIN-1": ("Renal function should be assessed before starting "
"metformin and at least annually."),
}
CONF_THRESHOLD = 0.35 # tune against your eval set; confirm with clinical owners
def generate(question, retrieved):
"""retrieved: list of (score, chunk_id). Returns a dict the UI/audit consumes.
The 'answer' is assembled ONLY from retrieved chunk text; every claim carries
a citation. If top score is below the gate, we ABSTAIN instead of guessing."""
if not retrieved or retrieved[0][0] < CONF_THRESHOLD:
return {
"answer": ("I don't have a grounded answer in the approved sources. "
"Please consult the on-call pharmacist."),
"citations": [],
"abstained": True,
"top_score": retrieved[0][0] if retrieved else 0.0,
}
cites = [cid for _, cid in retrieved if cid in CHUNKS]
sentences = [CHUNKS[c] + f" [{c}]" for c in cites]
return {
"answer": " ".join(sentences),
"citations": cites,
"abstained": False,
"top_score": retrieved[0][0],
}
on_topic = generate("metformin and low eGFR", [(0.44, "PROTO-CKD-07"), (0.08, "REF-METFORMIN-1")])
off_topic = generate("what wine pairs with salmon", [(0.02, "PROTO-SEPSIS-03")])
print("ON-TOPIC abstained:", on_topic["abstained"], "| cites:", on_topic["citations"])
print("ANSWER:", on_topic["answer"])
print("OFF-TOPIC abstained:", off_topic["abstained"], "| cites:", off_topic["citations"])
print("ANSWER:", off_topic["answer"])
ON-TOPIC abstained: False | cites: ['PROTO-CKD-07', 'REF-METFORMIN-1']
ANSWER: Metformin is contraindicated when eGFR is below 30. Between 30 and 45, do not initiate. [PROTO-CKD-07] Renal function should be assessed before starting metformin and at least annually. [REF-METFORMIN-1]
OFF-TOPIC abstained: True | cites: []
ANSWER: I don't have a grounded answer in the approved sources. Please consult the on-call pharmacist.This assembles the answer and gates it. Two safety rules live here: every sentence carries a citation, and a weak top score triggers abstention instead of a guess.
CONF_THRESHOLDis the dial between helpfulness and safety. The very first thinggenerate()does is check the top retrieval score against it.- If there are no hits or the top score is below the gate, it returns an abstention: an honest "I don't have a grounded answer… consult the on-call pharmacist," with empty citations and
abstained=True. - Otherwise it builds the answer only from retrieved chunk text, appending
[CHUNK-ID]to each sentence so every claim is traceable.
What the output means: The on-topic question produces a cited answer (abstained: False); the off-topic wine question is correctly refused (abstained: True, no citations). The generation is faked here so it runs offline — production calls Claude with the chunks in context.
Try this: Lower CONF_THRESHOLD to 0.05 and re-run: the off-topic question now gets 'answered' from an irrelevant chunk. That's the failure the gate exists to prevent — which is why you tune it toward caution with clinical owners.
7 · Evaluation — faithfulness, citation, abstention production
You cannot ship a safety-critical assistant on vibes. Build an offline eval harness with three gates: faithfulness (does the answer stay within its sources?), citation-presence (does every in-scope answer cite?), and abstention (does it correctly decline out-of-scope questions?). The scoring below uses token-overlap and simple rules for a deterministic offline run; in production, use an LLM-as-judge for faithfulness on a held-out labelled set — verify metric choices against Anthropic's current eval guidance. Notice the harness is designed to catch a hallucination, so one case fails on purpose.
evals.py# Offline eval harness: faithfulness, citation-presence, and abstention on
# out-of-scope. DEMONSTRATION scoring uses token-overlap and simple rules; in
# production use an LLM-as-judge for faithfulness and a held-out labelled set.
# Verify metric choices against Anthropic's current eval guidance.
import re
def toks(s): return set(re.findall(r"[a-z0-9]+", s.lower()))
def faithfulness(answer, context):
"""Fraction of answer content-tokens supported by the retrieved context.
A proxy for 'no claims beyond the sources'. 1.0 = fully grounded."""
a, c = toks(answer), toks(context)
a -= {"the","a","and","is","of","to","in","if","at","be"} # stopwords
if not a: return 1.0
return round(len(a & c) / len(a), 2)
def has_citation(answer):
return bool(re.search(r"\[[A-Z0-9\-]+\]", answer))
def abstained(answer):
return "don't have a grounded answer" in answer.lower()
CASES = [
{"q": "metformin low eGFR", "in_scope": True,
"answer": "Metformin is contraindicated when eGFR is below 30. [PROTO-CKD-07]",
"context": "Metformin is contraindicated when eGFR is below 30 do not initiate"},
{"q": "wine with salmon", "in_scope": False,
"answer": "I don't have a grounded answer in the approved sources.",
"context": ""},
{"q": "hallucinated dose", "in_scope": True,
"answer": "Give 2000mg metformin nightly regardless of kidney function.",
"context": "Metformin is contraindicated when eGFR is below 30"},
]
passed = 0
for c in CASES:
if c["in_scope"]:
f = faithfulness(c["answer"], c["context"])
cite = has_citation(c["answer"])
ok = f >= 0.5 and cite
tag = "PASS" if ok else "FAIL"
print(f"[{tag}] {c['q']:<20} faith={f} cite={cite}")
else:
ok = abstained(c["answer"])
tag = "PASS" if ok else "FAIL"
print(f"[{tag}] {c['q']:<20} abstained={ok}")
passed += ok
print(f"SCORE: {passed}/{len(CASES)} gates passed")
[PASS] metformin low eGFR faith=0.67 cite=True
[PASS] wine with salmon abstained=True
[FAIL] hallucinated dose faith=0.14 cite=False
SCORE: 2/3 gates passedThis is the offline eval harness — three gates that decide whether a build is safe to ship. It's deliberately built to catch a hallucination, so one case fails on purpose.
faithfulness()measures the fraction of the answer's content words that appear in the retrieved context — a cheap proxy for "no claims beyond the sources." Production uses an LLM-as-judge instead of token overlap.has_citation()checks an in-scope answer actually cites a[SRC];abstained()checks an out-of-scope answer correctly declined.- The loop applies the right gates per case: in-scope answers must be faithful and cited; out-of-scope answers must abstain. It tallies a score at the end.
What the output means: The grounded answer passes, the out-of-scope wine question passes (it abstained), and the invented-dose answer fails — low faithfulness, no citation — so the score is 2/3. That failure is the harness working.
Try this: Add a case where the answer cites a source but states something the context doesn't support, and watch faithfulness catch it. Then imagine this running in CI, failing the deploy on any regression against a held-out set.
8 · Failure modes & hardening — the fail-closed chain tech-lead
A lead owns the ways this system can hurt someone and designs the defenses before the incident. Below is the failure catalogue, then a runnable defense chain that runs after generation and before any clinician sees a word — every gate fails closed (block, don't guess) and the final step writes a PHI-free audit record.
| Failure mode | What it looks like | Hardening |
|---|---|---|
| Hallucinated fact | A fluent claim with no support in the corpus. | Faithfulness eval + a runtime grounding gate; block answers citing non-retrieved chunks. |
| Missing / fake citation | An answer with no [SRC], or a citation to a chunk never retrieved. | Require ≥1 citation on non-abstentions; verify each cited id was actually retrieved. |
| Overconfident on out-of-scope | Answers a question the corpus can't support. | Confidence gate tuned toward abstention; abstention eval in CI. |
| Scope creep into advice | Emits an imperative order ("give 2000 mg…"). | Block directive dosing language; keep the product decision-support only, in code. |
| PHI leak into logs | Raw names/MRNs written to an audit or trace store. | Redact before egress; log a salted hash + metadata, never the raw text. |
| Stale corpus | A retired protocol still answers as current. | Version chunks; expire/replace on protocol updates; cite the exact revision. |
harden.py# Failure-mode hardening: a defense chain that runs AFTER generation and BEFORE
# anything reaches a clinician. Deterministic, stdlib only, no network. Each gate
# is a cheap check that fails CLOSED (block, don't guess). The final step writes a
# PHI-free audit record. This SUPPORTS a clinician's review; it never replaces
# clinical judgment.
import re, hashlib, json
def check_citations(resp):
if resp["abstained"]:
return True, "abstention is allowed"
return (len(resp["citations"]) > 0), "answer must cite approved sources"
def check_grounding(resp, allowed_ids):
bad = [c for c in resp["citations"] if c not in allowed_ids]
return (not bad), f"cited non-retrieved chunk(s): {bad}" if bad else "ok"
def check_no_advice_language(resp):
# block imperative dosing directives -- decision-support must present evidence,
# not issue orders. (Illustrative pattern list.)
banned = re.compile(r"\b(give|administer|prescribe|start)\b.*\b\d+\s?mg\b", re.I)
hit = banned.search(resp["answer"])
return (hit is None), "contains directive dosing language" if hit else "ok"
def audit_record(user, resp, verdict):
# NEVER log raw PHI or the raw question -- log a salted hash + metadata only.
return {
"user_hash": hashlib.sha256(("salt::" + user).encode()).hexdigest()[:12],
"abstained": resp["abstained"],
"citations": resp["citations"],
"verdict": verdict,
}
def review_gate(user, resp, allowed_ids):
for name, (ok, why) in {
"citations": check_citations(resp),
"grounding": check_grounding(resp, allowed_ids),
"no_advice": check_no_advice_language(resp),
}.items():
if not ok:
rec = audit_record(user, resp, f"BLOCKED:{name}")
return "BLOCKED", name, rec
rec = audit_record(user, resp, "SENT_FOR_REVIEW")
return "SENT_FOR_REVIEW", "-", rec
good = {"answer": "Metformin is contraindicated below eGFR 30. [PROTO-CKD-07]",
"citations": ["PROTO-CKD-07"], "abstained": False}
bad = {"answer": "Give 2000 mg metformin nightly.",
"citations": [], "abstained": False}
allowed = {"PROTO-CKD-07", "REF-METFORMIN-1"}
for label, resp in [("grounded", good), ("hallucinated", bad)]:
verdict, gate, rec = review_gate("clinician-42", resp, allowed)
print(f"{label:<14} -> {verdict} (gate: {gate})")
print(" audit:", json.dumps(rec, sort_keys=True))
grounded -> SENT_FOR_REVIEW (gate: -)
audit: {"abstained": false, "citations": ["PROTO-CKD-07"], "user_hash": "099607778fd8", "verdict": "SENT_FOR_REVIEW"}
hallucinated -> BLOCKED (gate: citations)
audit: {"abstained": false, "citations": [], "user_hash": "099607778fd8", "verdict": "BLOCKED:citations"}This is the runtime defense chain — it runs after generation and before any clinician sees a word. Every gate fails closed (block, don't guess), and the best possible outcome is SENT_FOR_REVIEW, never a verdict.
- Each
check_*function returns(ok, why): citations must be present on non-abstentions; cited ids must have actually been retrieved (allowed_ids); and no imperative dosing language is allowed. review_gate()runs the checks in order and stops at the first failure, recording the specific gate name — so the audit trail says exactly why something was blocked.audit_record()writes a PHI-free trace: a salted hash of the user plus metadata (abstained?, citations, verdict) — never the raw question or answer.
What the output means: The grounded response reaches SENT_FOR_REVIEW; the hallucinated "give 2000 mg" response is BLOCKED at the first gate (no citation) before the advice-language gate even runs. Both write an audit record with the same hashed user id and no PHI.
Try this: Reorder the checks so no_advice runs first and see the blocked reason change from citations to no_advice. The chain always fails at the earliest gate — which is why ordering is a design choice, not an accident.
The chain never returns "answer" — its best outcome is SENT_FOR_REVIEW. That is the architecture stating, in code, that a human decides. The hallucinated response is blocked at the first gate (no citation) before the advice-language gate even runs; a defense chain fails at the earliest gate so the audit trail names the specific reason. And every branch — allow or block — writes an audit record built from a salted hash and metadata, so you can reconstruct what happened without ever having stored who or what in the clear.
| Dimension | Meets the bar | Above the bar (tech-lead) |
|---|---|---|
| Privacy boundary | PHI is redacted before any egress; logs carry no raw PHI. | Redactor recall is measured against a PHI test corpus; egress fails closed when text can't be cleaned; residency confirmed with legal. |
| Grounding & citations | Answers come only from a vetted corpus; every in-scope answer cites a source. | Citations point to versioned chunk revisions; a runtime gate blocks answers citing non-retrieved chunks. |
| Abstention | A confidence gate abstains on thin evidence instead of guessing. | Threshold is tuned on a labelled set with clinical owners and evaluated in CI; abstention framed to users as correct. |
| Evaluation | Faithfulness, citation, and abstention are measured offline. | Held-out set never tuned against; deploys gated on regression; LLM-as-judge for faithfulness in prod. |
| Human in the loop | A clinician reviews every answer; the system issues no orders. | Advice-language is blocked in code; the pipeline is structurally incapable of a verdict; audit trail is PHI-free and complete. |
| Honesty about limits | Decision-support-only scope is explicit to users and in code. | Regulatory claims are deferred to compliance/legal; Anthropic features cited at a verifiable, general level. |
Score each row 0 (absent) / 1 (meets) / 2 (above). A clinical assistant that scores below 1 on Privacy boundary or Human in the loop is not shippable at any total — those two are gates, not points.
The retrieval confidence score for a clinician's question comes back just below your gate threshold. What should the assistant do, and why is that the correct behavior rather than a failure?
Show answer
A teammate proposes logging the full clinician question and the model's raw answer to a trace store "so we can debug retrieval." Name two things wrong with that in this system, and what to do instead.
Show answer
Exercise CS4.1 — Move the abstention gate
Context: Moving the confidence threshold is the single most consequential knob in a clinical assistant, and it's a safety decision, not an engineering one. Watching a case flip makes the helpfulness-vs-safety trade concrete.
Your task: Take ground.py and evals.py together, raise CONF_THRESHOLD from 0.35 to 0.50, re-run both, and identify which case flips — then write two sentences on the trade you made and say which direction you'd err and who should sign off on the final value.
Requirements:
- Raise the threshold, re-run, and identify the case that flips from answered to abstained
- Explain the helpfulness-vs-safety trade the change represents in two sentences
- State which direction you'd err in this clinical context (toward caution / abstention)
- Name who owns the final value — the clinical owners, not engineering intuition
- Tie it to the rule that a confident wrong answer can reach a patient while a miss just routes to another source
💡 Hint: Focus on the case that flips and what its flip costs — a missed answer versus a confident wrong one.
Exercise CS4.2 — Add a failure mode
Context: A stale-corpus answer that cites a retired protocol is a silent error the current chain doesn't catch. Adding a gate for it — keeping earliest-gate-wins ordering and a PHI-free record — is how the defense chain grows.
Your task: Pick a failure mode not yet gated in harden.py — e.g. a stale-corpus answer citing a retired protocol revision — add a check (and a fake "retired ids" set) that fails closed, then add a matching case to evals.py so CI would catch the regression.
Requirements:
- Add a new
check_*that fails closed on the chosen failure mode (e.g. a citation to a retired protocol id) - Supply a fake "retired ids" set the check consults
- Preserve the earliest-gate-wins ordering so the block reason names the specific gate
- Keep the audit record PHI-free (salted hash + metadata, never the raw text)
- Add a matching
evals.pycase so CI would catch the regression
💡 Hint: Turn the silent failure into a visible one — a block with a named reason — and lock it in with an eval row.
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: In a clinical setting the most important requirement is what the system must not do. A RAG assistant over guideline documents is decision-support — naming the scope line bounds liability and shapes everything downstream.
Your task: For a RAG assistant over internal clinical guideline documents (not a diagnostic tool), write three scope/privacy requirements that draw the safety line clearly.
Requirements:
- Scope line: it answers questions about the guideline documents; it does not give individualized medical advice or diagnoses
- Out-of-scope questions are refused with a pointer to a clinician
- Privacy (PHI): no patient health information leaves the trust boundary; inputs are redacted before any external model call
- Data residency: data and model inference stay within the required region/jurisdiction
- Note that naming what the system won't do is a first-class requirement in regulated domains
💡 Hint: Start from the scope line and the trust boundary — the redaction and abstention design follow from them.
Show solution
- Scope line: it answers questions about the guideline documents; it does not give individualized medical advice or diagnoses. Out-of-scope questions are refused with a pointer to a clinician.
- Privacy (PHI): no patient health information leaves the trust boundary; inputs are redacted before any external model call.
- Data residency: data and model inference stay within the required region/jurisdiction.
Naming what the system won’t do is a first-class requirement in regulated domains — it bounds liability and shapes the redaction and abstention design that follow.
Context: Every clinician query flows through the same guarded pipeline, and the order of the boxes is the safety design. Drawing where PHI can and cannot flow is the point of the diagram.
Your task: Design the path a question takes, making the trust boundary and the redaction point explicit.
Requirements:
- Place the redactor before anything that could leave the trust boundary — raw PHI is cleaned first
- Retrieve top-k chunks from the guideline index, carrying no PHI onward
- Run retrieval and inference in-region per the residency constraint
- Produce a grounded answer with citations, then pass it through an abstention gate as the last stop before a user sees text
- Draw the trust boundary as a box so reviewers can see exactly where PHI can and cannot flow
💡 Hint: The ordering is the security design: nothing reaches a model until PHI is stripped, nothing reaches a user without the gate.
Show solution
User (inside trust boundary)
| question
v
[Redactor] --- strips PHI ---> [Retriever: guideline doc index] (in-region)
| | top-k chunks (no PHI)
v v
[Prompt assembler] ----> [LLM inference (in-region / approved)] --> grounded answer + citations
|
v
[Abstention gate] -> answer OR "out of scope / see a clinician"The redactor sits before anything that could leave the boundary, and both retrieval and inference are in-region. The abstention gate is the last stop before a user sees text. Draw the boundary as a box on the diagram so reviewers can see exactly where PHI can and cannot flow.
Context: The redactor is the first executable guard, and its contract is strict: return text safe to send, report what type of PHI was removed, and never store the raw value — failing closed when uncertain.
Your task: Design the PHI redaction step and explain why you fail closed if redaction is uncertain.
Requirements:
- Detect PHI categories (names, MRNs, dates of birth, contact info) with pattern rules for structured identifiers plus a model/NER pass for free text
- Replace matches with typed placeholders (e.g.
[PATIENT_NAME]) so the question stays answerable without identity - Verify by re-scanning the redacted text; if residual PHI is found or the detector is low-confidence, fail closed and block egress
- Report what type was stripped for the audit log without ever storing the raw value
- Explain the fail-closed rationale: leaking PHI costs far more than refusing one question — the opposite of a UX-first default
💡 Hint: Treat redaction like a safety check, not a formatter: when it's unsure, the safe default is to not proceed.
Show solution
- Detect PHI categories (names, MRNs, dates of birth, contact info, etc.) with a detector — pattern rules for structured identifiers plus a model/NER pass for free text.
- Replace with typed placeholders (e.g.
[PATIENT_NAME]) so the question stays answerable without carrying identity. - Verify: re-scan the redacted text; if residual PHI is detected or the detector is low-confidence, fail closed — block the external call and route to an in-boundary path or a human.
Why fail closed: the cost of leaking PHI is far higher than the cost of refusing to answer one question. When the safety check is uncertain, the safe default is to not proceed — the opposite of a normal UX-first default.
Context: With PHI and residency constraints written down, the deployment choice is largely decided for you — it's a compliance-and-cost decision, not a preference. Under strict residency, one option wins clearly.
Your task: Compare API vs region-locked API vs self-hosted for the inference step under a residency constraint, present the tradeoff, and pick one for strict residency.
Requirements:
- Compare the options on residency control (weakest → strongest as you move to region-locked then self-hosted)
- Compare ops burden (lowest for API, highest for self-hosting: you run/patch/scale)
- Compare capability (frontier via API/region-locked; limited to what you can host if self-hosted) and the cost model
- For strict residency, prefer an in-region managed offering if one meets the requirement — frontier capability + residency without the ops burden
- Self-host only when no compliant managed option exists and you have the team to run it
- Verify the provider's current residency guarantees against the regulator's requirements — treat vendor claims as things to confirm
💡 Hint: Let the constraints do the choosing: the correct answer can even be BLOCK when a privacy requirement isn't met.
Show solution
| Standard API | Region-locked / in-region API | Self-hosted | |
|---|---|---|---|
| Residency control | Weakest | Strong (data stays in region) | Strongest (you own it) |
| Ops burden | Lowest | Low–moderate | Highest (you run/patch/scale) |
| Capability | Frontier | Frontier (where offered) | Limited to what you can host |
| Cost model | Per token | Per token | Fixed infra + expertise |
For strict residency: prefer an in-region managed offering if one meets the requirement — you keep frontier capability and residency without the ops burden of self-hosting. Self-host only when no compliant managed option exists and you have the team to run it. Verify the specific provider’s current residency guarantees against your regulator’s requirements; treat vendor claims as things to confirm.
Context: Retrieval is the heart of the system, and its safety comes from two rules: every sentence cites the chunk it came from, and a weak top score triggers abstention rather than a plausible-sounding guess.
Your task: Design the retriever + generation so answers are grounded in guideline docs, plus the gate that refuses out-of-scope or weakly-grounded questions.
Requirements:
- Retrieve top-k guideline chunks with source ids; if the top similarity is below a floor, treat it as no good grounding
- Generate with an instruction to answer only from the sources, cite them, and say so if they don't cover it
- Fire the abstention gate when retrieval is weak, OR the model abstained, OR the question is individualized medical advice (out of scope)
- Respond to abstention with an honest refusal that points to a clinician, not a guess
- Use two independent triggers (weak retrieval and model self-abstain) so the gate is robust to either failing
- Frame a calibrated refusal as a correct answer in this domain, not a product gap
💡 Hint: Make the gate lean toward caution and give it two independent reasons to abstain, since a confident wrong answer can reach a patient.
Show solution
- Retrieve top-k guideline chunks with source ids; if the top similarity is below a floor, treat as no good grounding.
- Generate with: “Answer only from the sources; cite them; if they don’t cover it, say so.”
- Abstention gate fires when: retrieval is weak, OR the model abstained, OR the question is individualized medical advice (out of scope). Response: “I can’t answer that from the guidelines — please consult a clinician,” not a guess.
Two independent triggers (weak retrieval and model self-abstain) make the gate robust to either failing. In this domain, a calibrated refusal is a correct answer, not a product gap.
Context: The final job is tying the safety mechanisms into one fail-closed chain where any failing link yields a safe refusal, and designing the eval whose safety metrics are hard gates, not averages you can trade away.
Your task: Tie the safety mechanisms into one fail-closed chain and design the eval that proves it, then state the release gate.
Requirements:
- Build a fail-closed chain: redaction uncertain → block the external call; weak retrieval → abstain
- Continue the chain: out-of-scope (advice) → refuse + refer to clinician; citation unverifiable → downgrade to "can't answer"
- Measure faithfulness (claims supported by cited guideline text) and citation validity (cited chunk exists & contains the claim — must be 100%)
- Measure abstention recall (out-of-scope/ungrounded questions correctly refused) and PHI-leak rate (residual PHI reaching the external call — must be 0)
- Set the release gate: block on any PHI leak, on citation validity < 100%, or on a drop in abstention recall
- Treat safety metrics as hard gates, not averages you can trade against a quality gain
💡 Hint: Design every link so its failure produces a safe refusal, and let no quality improvement buy back a safety regression.
Show solution
Fail-closed chain (any link failing → safe refusal, never an unsafe answer):
redaction uncertain -> block external call
weak retrieval -> abstain
out-of-scope (advice)-> refuse + refer to clinician
citation unverifiable-> downgrade to "can't answer"| Metric | Definition |
|---|---|
| Faithfulness | claims supported by cited guideline text |
| Citation validity | cited chunk exists & contains the claim (must be 100%) |
| Abstention recall | % of out-of-scope / ungrounded questions correctly refused |
| PHI-leak rate | redaction test set: residual PHI reaching the external call (must be 0) |
Release gate: block on any PHI leak, on citation validity < 100%, or on a drop in abstention recall. Safety metrics are hard gates, not averages you can trade against a quality gain.
✓ Checkpoint — you can move on when you can…
- List the requirements that dominate a clinical RAG design and say why abstention and decision-support-only are non-negotiable.
- Draw the request path: query → redact → RAG → grounded answer + citations → confidence gate → clinician review → audit.
- Justify a model-sourcing choice from PHI + residency constraints, and know when the correct answer is BLOCK.
- Explain why redaction is first, citations are mandatory, and the confidence gate is tuned toward caution.
- Evaluate the system on faithfulness, citation, and abstention, and gate deploys on regression against a held-out set.
- Design a fail-closed defense chain with a clinician in the loop and a PHI-free audit trail — and say, honestly, where you defer to compliance/legal and to Anthropic's current docs.