Regulated-industry AI system

Meet "Meridian Bank" — a representative composite. Meridian wants an AI assistant that helps loan officers decide consumer-credit applications faster: it reads the application and the supporting documents, retrieves the relevant policy, and drafts a recommendation with a rationale. The catch is the whole reason this is a case study: in a regulated setting the assistant is not allowed to be a black box. A declined applicant may be entitled to an explanation; a regulator may ask you to reconstruct why a decision was made; and a human — not the model — must own the outcome. Every design choice below is dominated by three forces a consumer chatbot never feels: you must be able to explain it, you must be able to audit it, and a human must stay accountable for it.

1 · Requirements & the constraints that veto architectures

Picture the demo that goes beautifully and then dies in the review room. Your team builds a slick lending assistant, the accuracy numbers look great, the loan officers love the speed — and then compliance asks four questions and the whole thing is dead on arrival. "Can you show a declined applicant the reasons?" "Can you reconstruct a decision from last March?" "Who, exactly, is accountable when the model is wrong?" "Where does the applicant's data live, and for how long?" Nobody wrote those questions down before building, so the answers are all "no," and the elegant architecture has to be torn out. This is the failure a regulated project lives in fear of, and §1 exists to prevent it.

Here is the mental shift that separates a consumer product from a regulated one. In a normal app you start with features — what the user can do — and worry about the boring stuff later. In a regulated industry you invert that: you start with the non-functional constraints, because they don't merely shape the design, they veto whole classes of it before a single feature is on the table. Think of them like the building codes an architect must satisfy before drawing a single beautiful room: fire exits, load limits, accessibility. You don't add fire exits after the house is built; if you try, you rebuild the house. A constraint like "a human must own every decision" is a load-bearing wall — decide it late and you demolish everything resting on the wrong assumption.

So the discipline is simple to state and hard to practice: write the constraints down first, in plain language, before anyone sketches an architecture. The reason it's hard is that these requirements are nearly invisible. They almost never show up as something a user clicks; they surface only when a regulator, an auditor, or a lawsuit comes asking. That invisibility is exactly why teams skip them — and exactly why skipping them is the most expensive mistake in the project. The common error is to treat auditability or explainability as "we'll add logging later." They aren't features you bolt on; they are properties the whole system either has by construction or can never fully retrofit. The table below is that written-down list for Meridian: read each row as a veto power, and ask of any design you later propose, "which of these six does this choice quietly violate?"

2 · Architecture — the decision path end to end

Here is where a regulated architecture stops looking like the diagrams you're used to. If you've built a consumer chatbot or an internal tool, your instinct is to let the model be clever: hand it the request, give it some tools, and let it figure out the path — retrieve what it needs, reason freely, answer fast. That instinct is exactly wrong here, and it's worth feeling why before we look at the boxes. The moment a free-running agent decides its own steps, you lose the two things this whole system is built to guarantee: you can no longer say with certainty which stage produced what, and you can no longer prove a human — not the model — owned the outcome. Cleverness is the enemy of auditability.

So the architecture below is deliberately a pipeline, not an agent: a fixed conveyor belt where every application passes through the same stations in the same order, and each station does one inspectable job. Think of an assembly line in a regulated factory — pharmaceuticals, aircraft parts — where the point is not speed but that every unit takes the identical, logged, inspectable path so any defect can be traced to the exact station that caused it. That traceability is the feature. You are trading the flexibility of an agent for the certainty of a workflow, and under compliance that is almost always the right trade (it's the agents-vs-workflows call from CS1, made under the harshest possible constraints).

Two of the stations on that belt aren't optional stops — they're load-bearing walls, and the term is precise. Remove a decorative feature and the system is uglier; remove the PII-minimization step and personal data leaks to the model; remove the human approval gate and there is no accountable person; remove the immutable audit record and no past decision can ever be reconstructed. Each of those failures isn't a bug you patch — it's a whole missing requirement from §1. As you read the diagram left to right, notice the order isn't arbitrary: data is minimized before it can ever reach the model, and a recommendation becomes a decision only after a human signs and that signature is itself logged. The sequence is the safety property. A design that produces the same boxes in a different order — say, logging after the decision instead of at every stage — quietly fails the audit even though the picture looks similar.

3 · Grounding, explainability & the output contract

Try this thought experiment, because it's the trap almost everyone falls into first. You add one line to the prompt — "explain your reasoning" — and the model dutifully produces a paragraph: "I recommend declining because the applicant's profile suggests elevated risk based on several factors." It reads beautifully. It would satisfy a casual product manager. And it is worse than useless the moment a regulator reads it, because it explains nothing you can check: which factors? which policy? is any of it even true, or did the model invent a rule that doesn't exist? A fluent, unverifiable justification isn't explainability — it's a liability with good grammar. Feeling why that paragraph fails is the whole point of this section.

The mental model that fixes it is librarian, not oracle. An oracle pronounces answers from some inner wisdom you must take on faith. A librarian, asked a question, walks to the shelf, pulls the specific book, and points at the specific passage — and if there's no book on the topic, a good librarian says "I can't answer that" rather than making something up. Meridian's assistant is built to be the librarian: every recommendation must be grounded in the retrieved, versioned policy, and it must hand back not just an answer but the citations — which clauses, exactly, drove it. Crucially, "I found nothing in policy to support this" is not treated as a weak, low-confidence answer that squeaks through; it is a refusal that routes to a human. In a regulated system, "I don't know" is a correct and safe output; a confident guess is the dangerous one.

The word doing the heavy lifting here is contract. Explainability is not a request you make of the model and hope it honors — it is a shape the output must satisfy or it does not ship. Treat the model's response as untrusted input, the way a careful web server treats form data: you don't trust that the citations are real, so your code checks that every cited clause id actually exists in the set that was retrieved; you don't trust that no prohibited factor slipped in, so your code screens for protected attributes; you don't trust a borderline call, so a confidence below a floor forces a referral. The recurring principle — you'll see it again in the exercises — is the model proposes, code disposes. And every one of those checks fails closed: the default when anything is wrong is not "approve anyway," it's "send it to a human." The numbered contract below is that discipline written out; read each item as a gate the output must pass, not advice the model should follow.

The output contract every recommendation must satisfy

1. Decision field is a recommendation only (recommend_approve / recommend_decline / refer) — never a final approve/decline.
2. Rationale lists the factors that drove it, each tied to a cited policy clause id from the retrieved set.
3. Citations are verified in code to exist in the retrieved policy — a fabricated clause id fails closed to refer.
4. Confidence is reported and, below a floor, forces refer regardless of the recommendation.
5. No protected-attribute reasoning: the rationale is checked for prohibited factors; any hit blocks the recommendation and escalates.

4 · The human approval gate & the audit trail

Imagine the phone call every regulated-AI team dreads. It's a Tuesday, eleven months after launch. An auditor — or worse, a lawyer for a declined applicant — asks a single question: "Show me exactly why this person was declined last March." Not roughly. Not "the model was fairly accurate." Exactly: what inputs, which policy rules were in force that day, which version of the model and prompt produced the recommendation, and — the question that decides whether your company is liable — which named human approved it. If your system can answer that in five minutes, this section did its job. If it can only shrug, no amount of clever engineering elsewhere will save you. This is the section a regulator cares about most, and it rests on two components.

The first is the human approval gate, and the emphasis is on gate, not "approval button." A button is something a busy person clicks past; a gate is a wall with exactly one door, and the door does not open until an accountable human passes through it. The purpose isn't to slow things down for its own sake — it's to make certain there is always a specific person whose name is attached to the decision, someone the organization can point to and say "she owned this." The design consequence is strict: a recommendation is pending, not a decision, until a human acts, and the gate must be un-bypassable in code — not a UI convention a developer can route around under deadline pressure, but a structural fact of the system. If there's a code path that issues a decision without a logged human action, you don't have a gate; you have a suggestion.

The second is the audit trail, and the mental model that gets it right is a flight recorder — the black box on an aircraft. Nobody looks at it on a good day; its entire value is that when something goes wrong, it can reconstruct precisely what happened, and it is sealed so no one can quietly rewrite history after the fact. The gate answers who is accountable; the trail answers what actually happened. The subtle, expensive mistake teams make is treating the audit trail as "logs we can grep later." Logs are best-effort and mutable; an audit trail is a first-class component that is immutable, versioned, and tamper-evident (a per-record hash or seal so any later edit is detectable). And here is the counterintuitive twist the danger callout below drives home: because this record must survive for years, it is the worst place to store raw personal data — so it holds references and versions, not copies. The table walks each field; read it asking "if this one field were missing, what question could I no longer answer?" — because a trail is only as strong as its most-forgotten field.

5 · Evaluation — fairness & consistency, not just accuracy

Here's the sentence that feels like a win and is actually a trap: "The model is 96% accurate on our test set." In a consumer app, that's a ship-it number. In a regulated lending system, it's the number that gets you sued — because accuracy measures whether the recommendations are right on average, and nobody sues you over the average. They sue you over the specific applicant who was treated differently from a near-identical one, or over the cohort whose approval rate came out systematically lower. A 96%-accurate system can still be the one that quietly declines people for the wrong reasons 4% of the time — and if that 4% falls unevenly across groups, "accurate" is precisely no defense at all. Internalizing that accuracy is not the metric that protects you is the pivot of this section.

So the eval suite has to measure two things a normal one ignores. The first is consistency: does the system treat similar applications alike? The test for it is a perturbation test — take an application, change something that shouldn't matter to the outcome, and confirm the recommendation doesn't flip. Think of it like a scale that must read the same weight whether you place the object on the left or the right of the pan; if it doesn't, the scale is untrustworthy regardless of how often it's "close." The second is cohort screening: compare outcome rates across representative cohorts to catch disparate treatment — using synthetic, de-identified fixtures so you're never experimenting on real people. Both are hard gates, not averages you can offset: citation validity must be 100%, and a cohort gap beyond the defined tolerance blocks the ship, it doesn't get balanced out by good numbers elsewhere.

Two points guard against the mistakes engineers make here. First, this whole apparatus runs in CI, on every change — model, prompt, or policy — and a red result blocks the deploy and names the failing case. That's SOC 2-style change control made concrete: the release decision moves out of a person's judgment ("looks fine to me") and into an automated gate that can't be talked around at 5pm on a Friday. Second — and this is the humbling part for engineers — you do not get to define "fair" by engineering taste. Which cohorts to compare, which parity definition to use, what tolerance is acceptable: those are compliance and legal decisions grounded in the applicable regulation. Your job is not to invent the fairness metric; it's to take whatever compliance defines and make it measurable and gate-able in CI, and to re-confirm it as the rules evolve. Picking a fairness definition yourself because it was convenient to compute is a classic, well-intentioned way to fail an audit.

What the eval gate checks before any model/prompt/policy change ships

1. Recommendation quality against a labeled golden set of representative applications.
2. Citation validity = 100% — every cited policy clause exists in the retrieved set.
3. Consistency — near-identical applications get near-identical recommendations (perturbation tests).
4. Cohort screening — outcome rates compared across synthetic cohorts; a defined gap trips the gate for human review.
5. Prohibited-factor screen — no rationale references a protected or disallowed attribute.
6. The gate runs in CI on every change; a red result blocks the ship and names the failing case.

6 · Monitoring, drift & a staged rollout

The most dangerous moment for a regulated system isn't launch day — it's six months later, when nobody's looking. On launch day everyone is watching: the evals are green, the demo went well, the compliance sign-off is fresh. Then the team moves on to the next project, and the system keeps running against a world that quietly refuses to hold still. The applicant mix shifts as the economy changes; the bank updates a lending policy; a regulator revises a rule. None of these ring an alarm. The code is byte-for-byte identical to the version compliance approved — and yet it may now be making decisions against rules that no longer apply, on a population it was never validated for. A regulated system is never "done" at launch; the obligation to monitor is continuous, and this section is about not being blindsided by the slow, silent kind of failure.

Two things drift after ship, and it helps to name them separately because they need different watchmen. The data drifts — the shape of incoming applications moves away from what your evals were built on, so yesterday's validation slowly stops describing today's traffic. And the rules drift — policy and regulation change, so a decision that was correct in March is wrong in September even though the model never changed. The insidious part is that both invalidate yesterday's approvals silently: there's no crash, no error rate spike, just recommendations that are increasingly, quietly wrong. That's why you can't wait for something to break; you have to watch for the drift itself.

The response has two halves. The first is staged rollout, and the principle is: earn autonomy, never assume it — like a surgical resident who observes, then assists, then operates under supervision, and only much later works independently. The stages below move from shadow (the AI recommends but no one sees it) through assist (the officer must actively accept or override each rec) to a fast-path (quick human sign-off on low-risk, high-confidence cases, with sampled audits) — and notice the human owns every decision at every stage; the fast-path buys speed, never oversight. The second half is monitoring the right signals. The trap here is thinking your standard SRE dashboard — latency, error rate, cost — has you covered. It doesn't: a system can be fast, cheap, and error-free while drifting badly out of compliance. You need the compliance-specific signals, and the sharpest of them is the override rate — how often humans reject the AI's recommendation. A rising override rate is your earliest, cheapest alarm that the recs are drifting, long before an auditor would ever notice. Watch it the way you'd watch a fever: not the disease itself, but the first sign something's wrong.

7 · Failure modes & the control that hardens each

There's a moment in every design review that quietly tells you whether a system was really built for compliance or just decorated to look like it. Someone names a way it could go wrong — "what if the model cites a policy that doesn't exist?" — and you watch the response. If the answer is "we already handle that, it fails closed to a referral in §3," the system was designed for the threat. If the answer is "good point, we should add a check for that," you've just found where the next audit finding is hiding. The difference isn't intelligence or effort; it's whether the failure modes were enumerated first and each one assigned a home in the architecture, or whether they're being discovered one uncomfortable question at a time.

This is what "designed for compliance" actually means, and it's a stronger claim than it sounds. It doesn't mean the system never fails — every system fails. It means that for each way it can fail, there is already a named control in the architecture that catches it, and you can point to the section where that control lives. Think of it like a building's safety systems: the mark of good design isn't that fires never start, it's that every foreseeable fire — kitchen, electrical, chemical — has a specific, already-installed response: sprinklers here, extinguishers there, alarms wired throughout. A fire nobody planned for is the one that burns the building down. A failure mode with no pre-assigned control is your unplanned fire.

The table below is that enumeration for Meridian, and it's worth reading as a self-audit rather than a summary. Every failure mode in the left column is a real way regulated AI systems have caused harm — a black-box decline no one can explain, a rubber-stamped "approval" where the human never really looked, a cohort treated inconsistently, a fabricated policy clause. The right column points each one back to a control you've already met in §2 through §6 — which is the tell that the earlier sections weren't a grab-bag of best practices but a deliberate answer to a specific list of threats. Read each row and ask the reviewer's question: if this happened tomorrow, could I name the control that stops it, and point to where it lives? Any row where you can't is exactly the bolt-on you're about to regret.

Output = {
  recommendation: recommend_approve | recommend_decline | refer,
  rationale: [ {factor, cited_clause_id}, ... ],
  confidence: 0..1
}

verify(output):
  for r in rationale:
    if r.cited_clause_id not in retrieved_clause_ids:  return refer   # fabricated -> fail closed
    if r.factor in PROHIBITED_FACTORS:                 escalate()     # disparate-treatment guard
  if confidence < FLOOR:                               return refer
  return output