Regulated-industry AI system

Meet "Meridian Bank" — a representative composite. Meridian wants an AI assistant that helps loan officers decide consumer-credit applications faster: it reads the application and the supporting documents, retrieves the relevant policy, and drafts a recommendation with a rationale. The catch is the whole reason this is a case study: in a regulated setting the assistant is not allowed to be a black box. A declined applicant may be entitled to an explanation; a regulator may ask you to reconstruct why a decision was made; and a human — not the model — must own the outcome. Every design choice below is dominated by three forces a consumer chatbot never feels: you must be able to explain it, you must be able to audit it, and a human must stay accountable for it.

1 · Requirements & the constraints that veto architectures

Design starts with the non-functional constraints, because in a regulated industry they veto architectures before a feature is even discussed. Write them down first. Most never appear as a user-facing feature, yet each one can kill an otherwise-elegant design.

2 · Architecture — the decision path end to end

Every application flows through the same guarded pipeline. Read it left to right: nothing reaches the model until personal data is minimized, nothing becomes a decision until a human approves it, and nothing happens without an immutable audit record. The approval gate and the audit trail are load-bearing walls, not add-ons.

3 · Grounding, explainability & the output contract

The assistant is only useful if a human can trust and check it, so it is built as a librarian, not an oracle. Every recommendation must be grounded in the retrieved, versioned policy and must carry a structured rationale: the factors, the policy clauses cited, and an explicit confidence. An answer with no policy support is not a low-confidence answer — it is a refusal that routes to a human.

The output contract every recommendation must satisfy

1. Decision field is a recommendation only (recommend_approve / recommend_decline / refer) — never a final approve/decline.
2. Rationale lists the factors that drove it, each tied to a cited policy clause id from the retrieved set.
3. Citations are verified in code to exist in the retrieved policy — a fabricated clause id fails closed to refer.
4. Confidence is reported and, below a floor, forces refer regardless of the recommendation.
5. No protected-attribute reasoning: the rationale is checked for prohibited factors; any hit blocks the recommendation and escalates.

4 · The human approval gate & the audit trail

This is the section a regulator cares about most. Two components make the system defensible: an approval gate a human cannot bypass, and an audit trail that can reconstruct any decision long after the fact. The gate decides who is accountable; the trail proves what happened. The ordering below is the design: a recommendation is pending until a human acts, and the human's action is recorded as part of the immutable record.

5 · Evaluation — fairness & consistency, not just accuracy

You cannot ship what you cannot measure, and in a regulated setting accuracy is not the metric that gets you sued. The eval suite must also measure consistency (does the system treat similar applications alike?) and screen for disparate treatment across representative cohorts — using synthetic, de-identified fixtures, and with a compliance/legal partner defining what "fair" means here. The gate blocks the deploy if any of these regress. This is SOC 2-style change control made concrete.

What the eval gate checks before any model/prompt/policy change ships

1. Recommendation quality against a labeled golden set of representative applications.
2. Citation validity = 100% — every cited policy clause exists in the retrieved set.
3. Consistency — near-identical applications get near-identical recommendations (perturbation tests).
4. Cohort screening — outcome rates compared across synthetic cohorts; a defined gap trips the gate for human review.
5. Prohibited-factor screen — no rationale references a protected or disallowed attribute.
6. The gate runs in CI on every change; a red result blocks the ship and names the failing case.

6 · Monitoring, drift & a staged rollout

A regulated system is not "done" at launch — the obligation to monitor is continuous. Two things move after ship: the data (applicant mix drifts) and the rules (policy and regulation change). Both can silently invalidate yesterday's approval. Roll out in stages so the human stays firmly in the loop while trust is earned, and monitor the signals that would tell you the system has drifted out of compliance long before an auditor does.

7 · Failure modes & the control that hardens each

The mark of a system "designed for compliance" is that every failure mode maps to a control that already exists in the architecture. If a row below points at something you'd have to bolt on later, that is where your next audit finding is hiding.

Output = {
  recommendation: recommend_approve | recommend_decline | refer,
  rationale: [ {factor, cited_clause_id}, ... ],
  confidence: 0..1
}

verify(output):
  for r in rationale:
    if r.cited_clause_id not in retrieved_clause_ids:  return refer   # fabricated -> fail closed
    if r.factor in PROHIBITED_FACTORS:                 escalate()     # disparate-treatment guard
  if confidence < FLOOR:                               return refer
  return output