Governance, compliance & bias

From safe to accountable advanced

Picture the moment every AI team dreads. Your support agent has been live for six months. Then a customer forwards a screenshot: the bot approved a refund it should have escalated, quoting a policy that doesn't exist. Legal calls. Your VP asks three questions in a row — Who signed off on this going live? Can you show me exactly what the model was told and what it decided? What happens now, this hour, to stop it? — and you realize you can't answer any of them. The model was safe in the sense that it had guardrails. It was not accountable: no one owned it, nothing was written down, and there was no plan for the bad day. That gap between "we added defenses" and "we can stand behind this in front of a regulator" is exactly what this chapter closes.

Here is the distinction in one line: defenses reduce the chance of harm; governance is how the organization takes responsibility for it. Think of it like a commercial airliner. Anti-lock brakes and redundant hydraulics are the defenses — engineering that makes the plane safer to fly. But you'd never board a plane that had only those. What lets it carry passengers is the governance around it: a certified type rating, a maintenance logbook every mechanic signs, a checklist the crew runs before every takeoff, and an accident-investigation process for when something does go wrong. Governance is the paperwork, the ownership, and the process that turn "probably safe" into "answerable to someone."

Concretely, governance for a live AI system is four things, and every later section of this chapter is one of them: a risk classification (how much scrutiny does this system legally require?), documentation (what is it for, and what did we test?), an audit trail (what actually happened, provably?), and an incident process (what do we do the day it breaks?). The common mistake engineers make is treating all of this as bureaucracy that slows shipping — a tax paid to a compliance team after the real work is done. It's the opposite: governance decided early shapes the architecture. If you know on day one that a hiring tool is legally high-risk, you build in human review, logging, and fairness testing from the first commit, rather than bolting them on in a panic after launch. This is why it belongs in a red-team track and not just a legal binder: the engineer who understands governance builds systems that are defensible by design.

Defenses (RT4) keep a system safe; governance makes it accountable — the documentation, compliance, and process a regulated org (like a financial-services company) needs. It extends O4 and W14 into legal and ethical obligations.

EU AI Act risk tiers advanced

Two teams build a chatbot. One ships in a week; the other spends three months in review. Why? Because one bot answers product questions and the other decides who gets a mortgage. Same technology, wildly different consequences if it's wrong — and the law noticed. Regulators realized long ago that you cannot write one rule for "AI," any more than you'd write one safety rule for both a toaster and a nuclear reactor. So instead of regulating the technology, they regulate the stakes: the more a system can hurt someone, the more you have to prove before you turn it on.

That is the whole idea behind a risk-tiered framework, and the EU AI Act is the one that has become the global reference point (much as GDPR became the default for privacy — companies far outside Europe adopt it because it's the strictest bar, and clearing it clears the others). It sorts every AI system into four buckets by potential for harm. Think of it as a customs form: before your system crosses into production, you declare what it does, and that declaration determines how much inspection it gets. A spam filter waves through; a credit-scoring model gets opened and searched.

• Unacceptable — uses judged to violate fundamental rights, like government social-scoring of citizens. These aren't regulated, they're banned. The answer is no.
• High-risk — systems that make consequential decisions about people's lives: credit, hiring, medical, education, law enforcement. Allowed, but only with heavy obligations (documentation, testing, human oversight, audit).
• Limited — systems where the main risk is deception, like a chatbot a user might mistake for a human. The obligation is transparency: disclose that it's AI.
• Minimal — everything else, like a spam filter or a game AI. Few or no requirements.

The reason this matters to you as a builder and not just a lawyer: your tier is a design constraint, not a paperwork step at the end. The single most expensive mistake here is discovering late that your "simple internal tool" is actually high-risk — because it screens job applicants, say — after you've already built it with no logging, no human-in-the-loop, and no fairness data. Now you're re-architecting under deadline pressure. Classify honestly on day one and the obligations become a checklist you build toward, not a wall you hit. And classify to the use, not the tech: the same LLM is minimal-risk writing marketing copy and high-risk ranking loan applications.

The EU AI Act classifies AI systems by risk, and the tier sets your obligations. Even outside the EU it's becoming the reference framework.

Model cards & audit trails expert

A new engineer joins your team and inherits a model that's been in production for a year. They ask the reasonable questions: What is this thing actually supposed to do? What data was it trained on? What does it explicitly not handle? How good is it, and for whom? If the only answers live in the head of someone who left the company, you don't have a model — you have a liability with an API. This is the everyday failure a model card prevents: undocumented systems that no one can safely change, audit, or defend.

A model card is the equivalent of the nutrition label on a food package, or the datasheet that ships with an electronic component. A resistor's datasheet tells you its resistance, its tolerance, and the voltage beyond which it burns out — the honest operating envelope. A model card does the same for an AI system: its intended use, its out-of-scope uses, the data it learned from (and the gaps in that data), how it scored in evaluation, how it performed per segment for fairness, the known risks, and — crucially — a named human owner. The format is deliberately short and boring; the value isn't in the prose, it's in forcing someone to write down the answers before shipping, where the empty fields expose the real gaps.

But a card describes intentions, and intentions aren't proof. That's the second half of this pairing: the audit trail. Where the model card says "here's what this system is supposed to do," the audit trail is the flight recorder that logs what it actually did — every prompt in, every response out, every decision, timestamped and retained. Why you need both is the key insight: the card lets you reason about the system in advance; the trail lets you reconstruct a specific incident afterward. When that refund-approval goes wrong from the opening story, the card tells the reviewer "this was never meant to auto-approve" and the trail proves exactly which input triggered which output at 3:47pm. One without the other leaves you half-blind. The common mistake is logging everything into an ordinary mutable database and calling it an audit trail — but if anyone (including an attacker or a panicked engineer) can quietly edit a past entry, it proves nothing. That's why real audit trails are append-only and often tamper-evident, a property you'll build with hash-chaining in the practice ladder below.

A model card documents what a model is for, its data, its limits, and its evaluations — the artifact that makes a system reviewable. Pair it with an audit trail (every prompt/response logged, from W14) so decisions are traceable after the fact.

MODEL_CARD.md# MODEL_CARD.md
## Model: support-triage-classifier v3
- **Intended use:** classify inbound support tickets by severity. NOT for
  automated account actions.
- **Out-of-scope:** legal/medical advice; any irreversible action without review.
- **Training data:** 12k internal tickets (2023-24), PII-scrubbed. Known gap:
  under-represents enterprise-tier tickets.
- **Evaluation:** accuracy 0.91 overall; see fairness section for per-segment.
- **Fairness:** measured across ticket language + region; max gap 3.2% (below
  our 5% threshold). Re-audited each release.
- **Risks & mitigations:** indirect injection via ticket body -> input guardrail
  + isolation (RT4). Human review for severity=critical.
- **Owner / contact:** platform-safety@company; last review 2026-09.

Bias & fairness auditing expert

Here is the trap that catches good teams. Your model reports 91% accuracy, the dashboard is green, everyone celebrates, and you ship. Six months later a journalist shows that it approves 70% of applications from one group and 35% from another. Both facts are true at once: the aggregate looked fine because the large, well-served group drowned out the small, badly-served one in the average. A single headline number is a blender — it purées every group into one smooth score and hides exactly the disparity that gets you sued. Bias auditing is the discipline of refusing to be fooled by that average.

The core move is simple and you already have the instinct for it: don't ask "how well does it work?" — ask "how well does it work for each group?" Break your evaluation data into segments (by language, region, or protected-attribute proxies) and compute the metric separately for each. It's the same reason a doctor checks blood pressure in both arms rather than averaging them: the average can be normal while one side is dangerous. A model that's excellent overall but consistently worse for one demographic isn't "mostly fine" — it's discriminatory, and increasingly, illegally so.

To turn that instinct into a defensible number, fairness work leans on standard, court-tested metrics rather than gut feeling. The classic one is the four-fifths rule (also called the 80% rule), borrowed from decades of US employment-discrimination law: take the group the model favors most as the benchmark, and if any other group's selection rate falls below 80% of that benchmark, you have a disparate impact flag worth investigating. It's not a verdict — a sub-0.8 ratio can have a legitimate explanation — but it's a defensible tripwire that says "stop and look here," which you'll implement in the practice ladder. The essential discipline, and the mistake teams make, is treating this as a launch-day formality run once. Data drifts, populations shift, and a model that was fair at launch can drift unfair. So you set a threshold, measure it every release, and record the result in the model card — turning fairness from a one-time hope into a monitored, documented property.

Measure outcomes across segments (language, region, demographic proxies) — an aggregate metric can hide that a model works well for one group and poorly for another. Set a fairness threshold, measure every release, and record it in the model card.

The AI incident runbook expert

It's 2am and your agent is doing something it shouldn't. Maybe a prompt injection has it leaking other customers' data; maybe a model update quietly regressed and it's giving dangerous advice. The pager goes off. In that moment, the worst possible thing is a room full of smart people improvising — arguing about whether to pull the whole service down, hunting for who has the access to do it, no one writing anything down. Panic doesn't scale, and improvisation under pressure is how a contained problem becomes a headline. The fix is the same one hospitals, fire departments, and airlines figured out long ago: when the stakes are high and adrenaline is running, you don't rely on judgment — you run a checklist.

An incident runbook is that checklist for the day your AI system misbehaves. It's written calmly, in advance, precisely so that the frightened 2am version of you doesn't have to think — just execute. The value isn't the document; it's that the hard decisions (What's our threshold for pulling the plug? Who's allowed to? What do we owe customers?) were made when everyone was rested and rational, not mid-crisis. The steps below map onto the same phases every mature incident process uses, adapted for AI:

• Detect — you can only respond to what you can see, so this leans entirely on the monitoring and alerting from W14/O4. No observability, no incident response.
• Contain — stop the bleeding before you understand it fully. This is the kill switch: a pre-built, pre-tested toggle that disables the feature, agent alias, or tool in one action. The cardinal rule — you must build and rehearse the kill switch before the incident, because you cannot invent one at 2am.
• Assess — now that it's contained, use the audit trail to work out what actually happened and the blast radius: who was affected, how many, how badly.
• Remediate — fix the layer that failed (a defense from RT4) and add a regression test (RT3) so the same hole can't reopen silently.
• Review — a blameless postmortem (the goal is a better system, not a scapegoat) that feeds its lessons back into the model card and the runbook itself.

Two ideas make this more than a to-do list. First, containment comes before understanding — the instinct to "just figure out what's wrong first" is exactly backwards; you hit the kill switch while the damage clock is still running, then investigate the stopped system safely. Second, the loop closes: every incident makes the runbook and the model card better, so your governance compounds instead of decaying. The mistake that undoes all of this is declaring victory too early — "we patched it, incident over" — without eradicating the root cause or writing down what happened. That's why the exercises below gate an incident from closing until Detect, Contain, Eradicate, and Document are all genuinely done.

When something goes wrong

1. Detect: monitoring/alerts (W14, O4) flag anomalous behavior or a breach.
2. Contain: hit the kill switch — disable the feature/agent alias/tool (W14).
3. Assess: what happened, blast radius, who/what was affected (audit trail).
4. Remediate: fix the layer that failed (RT4); add a regression test (RT3).
5. Review: blameless postmortem + update the model card and runbook.

TIERS = {
    "social_scoring": "unacceptable",
    "credit_scoring": "high", "hiring": "high", "medical": "high",
    "chatbot": "limited",
    "spam_filter": "minimal",
}
OBLIGATION = {
    "unacceptable": "banned",
    "high": "conformity assessment, docs, human oversight",
    "limited": "transparency (disclose it's AI)",
    "minimal": "few requirements",
}

def risk_tier(use_case):
    tier = TIERS.get(use_case, "limited")   # default cautiously
    return tier, OBLIGATION[tier]

print(risk_tier("credit_scoring"))  # ('high', 'conformity assessment, docs, human oversight')
print(risk_tier("chatbot"))         # ('limited', "transparency (disclose it's AI)")

REQUIRED = ["name", "intended_use", "out_of_scope", "training_data",
            "limitations", "evaluations", "owner"]

card = {
    "name": "refund-classifier-v3",
    "intended_use": "Triage refund requests for human agents.",
    "out_of_scope": "Auto-approving refunds without review.",
    "training_data": "12k anonymized 2024 tickets; English only.",
    "limitations": "Lower recall on non-English; no PII redaction upstream.",
    "evaluations": {"accuracy": 0.91, "false_approve_rate": 0.02},
    "owner": "trust-and-safety@acme",
}

def validate(card):
    missing = [k for k in REQUIRED if not card.get(k)]
    if missing:
        raise ValueError(f"model card incomplete: {missing}")
    return "valid"

print(validate(card))   # valid

import hashlib, json

def _h(prev, entry):
    return hashlib.sha256((prev + json.dumps(entry, sort_keys=True)).encode()).hexdigest()

class AuditLog:
    def __init__(self):
        self.entries = []
    def add(self, event):
        prev = self.entries[-1]["hash"] if self.entries else "GENESIS"
        self.entries.append({"event": event, "hash": _h(prev, event)})
    def verify(self):
        prev = "GENESIS"
        for e in self.entries:
            if e["hash"] != _h(prev, e["event"]):
                return False
            prev = e["hash"]
        return True

log = AuditLog()
log.add({"action": "decision", "id": 1, "outcome": "approve"})
log.add({"action": "decision", "id": 2, "outcome": "deny"})
print(log.verify())                       # True
log.entries[0]["event"]["outcome"] = "deny"   # forge
print(log.verify())                       # False

def selection_rates(rows):
    # rows: list of (group, approved:bool)
    agg = {}
    for g, ok in rows:
        n, k = agg.get(g, (0, 0))
        agg[g] = (n + 1, k + int(ok))
    return {g: k / n for g, (n, k) in agg.items()}

def four_fifths(rates):
    ref = max(rates.values())        # most-favored group
    flags = {g: r / ref for g, r in rates.items()}
    return {g: (ratio, "OK" if ratio >= 0.8 else "DISPARATE IMPACT")
            for g, ratio in flags.items()}

rows = [("A", True), ("A", True), ("A", False),
        ("B", True), ("B", False), ("B", False)]
rates = selection_rates(rows)
print(rates)                 # {'A': 0.666..., 'B': 0.333...}
for g, (ratio, verdict) in four_fifths(rates).items():
    print(g, f"{ratio:.2f}", verdict)   # B 0.50 DISPARATE IMPACT

PHASES = ["detect", "triage_severity", "contain", "eradicate",
          "recover", "document", "postmortem"]

class Incident:
    def __init__(self, title):
        self.title = title
        self.done = set()
    def complete(self, phase):
        assert phase in PHASES, f"unknown phase {phase}"
        self.done.add(phase)
    def can_close(self):
        required = {"detect", "contain", "eradicate", "document"}
        missing = required - self.done
        return (not missing), sorted(missing)

inc = Incident("prompt-injection leak in support bot")
for p in ["detect", "triage_severity", "contain"]:
    inc.complete(p)
print(inc.can_close())     # (False, ['document', 'eradicate'])
inc.complete("eradicate"); inc.complete("document")
print(inc.can_close())     # (True, [])

def governance_gate(system):
    unmet = []
    if system.get("tier") not in {"minimal", "limited", "high"}:
        unmet.append("tier not classified (or unacceptable)")
    if not system.get("model_card_valid"):
        unmet.append("model card incomplete")
    if system.get("tier") == "high":
        if not system.get("bias_audit_passed"):
            unmet.append("bias audit not passed (required for high-risk)")
        if not system.get("human_oversight"):
            unmet.append("human oversight not configured (required for high-risk)")
    if not system.get("audit_log_enabled"):
        unmet.append("audit trail not enabled")
    return (not unmet), unmet

ready = {"tier": "high", "model_card_valid": True, "bias_audit_passed": True,
         "human_oversight": True, "audit_log_enabled": True}
notready = {"tier": "high", "model_card_valid": True, "bias_audit_passed": False,
            "human_oversight": False, "audit_log_enabled": True}
print(governance_gate(ready))     # (True, [])
print(governance_gate(notready))  # (False, ['bias audit...', 'human oversight...'])

Knowledge check check yourself