Governance, compliance & bias
Defenses keep a system safe; governance makes it accountable: EU AI Act risk tiers, model cards, audit trails, bias auditing, and the incident runbook a regulated org needs.
Learning objectives
- Explain the governance an org needs around a live AI system.
- Place a system in the EU AI Act risk tiers and know the obligations.
- Write a model card and keep an audit trail.
- Audit for bias and fairness, and run an AI incident.
code/rt5-governance/ in the course, with a README. Run the scripts or copy the configs directly.From safe to accountable advanced
Defenses (RT4) keep a system safe; governance makes it accountable — the documentation, compliance, and process a regulated org (like a financial-services company) needs. It extends O4 and W14 into legal and ethical obligations.
EU AI Act risk tiers advanced
The EU AI Act classifies AI systems by risk, and the tier sets your obligations. Even outside the EU it's becoming the reference framework.
| Tier | Examples | Obligation |
|---|---|---|
| Unacceptable | social scoring | banned |
| High-risk | credit scoring, hiring, medical | conformity assessment, docs, human oversight |
| Limited | chatbots | transparency (disclose it's AI) |
| Minimal | spam filters | few requirements |
Model cards & audit trails expert
A model card documents what a model is for, its data, its limits, and its evaluations — the artifact that makes a system reviewable. Pair it with an audit trail (every prompt/response logged, from W14) so decisions are traceable after the fact.
MODEL_CARD.md# MODEL_CARD.md
## Model: support-triage-classifier v3
- **Intended use:** classify inbound support tickets by severity. NOT for
automated account actions.
- **Out-of-scope:** legal/medical advice; any irreversible action without review.
- **Training data:** 12k internal tickets (2023-24), PII-scrubbed. Known gap:
under-represents enterprise-tier tickets.
- **Evaluation:** accuracy 0.91 overall; see fairness section for per-segment.
- **Fairness:** measured across ticket language + region; max gap 3.2% (below
our 5% threshold). Re-audited each release.
- **Risks & mitigations:** indirect injection via ticket body -> input guardrail
+ isolation (RT4). Human review for severity=critical.
- **Owner / contact:** platform-safety@company; last review 2026-09.
A model card is a short document that says what an AI system is for, what data it used, its limits, and how it's evaluated — the artifact that makes a system reviewable and accountable. This is a filled-in skeleton you can copy for any project. It's plain Markdown, not code; the ## and - just format headings and bullets.
- Intended use / Out-of-scope — states exactly what the model should do (triage tickets by severity) and, just as importantly, what it must not do (no automated account actions, no legal/medical advice). Drawing the boundary is half the safety.
- Training data — where the data came from, that PII was scrubbed, and a known gap (under-represents enterprise tickets). Honest limitations are a feature of a good card, not a weakness.
- Evaluation / Fairness — an overall accuracy number plus a per-segment fairness check with a stated threshold (max gap 3.2%, below the 5% limit), re-run every release. Aggregate scores can hide that a model fails one group.
- Risks & mitigations / Owner — names each real risk (indirect injection via the ticket body) and the exact defense from RT4 (input guardrail + isolation), plus a named owner and review date so someone is accountable.
Try this: Write one of these for a course project before you ship it. If any line is hard to fill in (What's out of scope? Who owns it? What's the fairness gap?), that's a gap in the project, not just the doc.
Bias & fairness auditing expert
Measure outcomes across segments (language, region, demographic proxies) — an aggregate metric can hide that a model works well for one group and poorly for another. Set a fairness threshold, measure every release, and record it in the model card.
The AI incident runbook expert
When something goes wrong
- Detect: monitoring/alerts (W14, O4) flag anomalous behavior or a breach.
- Contain: hit the kill switch — disable the feature/agent alias/tool (W14).
- Assess: what happened, blast radius, who/what was affected (audit trail).
- Remediate: fix the layer that failed (RT4); add a regression test (RT3).
- Review: blameless postmortem + update the model card and runbook.
Exercise RT5.1 — Govern a project
Context: Governing a real project ties the whole track together: you classify its tier, document it, and plan for the day it misbehaves — including naming the kill switch before you need it.
Your task: For a course project, classify its EU AI Act tier, write a model card, and draft a one-page incident runbook naming the kill switch — and if it's high-risk, list the extra obligations you'd owe.
Requirements:
- Classify the project into an EU AI Act tier with justification
- Write a model card covering purpose, data, limits, and evaluations
- Draft a one-page incident runbook that explicitly names the kill switch
- If the project is high-risk, list the extra obligations (bias audit, human oversight, documentation, audit trail)
💡 Hint: Naming the kill switch concretely — the exact toggle or action that stops the system — is what separates a real runbook from a wish.
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: Under the EU AI Act, a system's risk tier sets its obligations — and classifying early shapes the whole design. Credit scoring and hiring are high-risk; a chatbot is limited; a spam filter is minimal; social scoring is banned.
Your task: Write risk_tier(use_case) that maps a use case to unacceptable | high | limited | minimal using the lesson's examples.
Requirements:
- Map social scoring → unacceptable, credit scoring / hiring / medical → high, chatbot → limited, spam filter → minimal
- Default unknown use cases cautiously (e.g. to limited)
- Return both the tier and its obligations
- Obligations reflect the tier: high → conformity assessment, docs, human oversight; limited → transparency (disclose it's AI); minimal → few requirements; unacceptable → banned
- Note that high-risk (credit/hiring) makes documentation, testing, oversight, and audit requirements, not extras — directly relevant to FICO-style use cases
💡 Hint: Two lookup tables — use case to tier, tier to obligations — keep the policy explicit and easy to audit.
Show solution
Classifying early shapes the whole design:
TIERS = {
"social_scoring": "unacceptable",
"credit_scoring": "high", "hiring": "high", "medical": "high",
"chatbot": "limited",
"spam_filter": "minimal",
}
OBLIGATION = {
"unacceptable": "banned",
"high": "conformity assessment, docs, human oversight",
"limited": "transparency (disclose it's AI)",
"minimal": "few requirements",
}
def risk_tier(use_case):
tier = TIERS.get(use_case, "limited") # default cautiously
return tier, OBLIGATION[tier]
print(risk_tier("credit_scoring")) # ('high', 'conformity assessment, docs, human oversight')
print(risk_tier("chatbot")) # ('limited', "transparency (disclose it's AI)")A credit or hiring model being high-risk means documentation, testing, human oversight and audit are requirements, not extras — directly relevant to FICO-style use cases.
Context: A model card documents purpose, data, limits, and evals so a system is reviewable. Encoding it as validated data means an incomplete card can fail CI, just like a missing test.
Your task: Define a model_card dict with the required fields and a validator that fails if any required section is missing or empty.
Requirements:
- Include required fields: name, intended use, out-of-scope, training data, limitations, evaluations, and owner
- Populate a realistic example card with concrete values
validate(card)collects any required key that is missing or empty- Raise an error naming the missing sections, or return valid when complete
- Make the point that the card as validated data lets an incomplete card fail CI
💡 Hint: The validator is a list comprehension over the required keys checking for falsy values; the discipline is in treating documentation like a test.
Show solution
The card is the artifact that makes a system auditable:
REQUIRED = ["name", "intended_use", "out_of_scope", "training_data",
"limitations", "evaluations", "owner"]
card = {
"name": "refund-classifier-v3",
"intended_use": "Triage refund requests for human agents.",
"out_of_scope": "Auto-approving refunds without review.",
"training_data": "12k anonymized 2024 tickets; English only.",
"limitations": "Lower recall on non-English; no PII redaction upstream.",
"evaluations": {"accuracy": 0.91, "false_approve_rate": 0.02},
"owner": "trust-and-safety@acme",
}
def validate(card):
missing = [k for k in REQUIRED if not card.get(k)]
if missing:
raise ValueError(f"model card incomplete: {missing}")
return "valid"
print(validate(card)) # validEncoding the card as validated data means an incomplete card can fail CI, just like a missing test.
Context: Audit trails make a system reviewable after the fact, and hash-chaining gives cheap tamper-evidence without a database: each entry chains a hash of the previous one, so any edit breaks the chain from that point forward.
Your task: Build an append-only log where each entry chains a hash of the previous one, and show verification passing, then failing after a forged edit.
Requirements:
- Each entry stores its event plus a hash computed from the previous hash and the event contents
- The first entry chains from a fixed genesis value
- Hash deterministically (e.g. sha256 over the previous hash plus a canonical JSON of the event)
verify()recomputes the chain and returns True when intact- Demonstrate verification returning True, then False after forging an earlier entry
- Note that any edit breaks the chain from that point on, proving the recorded decisions are the ones actually made
💡 Hint: Serialize each event with sorted keys before hashing so the hash is stable, and verify by walking the chain forward from genesis recomputing each link.
Show solution
Hash-chaining gives cheap tamper-evidence without a database:
import hashlib, json
def _h(prev, entry):
return hashlib.sha256((prev + json.dumps(entry, sort_keys=True)).encode()).hexdigest()
class AuditLog:
def __init__(self):
self.entries = []
def add(self, event):
prev = self.entries[-1]["hash"] if self.entries else "GENESIS"
self.entries.append({"event": event, "hash": _h(prev, event)})
def verify(self):
prev = "GENESIS"
for e in self.entries:
if e["hash"] != _h(prev, e["event"]):
return False
prev = e["hash"]
return True
log = AuditLog()
log.add({"action": "decision", "id": 1, "outcome": "approve"})
log.add({"action": "decision", "id": 2, "outcome": "deny"})
print(log.verify()) # True
log.entries[0]["event"]["outcome"] = "deny" # forge
print(log.verify()) # FalseAny edit breaks the chain from that point forward, so the log can prove the recorded decisions are the ones actually made.
Context: Fairness auditing uses standard, defensible metrics on your own eval data. The four-fifths rule — any group's selection rate below 80% of the most-favored group's — is the classic disparate-impact flag.
Your task: Given labeled outcomes per group, compute the selection rate per group and apply the four-fifths (80%) rule disparate-impact check, flagging groups below the threshold.
Requirements:
selection_rates(rows)computes approved/total per group from (group, approved) rows- Take the most-favored group's rate as the reference
- Compute each group's ratio against that reference
- Flag any group whose ratio is below 0.8 as DISPARATE IMPACT, others as OK
- Demonstrate a group falling below the threshold being flagged
- Note that a sub-0.8 ratio is a signal to investigate, mitigate, and document in the model card
💡 Hint: Two passes: aggregate rates per group, then divide every rate by the maximum rate and compare against 0.8.
Show solution
Standard, defensible fairness metrics on your own eval data:
def selection_rates(rows):
# rows: list of (group, approved:bool)
agg = {}
for g, ok in rows:
n, k = agg.get(g, (0, 0))
agg[g] = (n + 1, k + int(ok))
return {g: k / n for g, (n, k) in agg.items()}
def four_fifths(rates):
ref = max(rates.values()) # most-favored group
flags = {g: r / ref for g, r in rates.items()}
return {g: (ratio, "OK" if ratio >= 0.8 else "DISPARATE IMPACT")
for g, ratio in flags.items()}
rows = [("A", True), ("A", True), ("A", False),
("B", True), ("B", False), ("B", False)]
rates = selection_rates(rows)
print(rates) # {'A': 0.666..., 'B': 0.333...}
for g, (ratio, verdict) in four_fifths(rates).items():
print(g, f"{ratio:.2f}", verdict) # B 0.50 DISPARATE IMPACTA ratio below 0.8 versus the most-favored group is the classic disparate-impact flag — a signal to investigate, mitigate, and document in the model card.
Context: When a live system misbehaves you need a runbook. Encoding the incident phases as an ordered checklist with a closure gate stops the common failure mode of "we patched it" without eradication or a written record.
Your task: Encode the incident phases as an ordered checklist with a gate that won't let you close an incident until the detect / contain / eradicate / document steps are all done.
Requirements:
- Define the ordered list of phases (detect, triage, contain, eradicate, recover, document, postmortem)
- Track completed phases for an incident
complete(phase)validates the phase name before recording itcan_close()returns whether the required phases are all done and which are missing- Require at least detect, contain, eradicate, and document before closure is allowed
- Demonstrate a close being refused with missing phases, then allowed once they're complete
💡 Hint: A set of completed phases and a required subset is the whole model; can_close is just a set difference against the required phases.
Show solution
A machine-checkable runbook keeps incident response disciplined:
PHASES = ["detect", "triage_severity", "contain", "eradicate",
"recover", "document", "postmortem"]
class Incident:
def __init__(self, title):
self.title = title
self.done = set()
def complete(self, phase):
assert phase in PHASES, f"unknown phase {phase}"
self.done.add(phase)
def can_close(self):
required = {"detect", "contain", "eradicate", "document"}
missing = required - self.done
return (not missing), sorted(missing)
inc = Incident("prompt-injection leak in support bot")
for p in ["detect", "triage_severity", "contain"]:
inc.complete(p)
print(inc.can_close()) # (False, ['document', 'eradicate'])
inc.complete("eradicate"); inc.complete("document")
print(inc.can_close()) # (True, [])Gating closure on the required phases stops the common failure mode of "we patched it" without eradication or a written record.
Context: One governance gate encodes the org's release policy for high-risk AI: a regulated system can't ship until its tier is classified, the model card validates, a bias audit passed, and an audit log is enabled — enforced by the pipeline, not left to memory.
Your task: Write governance_gate(system) returning pass/fail with the list of unmet obligations, so a regulated system can't ship until every obligation is satisfied.
Requirements:
- Fail if the tier isn't classified (or is unacceptable) or the model card isn't valid
- For high-risk systems, additionally require a passed bias audit and configured human oversight
- Require an enabled audit trail regardless of tier
- Return a (passed, unmet) pair listing every unmet obligation
- Demonstrate a ready system passing and a not-ready one failing with its specific gaps
- Make the point that high-risk systems carry extra blocking obligations so accountability is enforced by the pipeline
💡 Hint: Accumulate unmet obligations into a list and gate on it being empty; the high-risk branch simply adds more required checks.
Show solution
One gate that encodes the org's release policy for high-risk AI:
def governance_gate(system):
unmet = []
if system.get("tier") not in {"minimal", "limited", "high"}:
unmet.append("tier not classified (or unacceptable)")
if not system.get("model_card_valid"):
unmet.append("model card incomplete")
if system.get("tier") == "high":
if not system.get("bias_audit_passed"):
unmet.append("bias audit not passed (required for high-risk)")
if not system.get("human_oversight"):
unmet.append("human oversight not configured (required for high-risk)")
if not system.get("audit_log_enabled"):
unmet.append("audit trail not enabled")
return (not unmet), unmet
ready = {"tier": "high", "model_card_valid": True, "bias_audit_passed": True,
"human_oversight": True, "audit_log_enabled": True}
notready = {"tier": "high", "model_card_valid": True, "bias_audit_passed": False,
"human_oversight": False, "audit_log_enabled": True}
print(governance_gate(ready)) # (True, [])
print(governance_gate(notready)) # (False, ['bias audit...', 'human oversight...'])High-risk systems carry extra obligations (bias audit, human oversight); the gate makes them blocking, so accountability is enforced by the pipeline, not left to memory.
✓ Checkpoint — you can move on when you can…
- Explain the governance a live AI system needs.
- Place a system in the EU AI Act tiers and state its obligations.
- Write a model card and keep an audit trail.
- Audit for bias and run an AI incident with a kill switch.
Knowledge check check yourself
What are the EU AI Act risk tiers described in the lesson, and what obligation does the high-risk tier carry?
Show answer
What is a model card, and what are the five steps of the AI incident runbook the lesson prescribes?