AI governance & compliance
You already ship AI. This lesson is the layer above the code: the governance program that lets an organization deploy AI systems it can defend to a regulator, an auditor, and its own board. We map genuinely public frameworks — the EU AI Act risk tiers, ISO/IEC 42001, and the NIST AI Risk Management Framework — onto engineering artifacts you can actually produce: model cards, audit trails, and a RACI that says who owns what. Framework-as-fact, attributed; company examples are representative scenarios. This is engineering-leadership framing, not legal advice.
Learning objectives
- Explain what an AI governance program is and why it is an engineering concern, not just a legal one.
- Place a system into the EU AI Act risk tiers (prohibited / high-risk / limited / minimal) and name the duties each tier implies — verifying specifics against the current text.
- Map controls to ISO/IEC 42001 (an AI management-system standard) and the NIST AI RMF functions (Govern, Map, Measure, Manage).
- Produce the two artifacts auditors actually ask for: a model card and an audit trail.
- Assign accountability with a RACI so 'who signs off on this model' has one answer.
1 · What a governance program actually is
A governance program is the standing machinery that decides which AI systems may exist, under what controls, and who is accountable when one misbehaves. For an engineer it is not paperwork for its own sake — it is the difference between "we think this is fine" and "we can show a regulator exactly why this is fine." The three moving parts are policy (what we will and won't build), process (the gates a system passes before and after launch), and evidence (the artifacts that prove the process ran).
The failure mode governance prevents is the orphaned model: a system in production that no one owns, whose training data no one can describe, and whose behavior no one is monitoring. Every framework below is, at heart, a structured way to make orphaned models impossible.
2 · EU AI Act — risk tiers as a design input
The EU AI Act is public, adopted law that classifies AI systems by risk tier and attaches escalating obligations. As an engineer, treat the tier as a design input the way you treat a latency budget: it changes what you must build. The four commonly-described tiers:
| Tier | Rough meaning | Illustrative obligation (verify current text) |
|---|---|---|
| Prohibited | Unacceptable-risk uses (e.g. certain social scoring, some biometric practices). | Not allowed to place on the market — a hard no, not a control. |
| High-risk | Systems in sensitive domains (e.g. specified employment, credit, critical-infrastructure uses). | Risk management, data governance, logging, human oversight, technical documentation, conformity assessment. |
| Limited-risk | Systems that interact with people or generate content. | Transparency: tell users they're dealing with AI / that content is AI-generated. |
| Minimal-risk | Everything else (e.g. spam filters, most game AI). | No specific obligations beyond existing law. |
Note the asymmetry engineers miss: an EU-facing product may be high-risk because of where it is used, not because of how clever the model is. A boring logistic-regression that decides loan eligibility can carry heavier obligations than a frontier LLM writing marketing copy.
3 · ISO/IEC 42001 & NIST AI RMF — the control catalogs
Two public frameworks tell you how to run governance rather than what is illegal. ISO/IEC 42001 is an international standard for an AI management system (AIMS) — the same management-system pattern as ISO 27001 for security, applied to AI: define policy, assign roles, run a plan-do-check-act cycle, and keep records. It is certifiable, which matters for enterprise sales. The NIST AI RMF is a voluntary US framework organized around four functions you can use as a checklist:
| NIST AI RMF function | The question it forces | Engineering artifact |
|---|---|---|
| Govern | Who is accountable and what is our risk appetite? | Policy doc, RACI, model inventory |
| Map | What is this system, its context, and its risks? | Model card, intended-use statement, risk register entry |
| Measure | How do we quantify performance, bias, and robustness? | Eval suite, bias metrics, red-team results |
| Manage | How do we act on risk over the lifecycle? | Monitoring, incident process, retirement plan |
4 · The artifacts auditors ask for: model cards & audit trails
Frameworks become real as two artifacts. A model card is a short, standardized document describing a model's intended use, training-data provenance, evaluation results, known limitations, and owner. An audit trail is the immutable record of what happened: which model version served which request, what it was asked, what it returned (privacy-preserving), and who approved its deployment. Together they answer the auditor's two questions — "what is this system?" and "can you prove it behaved as claimed?"
5 · Who owns the model? — the RACI
The single most common governance gap is diffuse ownership. A RACI matrix (Responsible, Accountable, Consulted, Informed) fixes this by naming, for each decision, exactly one Accountable role. The pipeline below shows the flow a system passes through and who owns each gate.
| Decision | Responsible | Accountable | Consulted | Informed |
|---|---|---|---|---|
| Risk-tier classification | ML engineer | Product owner | Legal / compliance | Security |
| Model-card sign-off | ML engineer | Eng lead | Data governance | Product |
| Deploy approval | Eng lead | Governance board / risk owner | Legal, Security | Execs |
| Incident response | On-call eng | Eng lead | Legal, Comms | Governance board |
✓ Checkpoint — you can move on when you can…
- Explain the difference between policy, process, and evidence in a governance program.
- Given a system description, place it in an EU AI Act tier and name one obligation that tier implies (noting you'd verify the current text).
- Map a single control to both a NIST AI RMF function and an ISO/IEC 42001 concern.
- Say what a model card and an audit trail each prove to an auditor.
- Fill a RACI row with exactly one Accountable owner and justify it.
A team wants to ship an internal tool that ranks job applicants' résumés for recruiters. An engineer says "it's just a classifier, low risk." Why is that reasoning dangerous under an EU AI Act lens, and what changes about how you'd build it?
Show answer
Your company is already SOC 2 compliant. Leadership asks, "doesn't that cover our AI governance?" Give a precise answer.
Show answer
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: Risk-tiering is the first governance reflex: before you design controls you decide how much control is warranted.
Your task: Given three representative systems, assign each a plausible EU AI Act tier and name one obligation implied — flagging that you would verify against the current text.
Requirements:
- System A: a spam filter for internal email
- System B: an LLM chatbot on a public website that answers product questions
- System C: a model that scores loan applications for approval
- For each, state the tier and one duty, and note the verify-against-current-text caveat
💡 Hint: The tier is about the use and its stakes, not the model's sophistication.
Show solution
System A — spam filter → minimal-risk. No AI-Act-specific obligations beyond existing law; ordinary security/privacy hygiene applies.
System B — public product chatbot → limited-risk. The salient duty is transparency: users should be told they are interacting with an AI system. Verify the exact transparency wording against the current Act.
System C — loan-approval scoring → high-risk. Credit/eligibility decisions are a commonly-cited high-risk use, pulling in duties such as risk management, data governance, record-keeping/logging, human oversight, and technical documentation, plus a conformity assessment. Confirm the current annex and obligations with legal.
The lesson: the simplest model (C could be logistic regression) carries the heaviest obligations, because tiering follows stakes, not complexity.
Context: A model card is the single artifact every framework asks for and most teams don't have.
Your task: Write a model-card template with the sections an auditor expects, then fill it for one representative system.
Requirements:
- Include: intended use, out-of-scope uses, training/reference data provenance, evaluation results, known limitations, and a named owner
- Mark which fields should be auto-filled by CI vs hand-written
- Keep it to one screen — a card no one maintains is worthless
💡 Hint: Separate the machine-generated facts (version, scores) from the human judgment (intended use).
Show solution
Model card template (representative fill for a support-triage classifier):
- Model: support-triage-classifier v2.3.1 (auto: version, git SHA)
- Owner: Support Platform team, accountable: Eng Lead (name/role) (hand-written)
- Intended use: route inbound support tickets to one of 6 queues to assist human agents. (hand-written)
- Out-of-scope: auto-closing tickets, any customer-facing action, non-English at launch. (hand-written — the most important section)
- Training data: 120k historical tickets, 2022–2024, PII-stripped; provenance hash
sha256:…(auto) - Evaluation: macro-F1 0.88 on held-out set; per-queue precision table; fairness check across language groups (auto from eval CI)
- Limitations: degrades on tickets >2k tokens; unknown behavior on new product lines. (hand-written)
Why the split matters: the auto fields update on every retrain so the card can't lie; the hand-written fields are the human accountability that a framework actually cares about.
Context: Governance maturity is auditing once and reporting many times.
Your task: Take a set of five concrete engineering controls and map each to a NIST AI RMF function and an ISO/IEC 42001 concern, showing the overlap.
Requirements:
- Controls: model inventory, eval-gated deploy, monitoring dashboard, incident runbook, data-provenance log
- For each, name the NIST function (Govern/Map/Measure/Manage) it primarily serves
- Note where one control satisfies multiple frameworks at once
💡 Hint: Most controls map to Measure or Manage; the org-level ones map to Govern.
Show solution
Crosswalk (verify clause references against the current standards):
| Control | NIST AI RMF | ISO/IEC 42001 concern |
|---|---|---|
| Model inventory (every prod model registered + owned) | Govern | AIMS scope, roles & responsibilities, asset management |
| Eval-gated deploy (CI fails deploy on metric regression) | Measure + Manage | Performance evaluation, operational controls |
| Monitoring dashboard (drift, error rate, refusal rate) | Manage | Monitoring, measurement & continual improvement |
| Incident runbook + post-mortem-to-eval loop | Manage + Govern | Nonconformity & corrective action |
| Data-provenance log (source, license, PII status) | Map | Data governance, documented information |
Takeaway: five controls, three frameworks, one implementation. The eval-gated deploy alone touches NIST Measure and Manage and two ISO clauses — build it once, report it three ways.
Context: Governance that treats a spellchecker like a credit model gets bypassed.
Your task: Design a two-lane review gate: a fast path for low-risk systems and a full path for high-risk ones, with explicit entry criteria and required evidence for each.
Requirements:
- Define the trigger that sends a system down each lane
- List the evidence required at each gate
- Name the accountable approver per lane
- Include an escalation rule if a fast-path system is later found to be higher-risk
💡 Hint: The lane is chosen by risk tier + blast radius (reversible? touches money/health/people?).
Show solution
Lane selection. Score two axes at intake: risk tier (EU-Act-style) and blast radius (can it take an irreversible or customer-facing action?). Low tier + low blast radius → fast path; anything high on either axis → full path.
Fast path (e.g. internal spam filter). Evidence: a one-page model card + a link to the eval run. Approver: Eng Lead. SLA: async, 1 business day. Auto-approve if eval gates pass and no PII/PHI is touched.
Full path (e.g. loan scoring). Evidence: full model card, bias/robustness eval report, red-team results, data-provenance log, human-oversight design, monitoring plan, rollback plan. Approver: Governance board / risk owner, with Legal and Security consulted. SLA: scheduled review.
Escalation rule. If monitoring, an incident, or a use-case change reveals a fast-path system now has higher blast radius (e.g. someone wired the spam filter to auto-delete), it is immediately reclassified to the full path, its deploy is frozen pending review, and the misclassification becomes a governance post-mortem. The fast path is a privilege contingent on staying low-risk.
Context: A policy no one can read in five minutes is a policy no one follows.
Your task: Draft a one-page AI governance policy for a mid-size company that an engineer, a lawyer, and an exec can all act on.
Requirements:
- State scope: what counts as an 'AI system' under the policy
- State the risk-tiering rule and the review lanes
- State the mandatory artifacts (model card, audit trail, owner)
- State the frameworks you align to and the verify-against-current-text stance
- Keep it to one page; link out to detailed procedures
💡 Hint: Policy = the durable rules; procedures = the changeable how-to. Don't mix them.
Show solution
AI Governance Policy (v1) — representative one-pager.
1. Scope. Any system that uses machine learning or an LLM to inform a decision, generate content, or take an action on behalf of the company — built or bought. Excludes purely deterministic rules.
2. Risk tiering. Every AI system is classified at intake into minimal / limited / high risk using the current EU AI Act tiers as a reference, plus a blast-radius check. Classification is recorded in the model inventory.
3. Review lanes. Low-risk systems take the fast path (Eng Lead approval + model card). High-risk or high-blast-radius systems take the full path (Governance board approval + full evidence pack).
4. Mandatory artifacts. No system reaches production without (a) a named accountable owner, (b) a current model card, and (c) an audit trail. These are deploy blockers.
5. Framework alignment. We align our control set to ISO/IEC 42001 and the NIST AI RMF, and we classify EU-facing systems under the EU AI Act. Regulatory specifics are verified against the current text with Legal; this policy is not legal advice.
6. Review cadence. Policy reviewed twice a year and after any material regulatory change. Detailed procedures (templates, gates, RACI) live in linked runbooks so the policy itself stays stable.
Context: Representative scenario: a mid-size fintech proposes an LLM feature that drafts explanations of loan decisions for customers. You chair the review.
Your task: Run the full governance review end to end and produce the artifacts and decision an auditor would expect to see.
Requirements:
- Classify the system and justify the tier (verify against current text)
- List the evidence you require before approval
- Identify the top three risks and the control mitigating each
- Fill the RACI for build, deploy, and incident response
- Issue a decision: approve / approve-with-conditions / reject, with reasons
- Name what monitoring must exist post-launch
💡 Hint: The feature 'only explains' a decision — but an inaccurate explanation of an adverse credit decision is itself a regulated, high-stakes harm.
Show solution
Classification. Although the model "only explains," it is tightly coupled to a credit decision and its output goes to consumers about an adverse action — treat it as high-risk (verify against the current Act with Legal). A wrong or misleading explanation can mislead a consumer and create legal exposure under credit/consumer-protection law.
Required evidence. Full model card; faithfulness eval proving the explanation matches the actual decision factors (not a plausible confabulation); bias check across protected groups; red-team for prompt injection and for explanations that contradict the decision; data-provenance log; human-oversight design; rollback plan.
Top three risks + controls. (1) Confabulated rationale — explanation doesn't reflect the real model → force the explanation to be generated from the decision model's actual feature attributions, and gate on a faithfulness eval. (2) Inconsistent/biased explanations across groups → bias eval + monitoring on explanation content by segment. (3) Scope creep — feature starts advising customers how to 'fix' their application → block advice-language, decision-support framing only, in code.
RACI. Build: Responsible = ML eng, Accountable = Eng Lead, Consulted = Legal/Compliance, Informed = Product. Deploy: Accountable = Governance board / risk owner. Incident: Accountable = Eng Lead, Consulted = Legal + Comms.
Decision: Approve-with-conditions — ship only after the faithfulness and bias evals pass a preset bar, with a human able to review any explanation, and with the advice-language block in place. Reject if faithfulness can't be demonstrated: an explanation you can't prove is worse than none.
Post-launch monitoring: faithfulness sampling, explanation-content drift, complaint rate, and segment-level fairness — feeding the risk register and a scheduled re-review.