Monitoring, Governance & Responsible AI in Production

The four signals advanced

Start with the failure that nobody sees coming. Your app is green. Latency is flat, the error rate is a rounding error, the pager is quiet — and yet, for the last three days, one in ten answers has been confidently, fluently wrong. The provider silently rolled a new model checkpoint; your retrieval corpus went stale; a prompt tweak shipped Tuesday. Every one of those requests returned HTTP 200 in 900ms. Your dashboard is telling the truth and missing the disaster at the same time, because it was built to watch a normal web service and an LLM is not a normal web service.

Here's the analogy. Classic web monitoring is a car dashboard with two gauges: speed (latency) and the check-engine light (errors). That's enough for a car, because a car that's moving and not on fire is basically fine. But an LLM app is more like a restaurant kitchen: the food can come out fast and the kitchen can be spotless, and the dish can still be inedible — or cost you triple what you charged for it. "Fast" and "didn't crash" simply do not tell you whether the thing you shipped is any good, or whether it's quietly draining your budget one token at a time.

So an LLM app needs two more axes bolted onto the classic two. Cost is unique because inference is metered — you pay per token in and per token out, so a longer prompt, a chattier answer, or an agent that loops one extra time all cost real money, and none of that shows up as an error. Quality is unique because a request can succeed at every layer of the stack — valid JSON, 200 status, no exception — and still hand the user a hallucination. Correctness lives above the HTTP layer, so the transport can't measure it for you; you have to sample and judge the actual content.

The mistake almost everyone makes first is to treat quality and cost as things you'll "add later, once we have time." You won't — because there's no error to remind you, the gap stays invisible until a customer or a finance report finds it for you. The fix is a discipline, not a tool: decide that every single request emits all four signals — quality, cost, latency, safety — from day one, even if "quality" starts as nothing more than a thumbs-up button. A signal you don't emit per request is a signal you can never aggregate, alert on, or trend later.

Classic monitoring watches latency and errors. An LLM app needs two more axes — cost (it's metered) and quality (correct HTTP 200s can still be wrong). Watch all four or you're flying blind on the ones that matter most.

Lab O4.1 · Tracing an agent advanced

Picture the bug report you'll actually get: "the agent gave a wrong answer to this question." That's all. Now — where do you even look? The agent didn't do one thing; it did eight. It planned, searched your docs, read the results, called a tool, got an error, retried, reasoned again, and wrote a reply. A single log line saying "request failed" or even "answer: …" tells you the outcome and nothing about the eight steps that produced it. You're left re-running it by hand and hoping it misbehaves the same way twice.

The fix is a trace, and the cleanest analogy is a medical bill versus an itemized receipt. A one-line log is the total at the bottom: "$47, something went wrong." A trace is the itemized receipt — every line item, what it cost, how long it took, and a red flag next to the one that failed. You stop guessing which organ hurts and just read where the pain is. Concretely, a trace is the nested, timed tree of every step in one request: each model call, each tool call, each retrieval, with its duration, its token count, and any error, all stitched together under one shared request id.

The common mistake here is logging at the wrong altitude. Teams instrument the outer request (start, end, total time) because that's what their web framework already does — and it's useless for an agent, because the interesting failures happen inside the loop. Retrieval returned nothing, or a tool timed out on the third of five calls, or the model reasoned fine but off stale context. Whole-request logging can't distinguish any of those; they all look like "slow request, bad answer." The discipline is to log at the step level and thread one trace_id through every step, so the individual lines reassemble into the tree. That single id is the whole trick — get it flowing and everything else is just fields on a record.

For a single call, a log line is enough. For an agent — a loop of model calls, tool calls, retrievals (L4/M4) — you need a trace: the full tree of steps for one request, so when it goes wrong you can see where.

trace.py# Minimal home-grown span — one structured log line per step
import time, json, uuid

def span(trace_id, name, **fields):
    start = time.perf_counter()
    def done(**extra):
        rec = {"trace_id": trace_id, "step": name,
               "ms": round((time.perf_counter()-start)*1000), **fields, **extra}
        print(json.dumps(rec))          # ship to your log/trace backend
    return done

tid = uuid.uuid4().hex
d = span(tid, "model.plan", model="claude-opus-4-8")
resp = client.messages.create(...)
d(in_tok=resp.usage.input_tokens, out_tok=resp.usage.output_tokens,
  stop=resp.stop_reason)              # quality/cost/safety signals captured per step

Detecting drift & live regressions advanced

Here's the uncomfortable truth about a shipped LLM app: it gets worse while you sleep, and nothing tells you. You didn't change a line of code. But the world your app lives in kept moving — new kinds of questions arrived, your knowledge base aged, the provider quietly swapped the model behind the same API name. The system you validated at deploy is not the system running today, and the gap only widens. This slow, silent decay is called drift, and it's the defining operational hazard of AI systems that ordinary software just doesn't have.

The analogy that makes it click: a deploy-time eval is like a doctor's checkup. It confirms you were healthy on the day you were examined. It says nothing about the months in between — and drift is exactly the thing that develops in the months in between. You can pass every offline test with flying colors and still be degrading in production three weeks later, for reasons that have nothing to do with your code. That's why "we tested it before launch" is a necessary answer and a dangerously incomplete one.

It helps to see that drift isn't one thing but a family, and each member has a different tell. Input drift is your users changing — they start asking about topics your prompts and retrieval were never built for, and your "I don't know" rate quietly climbs. Quality drift is the answers getting worse even on familiar questions, usually from a stale corpus or accumulating edge cases. Model drift is the ground moving — the provider updates the model and behavior shifts under you. Cost drift is the bill creeping — contexts grow, agents loop more, tokens-per-request trend up. Naming them matters because the detection method differs: you cluster inputs to catch the first, judge sampled traffic to catch the second, pin model ids and replay a golden set to catch the third, and trend spend to catch the fourth.

The mistake to avoid is assuming a green error dashboard means a healthy system. Drift never throws an exception — that's precisely what makes it dangerous. The only defense is active, scheduled measurement: sample live traffic through a judge and trend the score, and re-run your golden set against production config on a timer, not just at deploy. You're installing a smoke alarm for a fire that produces no smoke.

Offline evals (Ch 5) catch regressions before deploy. But production changes underneath you — inputs shift, the corpus grows, the provider updates the model. You need to catch degradation after deploy too.

Governance: audit, access & data expert

Imagine the phone call. A customer says your AI told them something that cost them money, or a regulator asks you to produce every decision your system made about a particular person, or a security review wants to know exactly what data left your walls and went to the model provider. In that moment, "governance" stops being a compliance buzzword and becomes a single, brutally practical question: can you actually answer? Governance is the machinery that lets you say yes — the answer to "who did what, with whose data, and can we prove it?"

The analogy is a hospital, not a filing cabinet. A hospital doesn't just store records; it controls who can open a chart (access control), stamps every entry with who wrote it and when (audit log), has rules for what leaves the building and how long things are kept (data handling), and requires a second signature for high-risk procedures (human oversight). Governance for an LLM app is that same discipline pointed at a system that talks. It becomes non-optional the moment your app touches real users or regulated data — and unlike features, you can't retrofit it after an incident, because the whole point is that the records had to already exist before the question got asked.

Here's the reframing that makes it concrete for engineers, and it's the section's central idea: the trace you built in the last section is your audit trail. The nested, timed record of who called what, with which prompt version and model, at what cost — that observability artifact and the compliance artifact are the same object viewed from two angles. Access control is least privilege on your tools (an agent shouldn't hold a permission it never uses); data handling is deciding deliberately what you send to the provider and what you retain; human oversight is your HITL approval gate for the actions that can't be undone. None of this is exotic — it's ordinary engineering rigor applied on purpose.

The trap is treating governance as paperwork that lives in a wiki, separate from the running system. Real governance is enforced by the code path: the permission check that actually blocks the call, the redactor that actually scrubs the log, the approval step the agent literally cannot skip. A policy document nobody can enforce is theater; a controlled code path is governance.

Governance is the answer to "who did what, with whose data, and can we prove it?" It becomes non-optional the moment an LLM app touches real users or regulated data.

Responsible AI in production expert

There's a failure mode worse than downtime, and it doesn't page anyone. A system that's fully up — fast, cheap, error-free — can still be quietly refusing one group of users more often than another, presenting itself as human, or stating guesses with the confidence of facts. No dashboard turns red, because from the machine's point of view nothing is wrong. Responsible AI is the practice of watching for the harms that work as designed: the outputs that are technically successful and ethically or legally unacceptable.

The mental shift this section asks for is captured in one line: these are signals you monitor, not a checklist you tick. The analogy is food safety versus a health inspection. An inspection is a snapshot — pass it once, frame the certificate, and nothing stops the kitchen from getting dirty the next day. Real food safety is the ongoing habit: temperatures logged every shift, surfaces checked continuously. Responsible AI works the same way. A pre-launch "bias review" that never runs again is an inspection; tracking refusal rates across user groups every day and feeding flagged outputs back into your evals is food safety. Fairness, transparency, and safety aren't states you achieve — they're properties you continuously measure or lose.

Concretely, that turns four abstract virtues into four operational habits. Fairness means you actually test outputs across user groups and put bias cases in the golden set, rather than hoping the model is even-handed. Transparency means telling users they're talking to AI and citing your sources — the same grounding discipline from RAG, now as an honesty requirement. Accountability means a named human owns the system's behavior and the audit log makes every decision traceable back to them. Contestability means a user who gets a bad answer can flag it and reach a person, and that flag flows back into your eval set instead of vanishing.

The mistake is assuming your provider's alignment work covers you. It doesn't — the base model's safety training knows nothing about your users, your domain, or the specific ways your deployment can skew. Responsible AI is your job at the application layer, and it's continuous.

Beyond keeping the system up, governance includes keeping it fair and honest. These are operational practices, not a one-time checklist.

Lab O4.2 · The AI incident response expert

Rethink what "incident" even means for an AI system. In classic ops, an incident is something breaking: a server down, a 500 storm, a database on fire. Those still happen — but the incidents that will actually keep you up at night with an LLM are the ones where nothing broke. The system was healthy and did something harmful: it emitted an offensive answer, leaked a customer record into a reply, looped a tool call until it burned $4,000, or got talked past its guardrails by a clever prompt. The machine thinks it succeeded. Only a human — or a safety signal you set up in advance — knows it was a disaster.

Because the failure is behavioral rather than a crash, the response can't be improvised. The analogy is a fire drill: you do not want people reading the evacuation plan while the building burns. You want a runbook rehearsed cold, so that under pressure everyone knows the next move. For AI incidents that runbook has a fixed shape — detect → contain → diagnose → fix → review — and the single most important word in it is contain. Your very first move is not to understand the problem; it's to stop the bleeding: flip the feature to a safe fallback or hit the kill switch. Diagnosis can take twenty minutes, and "we'll have a fix in twenty minutes" is not an acceptable answer while an agent is actively doing harm.

This is exactly where the earlier sections pay off, and it's the reason they came first. Detect only works if you're emitting the four signals — a safety flag or cost spike is what fires the alert. Diagnose only works if you have the trace — it's what tells you which prompt version, model, input, and tool were involved. Contain only works if the kill switch already exists — a flag you can flip with no redeploy. Observability isn't a nice-to-have that supports incident response; it's the thing that makes the runbook executable at all.

The habit that separates teams that improve from teams that just firefight is the last step, and it's easy to skip because the fire is already out: every incident becomes a golden-set case. Capture the exact input that broke it, add it to your evals, and the same mistake can never ship again. Skip it and you'll re-live the same incident next quarter with a different name.

An LLM incident isn't always a crash — it's often the system confidently doing the wrong thing: a harmful output, a leaked record, a costly loop, a jailbreak that worked. You need a runbook for that, and observability is what makes it executable.

Common pitfalls expert

Exercises expert

def signals(resp):
    return {
        "quality":  resp.get("judge_score"),                 # LLM-judge / thumbs
        "cost_usd": round((resp["in_tok"]*3 + resp["out_tok"]*15)/1_000_000, 6),
        "latency":  resp["latency_ms"],                      # p50/p95/p99 later
        "safety":   resp["stop_reason"],                     # refusal / flags
    }

rec = {"judge_score": 0.92, "in_tok": 1200, "out_tok": 300,
       "latency_ms": 840, "stop_reason": "end_turn"}
print(signals(rec))

import time, json, uuid

def span(trace_id, name, **fields):
    start = time.perf_counter()
    def done(**extra):
        rec = {"trace_id": trace_id, "step": name,
               "ms": round((time.perf_counter() - start) * 1000),
               **fields, **extra}
        print(json.dumps(rec))     # ship to your backend
        return rec
    return done

tid = uuid.uuid4().hex[:8]
d = span(tid, "model.plan", model="claude-opus-4-8")
time.sleep(0.01)                    # stand in for the model call
d(in_tok=1200, out_tok=300, stop="end_turn")

def detect_drift(baseline_scores, recent_scores, drop_threshold=0.05):
    base = sum(baseline_scores) / len(baseline_scores)
    recent = sum(recent_scores) / len(recent_scores)
    drift = base - recent
    return {
        "baseline": round(base, 3),
        "recent": round(recent, 3),
        "drop": round(drift, 3),
        "drift_flag": drift > drop_threshold,
    }

baseline = [0.91, 0.90, 0.92, 0.89, 0.90]     # last week
recent   = [0.84, 0.82, 0.83, 0.85, 0.80]     # this week
print(detect_drift(baseline, recent))          # drift_flag True

import re

SECRET = re.compile(r"(sk-[A-Za-z0-9]+|\b\d{3}-\d{2}-\d{4}\b)")  # api keys, SSNs

def audit_record(raw):
    def mask(text):
        return SECRET.sub("[REDACTED]", text)[:80]   # cap + scrub
    return {
        "who": raw["user"], "when": raw["ts"],
        "prompt_version": raw["prompt_version"], "model": raw["model"],
        "cost_usd": raw["cost_usd"],
        "prompt_preview": mask(raw["prompt"]),        # scrubbed, truncated
        "output_preview": mask(raw["output"]),
    }

raw = {"user": "alice", "ts": "2026-09-07T10:00Z", "prompt_version": "v7",
       "model": "claude-opus-4-8", "cost_usd": 0.004,
       "prompt": "my key is sk-abc123 and ssn 123-45-6789",
       "output": "noted"}
print(audit_record(raw))

def replay(golden, answer_fn, gate=0.90):
    passed = sum(1 for c in golden if c["must_include"] in answer_fn(c["q"]))
    return passed / len(golden)

def scheduled_check(golden, answer_fn, last_rate, gate=0.90):
    rate = replay(golden, answer_fn, gate)
    alert = ""
    if rate < gate:
        alert = f"ALERT: pass rate {rate:.0%} below gate {gate:.0%}"
    elif last_rate is not None and last_rate - rate > 0.05:
        alert = f"ALERT: dropped {last_rate:.0%} -> {rate:.0%} (model drift?)"
    return rate, alert or "ok"

golden = [{"q": "refund window?", "must_include": "30 days"},
          {"q": "reset password?", "must_include": "settings"}]
# simulate a provider model update that broke one answer:
def answer_after_update(q):
    return {"refund window?": "14 days", "reset password?": "settings"}[q]
print(scheduled_check(golden, answer_after_update, last_rate=1.0))

def run_incident(signal, kill_switch, trace, golden):
    log = []
    log.append(f"DETECT: alert fired on {signal}")
    kill_switch["enabled"] = False                 # CONTAIN: flip to safe fallback
    log.append("CONTAIN: kill switch flipped -> safe canned fallback")
    cause = trace.get("failing_step", "unknown")
    log.append(f"DIAGNOSE: pulled trace, root cause at step '{cause}'")
    log.append("FIX: patched guardrail; ran eval gate + canary")
    golden.append({"q": trace["input"], "must_include": trace["expected"]})
    log.append(f"REVIEW: added failing case to golden set (now {len(golden)} cases)")
    return log

ks = {"enabled": True}
golden = [{"q": "x", "must_include": "y"}]
trace = {"failing_step": "tool.refund", "input": "refund $9999",
         "expected": "requires approval"}
for line in run_incident("cost_spike", ks, trace, golden):
    print(line)

Knowledge check check yourself