Monitoring, Governance & Responsible AI in Production
A deployed LLM app that no one is watching is a liability with a URL. This chapter is the observability pillar (O1) plus the governance and responsible-AI practices that keep a live system trustworthy — the four signals to watch, how to trace an agent, and how to run an incident when the model does something it shouldn't.
Learning objectives
- Instrument the four core signals: quality, cost, latency, and safety.
- Trace a multi-step agent so you can debug what actually happened.
- Detect drift and quality regressions in production, not just offline.
- Apply governance: audit logs, access control, data handling, human oversight.
- Run responsible-AI practice — bias, transparency, and an AI incident response.
The four signals advanced
Classic monitoring watches latency and errors. An LLM app needs two more axes — cost (it's metered) and quality (correct HTTP 200s can still be wrong). Watch all four or you're flying blind on the ones that matter most.
This diagram names the four things you must watch for a live LLM app. Normal web apps watch only two of them; the picture points out the two extra ones that catch people out.
- Read the four boxes left to right. Each is one signal — a number you collect on every request. Under each name is the plain-English question it answers.
- Quality — "is it right?" A reply can be a successful HTTP 200 and still be wrong or made-up, so correctness is its own signal, not something the server status tells you.
- Cost — "$/req, tokens." Every call is metered: you pay per token in and per token out. This is unique to LLM apps and it can quietly balloon.
- Latency — "p50/p95, TTFT." How long a reply takes.
p50is the typical request,p95is the slow tail (95% are faster), and TTFT is time-to-first-token — how fast streamed text starts appearing. - Safety — "refusals, flags." How often the model refuses, trips a guardrail, or emits something flagged. The top caption calls out that Quality and Cost (the left two) are exactly what classic monitoring misses.
In short: The bottom caption is the rule: every request should emit all four. A green latency dashboard can hide a quality collapse — watch all four axes or you're flying blind on the ones that matter most.
| Signal | What to track | How |
|---|---|---|
| Quality | Groundedness, correctness, user thumbs, task success | Live LLM-judge sampling, user feedback, golden-set replay (Ch 5) |
| Cost | Input/output tokens, $/request, spend per feature & per user | usage on every response, aggregated in the gateway (C2, O2) |
| Latency | p50/p95/p99, time-to-first-token, per-step in agents | Timing spans; TTFT matters for streamed UX (A3) |
| Safety | Refusals, guardrail triggers, injection attempts, PII leaks | stop_reason, guardrail logs, red-team alerts (T1) |
Lab O4.1 · Tracing an agent advanced
For a single call, a log line is enough. For an agent — a loop of model calls, tool calls, retrievals (L4/M4) — you need a trace: the full tree of steps for one request, so when it goes wrong you can see where.
This is a trace: the full story of a single agent request, broken into the steps it took. An agent isn't one call — it's a loop of model calls and tool calls — so you need to see the whole tree to know where things went right or wrong.
- The top bar is the whole request, with its totals:
4.2selapsed,$0.03spent, and3 tool calls. Everything below is a step inside it. - Read top to bottom = time order, and indentation = nesting. The
└marks show each row is a child step of the request above it. - The steps alternate: a model call ("plan"), then a tool call (
search_docs), another model call that uses the results, another tool call, and a final model call that writes the answer. Each row carries its own duration. - Look at
run_query · 0.6s · ERROR (retried). The trace records that this tool failed and was retried — the kind of detail that explains a slow or wrong answer.
In short: When an agent gives a bad answer, this tree tells you which step to blame — retrieval missed, a tool errored, or the model reasoned badly. Without a trace you'd just be guessing.
Uses the client / objects set up earlier in this lesson, and needs ANTHROPIC_API_KEY set. Run the earlier blocks first.
trace.py# Minimal home-grown span — one structured log line per step
import time, json, uuid
def span(trace_id, name, **fields):
start = time.perf_counter()
def done(**extra):
rec = {"trace_id": trace_id, "step": name,
"ms": round((time.perf_counter()-start)*1000), **fields, **extra}
print(json.dumps(rec)) # ship to your log/trace backend
return done
tid = uuid.uuid4().hex
d = span(tid, "model.plan", model="claude-opus-4-8")
resp = client.messages.create(...)
d(in_tok=resp.usage.input_tokens, out_tok=resp.usage.output_tokens,
stop=resp.stop_reason) # quality/cost/safety signals captured per step
This tiny helper is real, ship-today observability: it wraps each step of your agent so that every step prints one structured log line carrying its timing, tokens, and safety signal. String a trace_id through all the steps and those lines reassemble into the trace tree above.
span(trace_id, name, **fields)is called before a step. It records a start time withtime.perf_counter()(a high-precision stopwatch) and returns an innerdonefunction — that's the pattern called a closure:doneremembers the start time and the fields.- You call
done(**extra)after the step. It builds a recordrecwith thetrace_id, the stepname, the elapsed milliseconds (ms), plus any fields you passed in either call. print(json.dumps(rec))turns that record into one line of JSON. Printing JSON is the trick: any log system can parse it, so this oneprintis enough to ship to your log/trace backend.- The bottom shows it in use:
tid = uuid.uuid4().hexmakes a unique id for this request;d = span(tid, "model.plan", ...)opens the step; after the model call,d(in_tok=..., out_tok=..., stop=resp.stop_reason)closes it, capturing cost (tokens) and a safety signal (stop_reason) for that step.
What the output means: Each step prints a JSON line like {"trace_id": "…", "step": "model.plan", "ms": 1100, "model": "claude-opus-4-8", "in_tok": …, "out_tok": …, "stop": "end_turn"}. Group those lines by trace_id and you have the whole request's timeline.
Try this: Add a second span around a fake tool call (e.g. span(tid, "tool.search")) and print both lines. Sharing the same tid is what lets a tool later stitch them into one trace — exactly what LangSmith automates for you.
stop_reason — is real observability you can ship today. When manual correlation gets painful, adopt a dedicated LLM-tracing tool (LangSmith and others) that visualizes the tree, diffs runs, and links traces to evals. The concept is the same; the tool just makes it navigable. (LangSmith gets its own chapter later in the course.)Detecting drift & live regressions advanced
Offline evals (Ch 5) catch regressions before deploy. But production changes underneath you — inputs shift, the corpus grows, the provider updates the model. You need to catch degradation after deploy too.
| Drift type | What it is | How to catch it |
|---|---|---|
| Input drift | Users start asking things your prompts/RAG weren't built for | Cluster/sample live inputs; watch for new topics & rising "I don't know"s |
| Quality drift | Answers slowly get worse (stale corpus, edge cases) | Sample live traffic through an LLM-judge; track the score over time |
| Model drift | The provider updates the model; behavior shifts | Pin model IDs; run the golden set on a schedule & on version change (O1) |
| Cost drift | Tokens/request creep up (longer contexts, more tool calls) | Trend $/request; alert on step changes |
Governance: audit, access & data expert
Governance is the answer to "who did what, with whose data, and can we prove it?" It becomes non-optional the moment an LLM app touches real users or regulated data.
| Control | What it means for an LLM app |
|---|---|
| Audit log | Immutable record of every request: who, when, prompt version, model, output, cost. The trace is your audit trail |
| Access control | Who can call which capability; least privilege on tools (L3, T1); which users see which data |
| Data handling | What you send to the provider, retention, PII redaction, residency; don't log secrets or raw PII (T1) |
| Human oversight | Which actions require approval (the L5 HITL gate); who reviews flagged outputs |
| Change control | Prompt/model changes go through review + eval gate + canary (O1, O3) |
Responsible AI in production expert
Beyond keeping the system up, governance includes keeping it fair and honest. These are operational practices, not a one-time checklist.
| Practice | In production it looks like… |
|---|---|
| Fairness / bias | Test outputs across user groups; watch for skew in who gets refused or mishandled; include bias cases in the golden set |
| Transparency | Tell users they're talking to AI; cite sources (Ch 3); don't disguise limits |
| Accountability | A human owns the system's behavior; the audit log makes decisions traceable |
| Contestability | Users can flag a bad output and reach a human; feedback flows back into evals |
| Safety monitoring | Track refusals, jailbreak attempts, and harmful-output flags as first-class signals (T1) |
Lab O4.2 · The AI incident response expert
An LLM incident isn't always a crash — it's often the system confidently doing the wrong thing: a harmful output, a leaked record, a costly loop, a jailbreak that worked. You need a runbook for that, and observability is what makes it executable.
Common pitfalls expert
| Pitfall | Fix |
|---|---|
| Monitoring only latency & errors | Track all four: quality, cost, latency, safety |
| Logging single calls, not agent traces | Thread a trace id through every step of the loop |
| Evals only at deploy time | Run the golden set continuously against prod config |
| Logging raw prompts/outputs with PII | Redact, limit retention, control access to traces |
| No way to stop a misbehaving feature fast | Build a kill switch / feature flag before launch |
| Fixing an incident without a regression test | Add the failing input to the golden set every time |
| Treating responsible AI as a one-off review | Make bias/transparency/safety ongoing signals |
Exercises expert
Exercise O4.1 — Instrument the four signals
Context: The four signals only earn their keep when every request emits them and you can aggregate a day into a handful of numbers a stakeholder understands.
Your task: Wrap one LLM feature so every request emits a structured record with quality, cost, latency, and safety, then aggregate a day of it into four numbers.
Requirements:
- Emit a quality signal (sampled judge score or user thumb)
- Emit cost (tokens and dollars)
- Emit latency (ms and time-to-first-token)
- Emit safety (
stop_reasonand any guardrail hit) - Aggregate a day's records into four summary numbers
💡 Hint: Design the record first — if a field isn't emitted per request, you can't aggregate it later.
Exercise O4.2 — Trace an agent
Context: A trace only helps during an incident if it captures enough to answer "what happened and what did it cost?" from the trace alone — the gap you find in a drill is exactly what bites you live.
Your task: Thread a trace id through every model and tool call of an agent using the span helper, run a query that triggers a tool error, and confirm the trace pinpoints the failure and its cost.
Requirements:
- Thread one trace id through every model and tool call
- Trigger a tool error deliberately
- Confirm the trace shows how many model calls ran and which tools
- Confirm the trace shows which step errored, total latency, and total cost
- Add any missing fields the drill reveals
💡 Hint: If you can't answer all of those questions from the trace, the missing field is your next task — add it before the real incident.
Show what to look for
You should be able to answer, from the trace alone: how many model calls, which tools ran, which step errored, total latency, and total cost. If you can't, add the missing fields — that gap is exactly what bites you during a real incident.
Exercise O4.3 — Write the incident runbook
Context: Every system needs a written five-step runbook with the specific tools named at each step — and a kill switch. If you don't have a kill switch, discovering that is the point of the exercise.
Your task: For a system you have built, write the detect→contain→diagnose→fix→review runbook with the specific tool you would use at each step, and identify your kill switch.
Requirements:
- Name a concrete tool or signal for each of the five steps
- Identify your kill switch explicitly
- If there is no kill switch, name building one as the next task
- Keep every step actionable, not aspirational
💡 Hint: Containment is impossible without a pre-built kill switch — a flag you can flip with no redeploy — so start there if you don't have one.
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: Classic monitoring watches latency and errors but misses Quality and Cost — the two signals unique to LLM apps. Emitting all four from every request is the foundation drift detection, cost alerts, and safety review build on.
Your task: Given a raw model response record, extract one metric for each of the four signals: Quality, Cost, Latency, Safety.
Requirements:
- Pull a quality signal (e.g. a judge score or thumb)
- Compute cost in dollars from input/output token counts
- Pull latency from the record
- Pull a safety signal (e.g.
stop_reason) - Return all four from one response record
💡 Hint: One response feeds all four dashboards — the point is that a single record carries quality, cost, latency, and safety together.
Show solution
Pull all four signals from one response record (pure stdlib):
def signals(resp):
return {
"quality": resp.get("judge_score"), # LLM-judge / thumbs
"cost_usd": round((resp["in_tok"]*3 + resp["out_tok"]*15)/1_000_000, 6),
"latency": resp["latency_ms"], # p50/p95/p99 later
"safety": resp["stop_reason"], # refusal / flags
}
rec = {"judge_score": 0.92, "in_tok": 1200, "out_tok": 300,
"latency_ms": 840, "stop_reason": "end_turn"}
print(signals(rec))Classic monitoring watches latency and errors but misses Quality and Cost — the two signals unique to LLM apps. Emitting all four from every request is the foundation everything else (drift detection, cost alerts, safety review) builds on.
Context: Multi-step agents need per-step traces. A span helper that times a step and emits a structured record lets a request's model, tool, and retrieval calls reassemble into a step tree — your debugger and audit trail in one.
Your task: Implement a span helper that times a step and emits a structured JSON record with a trace id, step name, elapsed ms, tokens, and stop reason.
Requirements:
- Correlate records by a shared
trace_id - Time the step from start to a
done()call - Emit one structured JSON line per step
- Carry step name, elapsed ms, tokens, and stop reason
- Demonstrate timing a stand-in model call
💡 Hint: Return a closure from span() that captures the start time and, when called, prints the finished record — one line per step, keyed by trace id.
Show solution
The lesson's span helper — self-contained and runnable (prints structured logs):
import time, json, uuid
def span(trace_id, name, **fields):
start = time.perf_counter()
def done(**extra):
rec = {"trace_id": trace_id, "step": name,
"ms": round((time.perf_counter() - start) * 1000),
**fields, **extra}
print(json.dumps(rec)) # ship to your backend
return rec
return done
tid = uuid.uuid4().hex[:8]
d = span(tid, "model.plan", model="claude-opus-4-8")
time.sleep(0.01) # stand in for the model call
d(in_tok=1200, out_tok=300, stop="end_turn")Each span emits one line correlated by trace_id, so a request's model calls, tool calls, and retrievals reassemble into a step tree with timing, tokens, and cost per step. That tree is both your debugger and your audit trail.
Context: Quality drift is answers slowly worsening — from a provider model update, corpus drift, or new user topics — and it never throws an error. Trending sampled judge scores against a baseline turns "answers feel worse" into an alert.
Your task: Implement a rolling-window drift detector that flags drift when the recent mean judge score drops more than a threshold below the baseline mean.
Requirements:
- Compute baseline and recent mean scores
- Compute the drop between them
- Flag drift when the drop exceeds a threshold
- Return the baseline, recent, drop, and flag together
- Demonstrate a drifting window firing the flag
💡 Hint: It is just two means and a threshold — confirm any flag by replaying the golden set on the current config.
Show solution
A rolling-window drift detector (pure stdlib, runnable):
def detect_drift(baseline_scores, recent_scores, drop_threshold=0.05):
base = sum(baseline_scores) / len(baseline_scores)
recent = sum(recent_scores) / len(recent_scores)
drift = base - recent
return {
"baseline": round(base, 3),
"recent": round(recent, 3),
"drop": round(drift, 3),
"drift_flag": drift > drop_threshold,
}
baseline = [0.91, 0.90, 0.92, 0.89, 0.90] # last week
recent = [0.84, 0.82, 0.83, 0.85, 0.80] # this week
print(detect_drift(baseline, recent)) # drift_flag TrueSilent quality decay — from a provider model update, corpus drift, or new user topics — never throws an error. Trending sampled judge scores against a baseline turns "answers feel worse lately" into an alert, which you confirm by replaying the golden set on the current config.
Context: The trace is the audit trail, but full prompts and outputs may hold PII or secrets — and a log that leaks PII is itself a governance failure. You keep the operational fields but redact and truncate raw content.
Your task: Implement a redactor that keeps the who/when/version/cost fields while masking and truncating raw prompt and output content before it is logged.
Requirements:
- Keep operational fields (user, timestamp, prompt version, model, cost)
- Mask secret patterns such as API keys and SSNs in the content
- Truncate the raw content preview to a capped length
- Return the redacted record ready to log
- Demonstrate on content that contains a fake key and SSN
💡 Hint: A regex substitution plus a length cap does the masking — the log must be complete enough to reproduce a decision yet safe enough to store.
Show solution
Redact before logging — a governance requirement, not an afterthought (pure stdlib):
import re
SECRET = re.compile(r"(sk-[A-Za-z0-9]+|\b\d{3}-\d{2}-\d{4}\b)") # api keys, SSNs
def audit_record(raw):
def mask(text):
return SECRET.sub("[REDACTED]", text)[:80] # cap + scrub
return {
"who": raw["user"], "when": raw["ts"],
"prompt_version": raw["prompt_version"], "model": raw["model"],
"cost_usd": raw["cost_usd"],
"prompt_preview": mask(raw["prompt"]), # scrubbed, truncated
"output_preview": mask(raw["output"]),
}
raw = {"user": "alice", "ts": "2026-09-07T10:00Z", "prompt_version": "v7",
"model": "claude-opus-4-8", "cost_usd": 0.004,
"prompt": "my key is sk-abc123 and ssn 123-45-6789",
"output": "noted"}
print(audit_record(raw))An audit log that leaks PII is itself a governance failure. You keep the who/when/version/cost that make decisions traceable, but redact and truncate raw content and set retention limits — the log must be complete enough to reproduce a decision yet safe enough to store.
Context: Deploy-time evals catch your own changes; scheduled replay catches the provider's. Providers update models silently, so pinning ids and re-running the golden set on a cron is how a silent update becomes a visible alert.
Your task: Model a scheduled golden-set replay that alerts when the pass rate drops below the gate or falls sharply versus the last run.
Requirements:
- Compute the pass rate over a golden set with a stubbed answer function
- Alert when the rate is below the gate
- Alert when the rate dropped sharply versus the previous run
- Return the rate and an alert-or-ok message
- Simulate a provider update that breaks one answer
💡 Hint: Two triggers — an absolute gate and a relative drop-versus-last-run — catch both a bad deploy and a slow provider drift.
Show solution
Scheduled golden-set replay with a regression alert (pure stdlib):
def replay(golden, answer_fn, gate=0.90):
passed = sum(1 for c in golden if c["must_include"] in answer_fn(c["q"]))
return passed / len(golden)
def scheduled_check(golden, answer_fn, last_rate, gate=0.90):
rate = replay(golden, answer_fn, gate)
alert = ""
if rate < gate:
alert = f"ALERT: pass rate {rate:.0%} below gate {gate:.0%}"
elif last_rate is not None and last_rate - rate > 0.05:
alert = f"ALERT: dropped {last_rate:.0%} -> {rate:.0%} (model drift?)"
return rate, alert or "ok"
golden = [{"q": "refund window?", "must_include": "30 days"},
{"q": "reset password?", "must_include": "settings"}]
# simulate a provider model update that broke one answer:
def answer_after_update(q):
return {"refund window?": "14 days", "reset password?": "settings"}[q]
print(scheduled_check(golden, answer_after_update, last_rate=1.0))Deploy-time evals catch your own changes; scheduled replay catches the provider's. Pinning ids and re-running the golden set on a cron (and on every version bump) is how a silent model update becomes a visible alert instead of a slow, unexplained quality slide.
Context: As incident lead you execute the runbook — detect, contain, diagnose, fix, review — where contain flips a pre-built kill switch and review adds the failing case to the golden set so the same mistake cannot ship twice.
Your task: Drive the incident runbook programmatically as an executable checklist that ends by adding the failing case to the golden set.
Requirements:
- Walk the five steps: detect, contain, diagnose, fix, review
- Flip a kill switch to a safe fallback in the contain step
- Use the trace to name the failing step in diagnose
- Add the failing case to the golden set in review
- Return the ordered log of what the runbook did
💡 Hint: Containment needs a flag (no redeploy) and diagnosis needs the trace — which is why the kill switch and audit trail are design requirements, not reactions.
Show solution
The incident runbook as an executable checklist (pure stdlib):
def run_incident(signal, kill_switch, trace, golden):
log = []
log.append(f"DETECT: alert fired on {signal}")
kill_switch["enabled"] = False # CONTAIN: flip to safe fallback
log.append("CONTAIN: kill switch flipped -> safe canned fallback")
cause = trace.get("failing_step", "unknown")
log.append(f"DIAGNOSE: pulled trace, root cause at step '{cause}'")
log.append("FIX: patched guardrail; ran eval gate + canary")
golden.append({"q": trace["input"], "must_include": trace["expected"]})
log.append(f"REVIEW: added failing case to golden set (now {len(golden)} cases)")
return log
ks = {"enabled": True}
golden = [{"q": "x", "must_include": "y"}]
trace = {"failing_step": "tool.refund", "input": "refund $9999",
"expected": "requires approval"}
for line in run_incident("cost_spike", ks, trace, golden):
print(line)The lead's discipline is the last step: every incident becomes a golden-set case, so the same mistake cannot ship twice. Containment needs a pre-built kill switch (a flag, no redeploy), and diagnosis needs the trace — which is why the audit trail and kill switch are design requirements, not reactions.
✓ Checkpoint — you can move on when you can…
- Name the four signals and why cost & quality are LLM-specific.
- Trace an agent request and read the step tree.
- Describe input, quality, model, and cost drift and how to catch each.
- List the governance controls and why logging is a data decision.
- Run the AI incident runbook and explain the kill switch & golden-set feedback loop.
Knowledge check check yourself
Classic monitoring watches latency and errors. Which two additional signals does an LLM app require, and why can't the classic two catch them?
Show answer
Why is a single log line insufficient for an agent, and what does the lesson's 'every incident becomes a test case' habit achieve?