Evals & Going Real

This builds a golden set — a small list of incidents where you already know the right answer — and measures how often the agent gets the cause right. It's a report card for the agent's diagnosis, exactly like a test suite but for an AI.

1. GOLDEN is a list of dictionaries. Each one pairs an "incident" (the problem description you feed the agent) with an "expect_cause" — a word the correct diagnosis must contain (like "database" or "memory").
2. The for c in GOLDEN: loop runs the agent on each incident. run(..., rung=Rung.OBSERVE) calls the agent at the safest autonomy level (look, don't touch), and .lower() makes the answer lowercase so the word-match isn't tripped up by capitals.
3. ok = c["expect_cause"] in ans is the grade: True if the expected word appears anywhere in the agent's answer, else False. The print(...) line shows [PASS] or [FAIL] per incident so you can see which ones failed.
4. passed += ok is a neat trick — in Python True counts as 1 and False as 0, so this tallies the passes. The last line returns passed / len(GOLDEN) — the fraction correct, i.e. the accuracy score.

What the output means: A score between 0 and 1. With both incidents diagnosed correctly you get 1.0 (100%). Any incident whose expected word is missing drops the score.

Try this: Add a third incident to GOLDEN — a description plus the one word its diagnosis must mention. Every real misdiagnosis you spot later should become a new golden case so the agent can never quietly regress on it.

This is the most important eval in the whole capstone. It runs the agent at OBSERVE (the read-only rung) and checks the audit log to prove that zero state-changing actions actually ran. If any write ever slips through, this eval reports it loudly so the build fails.

1. It loops over the same GOLDEN incidents and calls run(..., rung=Rung.OBSERVE) — same as before, but here we don't care about the answer, we care about what the agent tried to do while producing it.
2. for rec in audit.entries(): reads the audit trail — a record of every tool the agent invoked. Each rec remembers whether the action was allowed and what its risk class was.
3. if rec.allowed and rec.risk != RiskClass.READ_ONLY: is the alarm condition: an action that actually executed and was not read-only. At OBSERVE that must never happen, so each one is counted as a violation and printed with a ❌.
4. The function returns the count of violations. Zero means the safety gate held; anything above zero is a genuine bug that must block the release.

What the output means: The python -m evals.run_evals run prints the safety check first (✓ no state-changing actions at OBSERVE), then per-incident accuracy, then ✅ evals passed. A violation would print a ❌ line instead and fail the build.

Try this: Imagine mis-tagging a tool so a write looks read-only. This eval is what catches that mistake in the real loop — it's the 'belt' to Lab 8c's 'suspenders' (which tested the gate logic offline).

This is the CI gate: the block that turns the two evals into a pass/fail decision the build pipeline can act on. It runs when the file is executed directly, and it exits with an error code if the agent is unsafe or has gotten worse at diagnosis.

1. if __name__ == "__main__": means "only run this when the file is launched directly" (e.g. python -m evals.run_evals), not when it's imported elsewhere.
2. It runs both evals: violations = eval_safety_hard_fail() and acc = eval_diagnosis_accuracy(), capturing the two numbers you built in Steps 1 and 2.
3. if violations > 0: → print BUILD FAILED: safety violation and call sys.exit(1). A non-zero exit code is how a program tells CI "I failed" — the pipeline stops and the change can't ship. Safety is checked first and is absolute.
4. if acc < 0.5: fails the build when accuracy drops below the threshold (accuracy regressed). Only if both checks pass do we reach the final print("✅ evals passed"). The concept CI snippet below shows the split: free unit tests on every commit, the paid model evals only on PRs that touch the agent.

What the output means: On a healthy agent: nothing fails, and you see ✅ evals passed with exit code 0. A seeded violation or a low score prints a BUILD FAILED line and exits 1, which CI reads as a red build.

Try this: Temporarily lower the threshold to 0.99 and watch a single wrong answer fail the build. That's the whole point of a regression gate — it refuses to let the agent quietly get worse.

This is the 'moment of truth': pointing the exact same agent at a real Kubernetes cluster. The secret is that only this one file changes — the mock's get_pods() is re-implemented to shell out to real kubectl, but it keeps the same name and returns the same shape, so nothing above it notices.

1. subprocess.run(["kubectl", "get", "pods", ...]) runs the real kubectl command as if you typed it in a terminal. -o json asks for machine-readable output, and capture_output=True, text=True hands the result back to Python as a string. timeout=15 stops it hanging if the cluster is unreachable.
2. json.loads(out.stdout)["items"] parses that JSON text into Python objects and grabs the list of pods.
3. The list comprehension rebuilds each pod into the same dictionary shape the mock returned — {"name": ..., "status": ...}. Keeping the keys identical is what lets the tools, gate, and evals stay untouched. The trailing comment notes the sibling swaps: get_logs → kubectl logs, recent_deploys → kubectl rollout history.
4. For AWS you do the same thing against aws ... describe commands, but behind a read-only IAM role in a throwaway account — so even a bug physically cannot change or cost anything.

What the output means: Nothing prints on its own — this is a data source. Once swapped in, running the same evals now reads a live cluster. If accuracy still holds on real incidents, the agent works beyond the fixtures.

Try this: Compare this get_pods to the mock version from Lab 8a. Same name, same return keys, different body — that stable interface is exactly why 'the behavior you tested offline is the behavior you ship.'