Part V · Build Lab A

Setup, Schema & the Mock Cluster

Q: Why is RiskClass defined as an Enum rather than a plain string, given how the safety gate uses it?

The gate does comparisons like if risk == RiskClass.IRREVERSIBLE. A typo-proof enum means an action can never accidentally fall through the safety check because someone wrote 'irreversable' — an invalid value is impossible, so a mistagged action can't silently bypass the gate.

Chapter 8 gave you the design. Now you build it, step by step, exactly like the earlier chapters' labs. This first lab lays the foundation: the project skeleton, the risk-classification schema that drives all safety, and a mock cluster so everything runs locally with zero cloud cost or risk. By the end you'll have passing tests and no API calls yet.

⏱️ ~45 min🧪 mock-first💻 no API key needed yet✅ test cases included

Learning objectives

Create the capstone project structure alongside the course starter kit.
Define RiskClass and Diagnosis — the shapes everything else depends on.
Build a mock Kubernetes cluster from JSON fixtures (deterministic, offline).
Write your first tests and run them green — before spending a cent on the API.

Ready-made referenceEvery file in these labs already exists, complete, in llm-course-starter/devops-agent/. You can build along by typing each file yourself (recommended — it's how you learn), or read the finished version there and run it. The labs explain why each piece is shaped the way it is.

What you'll build across the four build labs intermediate

Imagine being handed the keys to production on your first day and told "go fix things." Terrifying, right? You'd have no idea which levers are safe to pull and which ones page the whole team at 3am. That's the exact position an autonomous DevOps agent is in — except it can pull those levers thousands of times a second and never feels the fear. The four labs ahead exist to solve one problem: how do you build something that acts on real infrastructure without ever letting it act recklessly while you're still figuring it out?

The answer is to build the way a careful surgeon trains — on models and cadavers long before a living patient. You assemble the agent one layer at a time, and every layer is proven safe and correct in isolation before the next is stacked on top. Think of it like constructing a building: you don't wire the lights before the foundation is poured and inspected. Each lab pours one floor, and you don't move up until the one below is signed off.

The common mistake here is the tempting shortcut — "let me just point it at the real cluster to see if it works." That instinct is how people delete a production namespace during a demo. The discipline this diagram encodes is the opposite: the real cluster is the last thing you connect, not the first. Everything is built, run, and tested against a fake environment first, so that by the time you flip the switch to real infrastructure (in Lab 8d), there are no surprises left — only a system whose every behaviour you've already watched happen safely a hundred times. Read the diagram below bottom-up; it's the whole capstone in one picture.

Build bottom-up, mock-first. 8a lays a tested offline foundation; 8b makes it act (read-only); 8c grounds it in runbooks and wraps every action in the safety gate; 8d proves it with evals and swaps the mock for real infrastructure. Each layer is green before the next is added.

🗺️ How to read this diagram

This is the map for the whole capstone. It shows the DevOps agent as a stack of four layers, one per lab (8a, 8b, 8c, 8d). You are building the bottom layer now; each later lab sits on top of the ones below it.

Read it bottom-up. The top row (8a) is what you build in this lesson: the risk schema, the mock cluster, and offline tests. It sits at the base of everything — nothing above works until this is solid.
8b adds the tool registry + agent loop so the thing can actually look at the cluster (read-only) and reason about it — that turns the foundation into a diagnostician.
8c adds RAG over runbooks (so answers are grounded in the company's real docs) and the policy gate (so risky actions are blocked or need approval), plus an audit trail.
8d adds evals — automated tests of accuracy and safety that must pass — and only then swaps the mock for real infrastructure.
The blue line at the bottom is the golden rule: mock-first. Everything runs and is tested against a fake cluster before it is ever pointed at real infra.

In short: Each layer is proven green before the next is added. So when something breaks in 8c, you already trust 8a and 8b — the bug is almost certainly in the layer you just added.

Lab	You build	Result
8a (here)	Skeleton, risk schema, mock cluster, first tests	Foundation runs offline, tests green
8b	Tool registry + the agent loop	A working read-only diagnostician
8c	RAG over runbooks + the policy gate + audit	Grounded, safe, gated actions
8d	Evals (incl. hard-fail safety) + go-real	Trustworthy, CI-gated, real-infra ready

Step 1 · Create the project skeleton intermediate

Every project you'll ever regret started the same way: "I'll just organise it later." Later never comes. The folders sprawl, imports break in ways nobody can explain, and a week in you're spending more time fighting ModuleNotFoundError than writing the agent. The five minutes you spend on a clean skeleton now is the cheapest insurance you will ever buy.

A good project layout is like a well-organised kitchen: the knives live in one drawer, the spices on one shelf, and anyone — including you, three months from now — can find things without opening every cupboard. Here that means a clear home for each concern: agent/ is the brain, mock/ is the fake world it practises on, runbooks/ holds the docs it will learn from, tests/ and evals/ hold the proofs that it works. Because you're building inside the existing course starter kit, you also inherit its virtualenv and .env — no re-installing, no fresh API keys, nothing to configure twice.

The one detail that trips up almost everyone is the pair of empty __init__.py files. They look like clutter — why create empty files? — but they are load-bearing: an empty __init__.py is what tells Python "this folder is a package you can import from." Skip them and a perfectly correct from mock import cluster later in the lab fails with a baffling error, and you'll waste twenty minutes suspecting your logic when the real culprit is a missing zero-byte file. Create them now, up front, and that entire class of confusion never happens. This whole step runs offline — no API key, no network, no cost.

A home for every concern. Five sub-folders, each owning one job; the two empty __init__.py files turn agent/ and mock/ into packages so from mock import cluster works later.

🗺️ How to read this diagram

Read it top-down: the top box is the project root, and each indented box below is one sub-folder with the single concern it owns.

The top box is devops-agent/, the project root you create inside the course starter kit.
The five rows under it are the sub-folders, each labelled with its one job: agent/ is the brain, mock/ is the fake world, runbooks/ is the RAG source, tests/ and evals/ hold the proofs.
The · __init__.py tag on agent/ and mock/ is the payoff detail — those two empty files are what make the folders importable Python packages. The blue line reminds you they are the load-bearing part.

In short: One folder per concern keeps imports sane; the two __init__.py files are what stop ModuleNotFoundError later.

Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.

Lab 8a · Step 1

Work inside the course starter kit so you reuse its virtualenv and .env.

terminalcd llm-course-starter
source .venv/bin/activate            # same venv as chapters 1-6
pip install pytest                    # for the test cases (also in requirements.txt)

mkdir -p devops-agent/{agent,mock,runbooks,tests,evals}
cd devops-agent
touch agent/__init__.py mock/__init__.py   # make them importable packages

Structure you're creating:

devops-agent/
├── agent/   # the brain: schema, tools, policy, rag, audit, loop
├── mock/    # fake cluster + actions (no real infra)
├── runbooks/# the company's docs (RAG source)
├── tests/   # safety tests — no API key needed
└── evals/   # accuracy + safety evals

▶ How this works

Before writing any logic you lay down the folder structure. These terminal commands create the project inside the course starter kit so it reuses the same virtualenv and .env file you set up in earlier chapters — no new setup.

cd llm-course-starter then source .venv/bin/activate moves into the starter kit and turns on its virtual environment — an isolated Python with the course's libraries already installed.
pip install pytest adds the test runner you'll use in Step 5.
mkdir -p devops-agent/{agent,mock,runbooks,tests,evals} creates the project folder and its five sub-folders in one line. The {...} is shell shorthand that expands to five separate mkdirs.
touch agent/__init__.py mock/__init__.py creates empty __init__.py files. That one detail is what makes agent and mock real Python packages you can import from — without them, from mock import cluster later would fail.

What the output means: No visible output — success is silent. Afterwards the tree shown below exists: agent/ (the brain), mock/ (the fake cluster), runbooks/, tests/, and evals/.

Try this: Run ls devops-agent to confirm the five folders exist. If a later step says No module named 'mock', the missing __init__.py is the usual culprit — check it's there.

Step 2 · The risk schema — the spine of all safety intermediate

Not all mistakes cost the same. An agent that reads a log file and gets it wrong wasted a few tokens. An agent that deletes the wrong pod in production just caused an outage. Any safety system worth the name has to know the difference before it can protect you — and the very first thing you build here is the vocabulary for that difference. This is not glamorous code; it's four words in an enum. But it is the single most important type in the entire project, because every later safety decision is a lookup against it.

Think of it like the classification stickers on hazardous materials in a lab: a bottle marked "corrosive" is handled differently from one marked "inert," and the handling rules key off the label, not the chemist's memory. Here the labels are READ_ONLY (just looking), REVERSIBLE (a restart or a PR you can undo), SIGNIFICANT (a non-prod apply, harder to walk back), and IRREVERSIBLE (a delete or a prod apply — the bottle with the skull on it). Once every tool wears one of these stickers, the safety gate in Lab 8c doesn't need to understand what a tool does; it only needs to read the sticker.

The common mistake — and it's a subtle, seductive one — is to represent risk as a plain string. It works right up until someone writes "irreversable" (note the typo), and suddenly you have an action tagged with a risk level the gate has never heard of, so it silently sails through unguarded. An enum makes that impossible: the set of legal values is closed and fixed, and a misspelling is a crash at definition time, not a breach at runtime. That's why this section comes first, before a single tool exists — you define the danger scale before you build anything that can be dangerous. Before any tool or agent, define how risky an action is. Every tool will be tagged with one of these, and the safety gate (Lab 8c) reads the tag to decide what's allowed. This is the most important type in the whole project.

A closed danger scale, from looking to deleting. Every tool later wears exactly one of these four labels; the gate reads the label, never guesses. Because it's an enum, a misspelling is impossible.

🗺️ How to read this diagram

Read it bottom-to-top as a rising staircase: each higher, further-right step is a more dangerous class of action.

The green bottom step is READ_ONLY — just inspecting; nothing can break.
The yellow and orange middle steps are REVERSIBLE (a restart or PR you can undo) and SIGNIFICANT (a non-prod apply that's harder to walk back).
The red top step is IRREVERSIBLE — a delete or prod apply, the bottle with the skull on it. The rising, right-shifting shape encodes the danger gradient the gate keys off.
Because these four are a fixed enum, no fifth, misspelled value can ever exist — a typo crashes at definition, never sails past the gate.

In short: Four labels, low to high danger, closed and typo-proof — so tagging a tool is the whole safety decision.

Lab 8a · Step 2

agent/schemas.pyfrom enum import Enum
from typing import Optional
from pydantic import BaseModel, Field

class RiskClass(str, Enum):
    READ_ONLY = "read_only"        # 🟢 inspect only
    REVERSIBLE = "reversible"      # 🟡 scale, restart, open PR
    SIGNIFICANT = "significant"    # 🟠 apply non-prod, merge PR
    IRREVERSIBLE = "irreversible"  # 🔴 delete, prod apply

class Diagnosis(BaseModel):
    likely_cause: str
    evidence: list[str]                # the facts it's citing
    suggested_fix: str
    fix_risk: RiskClass                 # how risky the fix is
    confidence: float = Field(ge=0.0, le=1.0)
    runbook_ref: Optional[str] = None

(The finished file also has a ToolCallRecord for the audit log — see Lab 8c.)

▶ How this works

This file defines the two data shapes the whole project is built around, using Pydantic — a library that lets you declare exactly what fields a piece of data has and what type each one is, then checks the data matches. Think of it as a typed form the rest of the code must fill in correctly.

class RiskClass(str, Enum) is an enum: a fixed set of allowed values. An action is exactly one of READ_ONLY, REVERSIBLE, SIGNIFICANT, or IRREVERSIBLE — the four risk levels, from 'just looking' (🟢) to 'delete / apply in prod' (🔴).
class Diagnosis(BaseModel) is the structured answer the agent must produce. BaseModel means Pydantic will validate every field.
The fields spell out a complete diagnosis: likely_cause (a sentence), evidence (a list[str] — the facts it cites), suggested_fix, and fix_risk which is a RiskClass — so the fix is tagged with its danger level.
confidence: float = Field(ge=0.0, le=1.0) forces the number to sit between 0 and 1 (ge = greater-or-equal, le = less-or-equal). runbook_ref: Optional[str] = None is an optional pointer to a doc, defaulting to nothing.

What the output means: Nothing runs — this is a definition. But now anywhere in the code, a Diagnosis is guaranteed to have these fields with these types, or Pydantic raises an error before the bad data can spread.

Try this: Later the safety gate will read fix_risk to decide what's allowed. Because it's an enum, a typo like "irreversable" is impossible — that is the whole point of using an enum instead of a plain string.

Why an enum, not free textBecause the safety gate does if risk == RiskClass.IRREVERSIBLE. A typo-proof enum means an action can never accidentally fall through the safety check because someone wrote "irreversable".

Step 3 · The mock cluster advanced

Here's a sentence that should make you nervous: "let's just test it against the real cluster." That is how staging outages become production outages, how a debugging session turns into an incident, and how a half-finished agent gets to practise its worst mistakes on infrastructure people depend on. The professional move — the one that separates engineers who ship dangerous systems from those who cause postmortems — is to build against a fake cluster first.

A mock is not a toy and it is not a shortcut. Think of a flight simulator: pilots log hundreds of hours in one before they ever touch a real cockpit, precisely because the simulator lets them practise engine failures, storms, and instrument faults without anyone dying. Our mock cluster does the same job. It gives you three superpowers you can never get from real infrastructure: it's deterministic (the same crash-looping pod is there every single run, so a test that passes today passes tomorrow), it's free (no cloud bill, no API key, no rate limits), and it is physically incapable of breaking anything — there is no real pod behind it to break. That last property is what lets you develop fearlessly.

The mistake to avoid is making the fake world too clean. A mock full of healthy, boring pods teaches the agent nothing, because there's no problem to solve — it's like a flight simulator that only ever shows clear skies. So the fixtures are seeded with genuine trouble: a pod stuck in CrashLoopBackOff whose logs contain the real root cause (password authentication failed), an OOMKilled pod, and — the detail that makes it a real diagnostic puzzle — a deploy from 18 minutes ago that changed the very secret the failing pod can't authenticate with. A good agent should connect those dots, and because the trouble is baked into the fixture, you can test whether it does, every time, for free. We build against a fake cluster first. This is not a shortcut — it's the professional way to develop something dangerous: deterministic, free, and incapable of breaking anything. The fixtures include a crash-looping pod (with the real error in its logs) and an OOMKilled pod, so the agent has real problems to solve.

Read one way only. The JSON fixture is loaded once and the accessors hand data out; there is no path from an accessor back into the state — the mock is structurally read-only.

🗺️ How to read this diagram

Follow the blue arrows left to right: the fake state becomes the accessor's input, and the accessor returns data — the flow only ever goes one direction.

The left box is fixtures.json — the whole cluster's state written down (the crash-looping pod, its logs, the recent deploy).
The middle box is cluster.py, loaded once into _STATE, exposing only reader functions: get_pods, get_pod, get_logs.
The green right box is the pod data handed back to the caller. The dashed loop at the bottom is crossed out in words: no arrow points back into the state — a read can never mutate the fixture.

In short: Data flows fixture → reader → caller, one way; because nothing writes back, the mock is deterministic and impossible to break.

Lab 8a · Step 3

The fixture data — the fake cluster state.

mock/fixtures.json (excerpt){
  "pods": [{
    "name": "checkout-api-7d9f", "namespace": "staging",
    "status": "CrashLoopBackOff", "restarts": 7,
    "logs": [
      "INFO  connecting to postgres...",
      "ERROR password authentication failed for user 'checkout'",
      "ERROR startup probe failed, exiting"
    ]
  }],
  "deploys": [{"name": "checkout-api", "revision": 12,
    "changed": "updated DATABASE_URL secret ref", "when": "18 min ago"}]
}

Notice the story hidden in the data: the DB auth fails, and 18 minutes ago someone changed the DATABASE_URL secret. A good agent should connect those.

The read-only accessor — this is what tools will call.

Illustrative fragment — defines demo values / files are needed before this runs standalone.

mock/cluster.pyimport json, os
_STATE = json.load(open(os.path.join(os.path.dirname(__file__), "fixtures.json")))

def get_pods(namespace="staging"):
    return [p for p in _STATE["pods"] if p["namespace"] == namespace]

def get_logs(pod, namespace="staging", lines=20):
    p = get_pod(pod, namespace)
    if not p: return f"Error: pod '{pod}' not found"
    return "\n".join(p.get("logs", [])[-lines:])
# ... get_pod, get_events, recent_deploys (see the finished file)

▶ How this works

The mock cluster is a fake Kubernetes cluster that lives entirely in a JSON file. It lets you develop something dangerous safely: it's deterministic (same answer every time), free, and physically cannot break anything real. Two files make it up — the data (fixtures.json) and the reader (cluster.py).

fixtures.json is just the cluster's state written down: a list of pods and deploys. One pod is stuck in CrashLoopBackOff with real error logs, and a deploy 18 minutes ago changed the DATABASE_URL secret — that's a deliberate story the agent should piece together.
In cluster.py, _STATE = json.load(open(...)) reads that JSON once into memory. The os.path.dirname(__file__) part means it finds the file next to the module, no matter which folder you run from.
get_pods(namespace="staging") returns the pods in a namespace — a plain list comprehension that filters _STATE["pods"].
get_logs(pod, ...) looks the pod up, and if it isn't found returns a friendly "Error: pod '...' not found" string instead of crashing. Otherwise it joins the last lines log entries with newlines.

What the output means: Every function only reads — it hands back data and changes nothing. get_logs("checkout-api-7d9f"), for example, returns the three log lines including the password authentication failed error.

Try this: Because it's all read-only, this is exactly the surface the read-only diagnostician in Lab 8b needs. Actions that change things live in a separate file (Step 4) so the safety gate can be tested without any risk.

Everything here is read-onlyThe mock cluster only returns data. There's no way to change it — which is exactly right for the read-only diagnostician you build in Lab 8b. State changes are simulated separately (Step 4) so we can prove the safety gate without touching anything.

Step 4 · Simulated (fake) actions advanced

You cannot test a fire alarm by starting a real fire. Yet the whole point of the agent is that it will eventually act — restart a rollout, scale a deployment, open a pull request, maybe delete a pod — and you need to know the safety gate catches the dangerous ones before you trust it near anything real. The resolution is the same one every serious system uses: you test the alarm with a button, not a blaze. These simulated actions are that button.

Each action is a function that returns a description of what it would do and changes absolutely nothing — rollout_restart hands back "[mock] restarted rollout staging/checkout-api", and no rollout is touched. The [mock] prefix is deliberate: it's a constant, visible reminder that you are looking at a rehearsal, not a performance. Because the return is just text, you can wire every one of these into the full safety gate in Lab 8c and watch it allow the harmless ones, block the dangerous ones, and pause for approval on the in-between — all with zero chance of collateral damage. The gate is exercised for real; only the consequences are fake.

The design decision worth internalising is why these live in a separate file from the reads. It would be easy to bundle "get the pods" and "delete a pod" into one cluster.py — they're both about the cluster, after all. But keeping the reads (Step 3) and the writes (here) in different modules is what lets the read-only diagnostician in Lab 8b import no function that can mutate anything. That's the difference between "safe because we were careful" and "safe because it is structurally impossible to be otherwise" — and the second kind is the only kind you should trust with production. The agent will eventually want to do things (restart, scale, open a PR). While building, those are simulated — they return a string saying what they'd do, and change nothing. This lets you exercise the entire safety gate in Lab 8c with zero risk.

Safety by construction, not by promise. Reads and writes are separate modules, so the read-only diagnostician imports cluster.py only and literally holds no function that can mutate state.

🗺️ How to read this diagram

Start at the top box (the agent) and follow which module it is allowed to import — the solid arrow is the connection that exists, the dashed one is the connection that deliberately does not.

The top box is the read-only diagnostician you build in Lab 8b.
The green left box is cluster.py — the read accessors that can only look. The solid blue "imports" arrow shows the diagnostician wires to it.
The right box is actions.py — the simulated writes (restart, open PR, delete). The dashed "does NOT import" line is the whole point: the diagnostician never pulls in a mutating function.

In short: Splitting reads from writes means the diagnostician has no way to change anything — a far stronger guarantee than "we promise not to".

Lab 8a · Step 4

mock/actions.pydef rollout_restart(name, namespace="staging"):
    return f"[mock] restarted rollout {namespace}/{name}"

def open_pr(title, body, branch):
    return f"[mock] opened PR '{title}' from branch '{branch}'"

def delete_pod(name, namespace="staging"):
    return f"[mock] DELETED pod {namespace}/{name}"   # never actually runs at OBSERVE
# ... scale_deployment, terraform_apply

▶ How this works

These are the actions the agent might eventually take — restart a rollout, open a pull request, delete a pod. While you're building, they are simulated: each just returns a string describing what it would do and changes nothing at all.

Every function returns an f-string like f"[mock] restarted rollout {namespace}/{name}". The [mock] prefix is a constant reminder that nothing real happened.
The functions span the risk range on purpose: rollout_restart and open_pr are reversible, while delete_pod is irreversible — the dangerous kind the safety gate must guard.
Because they only return text, you can wire them into the full safety gate in Lab 8c and watch it allow, block, or ask-for-approval on each one — with zero chance of touching real infrastructure.

What the output means: Calling any of these just gives you a [mock] ... sentence. delete_pod("x") returns "[mock] DELETED pod staging/x" and the real cluster is untouched.

Try this: Splitting reads (cluster.py) from writes (actions.py) is the key design move: it means the diagnostician literally has no function that can change anything, which is a much stronger safety guarantee than 'we promise not to'.

Step 5 · Write and run your first tests expert

Here's a quiet truth about building agents: the LLM is the least trustworthy part of your system, and also the most expensive to debug through. If your foundation is shaky and your agent gives a wrong answer, you'll spend hours squinting at prompts and model outputs when the real bug was a mock that loaded the wrong fixture. Tests written before the model is involved are how you take that entire failure mode off the table.

A test is just a small function that states an expectation and checks it automatically — "the crash-looping pod should exist," "its logs should contain the root cause," "asking for a pod that isn't there should return an error, not explode." Think of them as the guardrails on a mountain road: you don't notice them when you're driving carefully, but the moment you drift they stop you going over the edge. Every time you change the mock later, these tests re-run in a fraction of a second and tell you instantly whether you broke the foundation everything else stands on.

The mistake beginners make is treating tests as a chore you do after the code works — or skipping them because "it's just a mock." But that inverts their entire value. Running these green here, with no API key and no cost, is the whole mock-first payoff: it means when you add the agent loop in Lab 8b and something misbehaves, you already trust the layer beneath it, so the bug has to be in the new code. You've turned an open-ended "why is my agent wrong?" into a narrow "what did I just change?" That confidence, bought cheaply and early, is what makes the rest of the capstone tractable. Before any API call, prove the foundation works. These tests need no API key — they check the mock and (in Lab 8c) the safety logic directly.

Green before the model. Three cheap assertions prove the fixtures load, the root cause is present, and bad input is handled — all offline, so any later agent bug lives in the new code, not the base.

🗺️ How to read this diagram

Read the three top boxes left to right — each is one test — then follow the arrows down to the single result they must produce.

The three green boxes are the tests: the crash-loop pod is found (fixtures load), the root-cause string is in its logs (real evidence exists), and a missing pod returns an error instead of crashing.
The arrows converge on the outcome you want: 3 passed in 0.04s — fast, free, and with no API key.
The blue line is the payoff: a trusted foundation before the model exists, so when the agent misbehaves in Lab 8b the bug has to be in the new layer.

In short: Three offline asserts turn "why is my agent wrong?" into "what did I just change?" — that's the mock-first payoff.

Lab 8a · Step 5

tests/test_mock_and_tools.pyimport os, sys
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from mock import cluster

def test_mock_finds_crashloop_pod():
    pod = cluster.get_pod("checkout-api-7d9f")
    assert pod is not None
    assert pod["status"] == "CrashLoopBackOff"

def test_mock_logs_contain_root_cause():
    assert "password authentication failed" in cluster.get_logs("checkout-api-7d9f")

def test_missing_pod_is_handled():
    assert cluster.get_logs("does-not-exist").startswith("Error")

Run them:

terminalcd devops-agent
python -m pytest tests/ -v

tests/test_mock_and_tools.py::test_mock_finds_crashloop_pod PASSED
tests/test_mock_and_tools.py::test_mock_logs_contain_root_cause PASSED
tests/test_mock_and_tools.py::test_missing_pod_is_handled PASSED
3 passed in 0.04s

▶ How this works

This is your first test file. Tests are small functions that check your code does what you expect, automatically. These need no API key and cost nothing — they just poke the mock cluster and assert the answers are right.

The two sys.path.insert(...) / from mock import cluster lines at the top make the mock package importable when pytest runs the file.
Each def test_...() is one test. pytest finds every function whose name starts with test_ and runs it. Inside, assert checks a condition — if it's false, the test fails and pytest tells you exactly where.
test_mock_finds_crashloop_pod proves the fixtures load and lookup works. test_mock_logs_contain_root_cause proves the real error is present in the logs (so the agent will have genuine evidence). test_missing_pod_is_handled asks for "does-not-exist" and checks the reply starts with "Error" — proving bad input doesn't crash anything.
You run them with python -m pytest tests/ -v from inside devops-agent.

What the output means: pytest prints one line per test with PASSED, then a summary: 3 passed in 0.04s. Green here means the foundation is solid — with no API key and no cost. That's the mock-first payoff.

Try this: Break a test on purpose: change "CrashLoopBackOff" in the assert to "Running" and re-run. pytest will show a FAILED with the expected-vs-actual values — that's the feedback loop tests give you.

✅ Test cases for Lab 8a

Test	Proves
crashloop pod is found	Fixtures load; lookup works
logs contain the root cause	The agent will have real evidence to find
missing pod handled gracefully	No crash on bad input — returns an error string

All three pass with no API key and no cost. That's the mock-first payoff.

Troubleshooting expert

⚠️ Common issues & fixes

Symptom	Cause	Fix
`ModuleNotFoundError: No module named 'mock'`	Running from the wrong directory, or missing `__init__.py`	`cd devops-agent` first; ensure `mock/__init__.py` exists
`No module named pytest`	venv not active, or pytest not installed	`source .venv/bin/activate` then `pip install pytest`
`No module named pydantic`	Dependencies not installed in this venv	`pip install -r ../requirements.txt`
`FileNotFoundError: fixtures.json`	Path resolves relative to CWD, not the module	The provided `cluster.py` uses `os.path.dirname(__file__)` — use that pattern, don't hardcode
Tests collected: 0	Test files/functions not named `test_*`	Prefix files with `test_` and functions with `test_`

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · Create the project skeleton and importable packagesBeginner

Context: Every capstone starts with a clean, importable skeleton — the difference between code that runs offline on day one and a session lost to ModuleNotFoundError.

Your task: Write the terminal commands from Lab 8a Step 1 that create the devops-agent/ project with its five sub-folders and make agent and mock importable Python packages.

Requirements:

Create the project and its five sub-folders (one mkdir, brace expansion is fine)
Install pytest so the offline tests can run
Add __init__.py to agent and mock so they import as packages
Explain that the missing __init__.py is what causes ModuleNotFoundError
Everything runs offline with no API key

💡 Hint: The two __init__.py files are the load-bearing part — without them the imports later in the lab silently fail.

Show solution

cd llm-course-starter
source .venv/bin/activate            # same venv as chapters 1-6
pip install pytest

mkdir -p devops-agent/{agent,mock,runbooks,tests,evals}
cd devops-agent
touch agent/__init__.py mock/__init__.py   # make them importable packages

The {agent,mock,runbooks,tests,evals} brace-expansion creates all five sub-folders in one mkdir. The two touch'd __init__.py files are what turn agent/ and mock/ into real packages, so later from mock import cluster works. Without them you get ModuleNotFoundError: No module named 'mock'. This step runs entirely offline with no API key.

Exercise 2 · Define the RiskClass enumIntermediate

Context: The safety gate keys every decision on an operation's risk, so the risk levels themselves must be a fixed, typo-proof, JSON-friendly set — not free-form strings.

Your task: Reproduce the RiskClass enum from agent/schemas.py exactly, and explain why it subclasses both str and Enum.

Requirements:

Four members: READ_ONLY, REVERSIBLE, SIGNIFICANT, IRREVERSIBLE
Subclass str so members serialise cleanly to JSON
Subclass Enum so the set is fixed and a misspelling can't sneak in
Explain how this prevents a typo like "irreversable" ever reaching the gate
Pure definition — no API key

💡 Hint: A plain string field would let a typo become a brand-new, ungated risk level; the enum makes the set closed.

Show solution

from enum import Enum

class RiskClass(str, Enum):
    READ_ONLY = "read_only"        # 🟢 inspect only
    REVERSIBLE = "reversible"      # 🟡 scale, restart, open PR
    SIGNIFICANT = "significant"    # 🟠 apply non-prod, merge PR
    IRREVERSIBLE = "irreversible"  # 🔴 delete, prod apply

These four values are the spine of all safety: every tool is later tagged with exactly one of them, and the safety gate (Lab 8c) does comparisons like if risk == RiskClass.IRREVERSIBLE. Inheriting from str means each member is also a real string (so RiskClass.READ_ONLY.value == "read_only" and it serializes cleanly to JSON), while Enum makes the set fixed and typo-proof — you can never accidentally write "irreversable" and slip past the gate. This is a pure definition; nothing runs and no API key is needed.

Exercise 3 · Build the Diagnosis Pydantic model with a bounded confidenceAdvanced

Context: The agent's diagnosis is structured data the rest of the system trusts, so it needs a schema that validates its own fields — especially a confidence that can't drift outside [0, 1].

Your task: Write the Diagnosis Pydantic model from agent/schemas.py, including a fix_risk: RiskClass field and a bounded confidence, then construct one valid instance.

Requirements:

Fields for likely cause, a list of evidence, and the suggested fix
A fix_risk field typed as RiskClass
confidence constrained to [0.0, 1.0] with a Pydantic Field
An optional runbook reference defaulting to None
Constructing an out-of-range confidence raises a ValidationError

💡 Hint: Let the type system do the checking — a Field(ge=0.0, le=1.0) means you never hand-write a range check.

Show solution

from typing import Optional
from pydantic import BaseModel, Field
from agent.schemas import RiskClass

class Diagnosis(BaseModel):
    likely_cause: str
    evidence: list[str]                # the facts it's citing
    suggested_fix: str
    fix_risk: RiskClass                # how risky the fix is
    confidence: float = Field(ge=0.0, le=1.0)
    runbook_ref: Optional[str] = None

d = Diagnosis(
    likely_cause="Broken DATABASE_URL secret from a recent deploy",
    evidence=["password authentication failed for user 'checkout'",
              "deploy 18 min ago changed DATABASE_URL secret ref"],
    suggested_fix="Correct the secret and roll the deployment via a PR",
    fix_risk=RiskClass.REVERSIBLE,
    confidence=0.9,
)
print(d.fix_risk, d.confidence)

Pydantic validates every field: fix_risk must be a valid RiskClass member and Field(ge=0.0, le=1.0) forces confidence into the closed interval [0, 1] — passing confidence=1.5 raises a ValidationError before the bad data can spread. runbook_ref is optional and defaults to None. This is runnable pure Python (requires only pydantic, no API key).

Exercise 4 · Implement the read-only mock cluster accessorsExpert

Context: You develop the agent against a mock cluster so tests run offline and free. The read accessors must load fixtures relative to their own module and degrade gracefully on a missing pod.

Your task: Implement mock/cluster.py's module-level state load plus get_pod, get_pods and get_logs, matching the lesson.

Requirements:

Locate fixtures.json relative to the module file, not the current directory
Load the fixture state once at import
get_pods(namespace) returns the pods in that namespace; get_pod returns one or None
get_logs returns a friendly error string for a missing pod, else the last N log lines
Pure Python against the fixture — no API key

💡 Hint: Deriving the fixtures path from os.path.dirname(__file__) is what makes the mock work no matter where pytest is launched from.

Show solution

import json, os

_STATE = json.load(open(os.path.join(os.path.dirname(__file__), "fixtures.json")))

def get_pods(namespace="staging"):
    return [p for p in _STATE["pods"] if p["namespace"] == namespace]

def get_pod(name, namespace="staging"):
    return next((p for p in get_pods(namespace) if p["name"] == name), None)

def get_logs(pod, namespace="staging", lines=20):
    p = get_pod(pod, namespace)
    if not p:
        return f"Error: pod '{pod}' not found"
    return "\n".join(p.get("logs", [])[-lines:])

Using os.path.dirname(__file__) resolves fixtures.json next to the module, so it loads no matter which directory you run pytest from — hardcoding a relative path is the usual cause of FileNotFoundError. Every function only reads: get_logs looks the pod up and returns "Error: pod '...' not found" instead of crashing on bad input. This is pure Python and needs no API key (a valid fixtures.json alongside it is required to run).

Exercise 5 · Write the three offline mock tests and read the resultsProfessional

Context: Before wiring in a model, you prove the mock behaves — the crash-loop is findable, the root cause is in the logs, and a missing pod is handled — all for zero cost.

Your task: Write the three offline tests from Lab 8a Step 5 and give the command to run them.

Requirements:

Test that the crash-looping pod exists and its status is CrashLoopBackOff
Test that the root-cause string appears in that pod's logs
Test that requesting a missing pod returns a graceful error string, not an exception
Make mock importable from the test (e.g. via sys.path)
Give the exact pytest command; all three pass with no API key

💡 Hint: These are assertions against fixed fixture data — if one fails, the mock or the fixture is wrong, and that's exactly what you want to catch before the model is involved.

Show solution

import os, sys
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from mock import cluster

def test_mock_finds_crashloop_pod():
    pod = cluster.get_pod("checkout-api-7d9f")
    assert pod is not None
    assert pod["status"] == "CrashLoopBackOff"

def test_mock_logs_contain_root_cause():
    assert "password authentication failed" in cluster.get_logs("checkout-api-7d9f")

def test_missing_pod_is_handled():
    assert cluster.get_logs("does-not-exist").startswith("Error")

cd devops-agent
python -m pytest tests/ -v
# tests/test_mock_and_tools.py::test_mock_finds_crashloop_pod PASSED
# tests/test_mock_and_tools.py::test_mock_logs_contain_root_cause PASSED
# tests/test_mock_and_tools.py::test_missing_pod_is_handled PASSED
# 3 passed in 0.04s

The sys.path.insert(...) line makes the mock package importable when pytest runs the file. pytest auto-discovers every test_* function; each assert proves one property — fixtures load and lookup works, the real error is present so the agent will have genuine evidence, and bad input yields an error string rather than a crash. All three pass with no API key and no cost — the mock-first payoff.

Exercise 6 · Enforce read/write separation and prove no state can changeIndustry scenario

Context: The design splits reads (cluster.py) from simulated writes (actions.py) so safety is enforced by construction: the read module simply has no power to mutate state.

Your task: Implement the simulated actions spanning the risk range and write a test proving cluster.py exposes no function that mutates the fixture state.

Requirements:

actions.py provides restart, open-PR and delete that return marker strings without changing state
The actions span the risk range (reversible restart/PR through irreversible delete)
Snapshot the cluster state (deep copy), call every read accessor, then assert it is unchanged
The test proves reads are non-mutating by construction
Pure Python — no API key

💡 Hint: "Safe by construction" beats "safe by review" — if the read functions literally never write, no future edit to them can leak a mutation.

Show solution

# mock/actions.py — every action is SIMULATED (returns text, changes nothing)
def rollout_restart(name, namespace="staging"):
    return f"[mock] restarted rollout {namespace}/{name}"      # reversible

def open_pr(title, body, branch):
    return f"[mock] opened PR '{title}' from branch '{branch}'"  # reversible

def delete_pod(name, namespace="staging"):
    return f"[mock] DELETED pod {namespace}/{name}"             # irreversible

# tests/test_readonly_by_construction.py
import copy
from mock import cluster

def test_reads_never_mutate_state():
    before = copy.deepcopy(cluster._STATE)
    cluster.get_pods("staging")
    cluster.get_logs("checkout-api-7d9f")
    cluster.get_pod("checkout-api-7d9f")
    assert cluster._STATE == before   # nothing a read touched changed state

The [mock] prefix is a constant reminder nothing real happened, and the actions deliberately span the risk range (rollout_restart/open_pr reversible, delete_pod irreversible) so Lab 8c's gate has real targets. The key design move is safety by construction: because reads and writes live in separate modules, the read-only diagnostician literally imports no function that can change anything — a far stronger guarantee than promising not to. The deep-copy test asserts that invariant offline, with no API key.

✓ Checkpoint — Lab 8a complete when…

Your devops-agent/ skeleton exists with the five packages.
RiskClass and Diagnosis are defined and import cleanly.
The mock cluster returns the crash-looping pod and its logs.
python -m pytest tests/ -v shows all green — with no API key.

Knowledge check check yourself

✓ Knowledge check

Why is RiskClass defined as an Enum rather than a plain string, given how the safety gate uses it?

Show answer

The gate does comparisons like if risk == RiskClass.IRREVERSIBLE. A typo-proof enum means an action can never accidentally fall through the safety check because someone wrote 'irreversable' — an invalid value is impossible, so a mistagged action can't silently bypass the gate.

✓ Knowledge check

The build labs split the mock cluster into read-only accessors (cluster.py) and simulated writes (actions.py). Why is that separation a stronger safety guarantee than just promising not to change things, and why build against a mock at all?

Show answer

Because the read-only diagnostician then literally has no function that can change anything — safety by construction, not by promise. Building against a mock is the professional way to develop something dangerous: it's deterministic, free, and incapable of breaking real infrastructure while you get tests green with no API key.

Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.

Learning objectives

What you'll build across the four build labs intermediate

Step 1 · Create the project skeleton intermediate

Step 2 · The risk schema — the spine of all safety intermediate

Step 3 · The mock cluster advanced

Step 4 · Simulated (fake) actions advanced

Step 5 · Write and run your first tests expert

Troubleshooting expert

🪜 Practice ladder beginner → industry

✓ Checkpoint — Lab 8a complete when…

Knowledge check check yourself

More in Core curriculum