MLOps, Deployment & Experiment Tracking

These two comment lines aren't code you run — they're a signpost. They tell you the little tracker above is the same idea as the real tools you'll use on the job.

1. MLflow and Weights & Biases (W&B) are popular experiment-tracking products. You call mlflow.log_params/log_metrics or wandb.log instead of your own log_run.
2. In return they give you a web dashboard, side-by-side run comparison, and storage for artifacts (files) — all on top of exactly the config-plus-metrics data you just modelled by hand.

Try this: Skim the MLflow quickstart and spot log_params / log_metrics — you'll recognise them as the fancy version of log_run.

When you tune an AI app you try lots of combinations — different prompts, models, and settings. Experiment tracking is just keeping a labelled record of what you tried and how well it scored, so later you can say which version was best and reproduce it. This is a tiny hand-rolled version of what tools like MLflow do.

1. run_id(config) turns the settings you used (model, prompt, k) into a short 8-character fingerprint. Same settings always give the same id — a stable label for one experiment.
2. log_run(store, config, metrics) appends one record to a list: its id, the config (what you tried) and the metrics (how it did — pass rate, cost, latency).
3. The two log_run(...) calls record two experiments: prompt v3 vs v4. Each stores real-world numbers you'd care about — how often it passed, dollars per 1000 calls, and typical latency.
4. max(runs, key=lambda r: r["metrics"]["pass_rate"]) scans every recorded run and picks the one with the highest pass rate — the winner.

What the output means: It prints the winning run's id and prompt version. Here v4 wins because its pass_rate (0.95) beats v3's (0.92) — even though v4 costs more, which is the kind of trade-off tracking makes visible.

Try this: Lower v4's pass_rate to 0.90 and re-run — now v3 should win. That flip is the whole point: the data, not your memory, tells you which prompt to ship.

A prompt is really production logic, so it deserves the same care as code: named, versioned, and reversible. A registry is a small store that keeps every version of every prompt and remembers which one is currently 'active'. This lets you swap or roll back a prompt in seconds without shipping new code.

1. @dataclass(frozen=True) class PromptVersion defines one immutable record: a prompt's name, its version number, and its template text. frozen means once made it can't be changed — a safe, fixed snapshot.
2. publish(name, template) computes the next version number (one higher than the highest existing version for that name), stores the new version, and makes it the active one. First publish becomes v1, next v2, and so on.
3. get(name, version=None) hands back a specific version, or — if you don't ask for one — the current active version. That's what your app calls at request time.
4. rollback(name, version) just points 'active' back at an older version. No code change, no redeploy — the fix is instant.

What the output means: Nothing prints. The three calls at the bottom publish v1, publish v2 (now active), then roll back to v1 — so after this runs, get("diagnose") would return the v1 template again.

Try this: After the rollback, add print(reg.get("diagnose").version) — it prints 1, proving the active pointer moved back. Then call reg.rollback("diagnose", 2) to jump forward again.

Once your agent works as a script, you serve it behind a web API so other programs (or a front-end) can call it over HTTP. FastAPI is the standard Python tool for this: you declare the shape of the request, then write a function per URL. This stub shows the skeleton of a real service.

1. app = FastAPI() creates the web application object you attach routes to.
2. class Query(BaseModel) declares the expected request body: a required question string and an optional prompt_version. Pydantic (A6) auto-validates incoming JSON against this — bad input is rejected before your code runs.
3. @app.post("/ask") registers the ask function to handle POST requests at /ask. async def lets the server handle many callers at once (A3). Here it returns a stub answer; in reality it would pin the prompt version and call the agent.
4. @app.get("/health") is a tiny health check: orchestration systems like Kubernetes ping it to confirm the service is alive and restart it if not.

What the output means: No output on its own — this defines a service. The bottom comment shows how you'd start it: uvicorn app:app --host 0.0.0.0 --port 8000, after which POSTing JSON to /ask returns the answer object.

Try this: Run the uvicorn command, then open http://localhost:8000/docs — FastAPI auto-generates an interactive page where you can call /ask and /health from the browser.

An eval gate is the signature MLOps move for LLM apps: before any change ships, you run your quality tests (the golden set from Ch 5) and refuse to deploy if quality dropped. This function is that gate — it decides pass or fail and, on fail, stops the pipeline.

1. BASELINE = {"pass_rate": 0.90} is the quality bar, committed next to your code so everyone agrees on 'good enough'.
2. gate(results, tolerance=0.02) reads the new run's pass_rate and compares it to the baseline. The tolerance allows tiny, harmless wobble (here 2%) so normal noise doesn't block every deploy.
3. If the score falls below baseline − tolerance, it prints a regression message and calls sys.exit(1). A non-zero exit code is how a program tells the CI system 'I failed' — which stops the deploy.
4. Otherwise it prints a pass message (and exits 0 implicitly), letting the pipeline continue.

What the output means: Nothing runs yet — the real call is the commented gate(run_evals()). With a pass_rate of 0.85 it would print the ❌ regression line and fail; with 0.91 it prints ✅ and passes.

Try this: Call gate({"pass_rate": 0.85}) and watch it exit with an error; then try 0.89 — it still passes because 0.89 is within the 0.02 tolerance of 0.90.

This is a CI pipeline config (a GitHub Actions workflow), not Python. It describes the ordered steps a server runs automatically on every push. The point is where the gate sits: tests and evals run first, and the deploy is blocked if any step fails.

1. jobs: and steps: are YAML keys defining a job made of steps that run top-to-bottom. If any step exits non-zero, the whole job fails and later steps (and the deploy) don't happen.
2. run: pytest -q runs your unit tests (P6) — ordinary code correctness.
3. run: python evals.py --json runs the golden-set quality evals (Ch 5) and writes the scores to a file.
4. run: python gate.py results.json feeds those scores to the gate from the previous block. If quality regressed, gate.py exits non-zero and the deploy is blocked automatically.

Try this: Reorder the steps in your head: what happens if gate.py ran before evals.py? It would have no results.json to read — order matters, which is why produce-then-check is the pattern.

After you ship, you must watch the system. The cheapest, most powerful way is to emit one structured log line (JSON) per request, capturing the numbers you care about. Because it's JSON, dashboards and alerting tools can parse and aggregate it automatically.

1. log = logging.getLogger("agent.metrics") gets a named logger — a channel you can route to a file or a log shipper.
2. record(**fields) accepts any keyword fields and writes them as one JSON object. **fields means 'collect all the named arguments into a dictionary', so you can log whatever matters without changing the function.
3. log.info(json.dumps({"kind": "llm_call", **fields})) tags every line with kind: llm_call and merges your fields in — producing one tidy, machine-readable record.
4. The example call logs a real request's shape: model, prompt version, input/output tokens, latency, whether the prompt cache hit, and whether it escalated to a human.

What the output means: It writes a single JSON line to the log, e.g. {"kind": "llm_call", "model": "claude-opus-4-8", ..., "latency_ms": 1180, ...}. Tools like Grafana/Kibana/CloudWatch then aggregate thousands of these into charts and fire alerts on thresholds.

Try this: Add cost_usd=0.004 to the record(...) call. Because record takes **fields, it just appears in the JSON — no code change needed. That flexibility is why structured logging scales.

This block is the model answer to a senior interview question: how do you productionize an LLM feature? Each comment line is one pillar of the loop this whole page teaches — say the loop, not a laundry list of tools.

1. SHIP behind an eval gate — quality is checked before every deploy, and regressions block the release (section 5).
2. MONITOR latency, cost, quality, and drift with dashboards and alerts (section 7).
3. CONTROL cost with prompt caching, model routing, and token budgets (section 8).
4. VERSION the model and prompt (pinned), and roll out safely with canary/shadow deploys and instant rollback (sections 3 and 6).
5. LOOP — every production failure becomes a new golden-set case, so the same bug can never regress again. This closed loop is the heart of LLM MLOps.

Try this: In an interview, speak these five words in order — Ship, Monitor, Control, Version, Loop — and give one concrete example of each. Structure beats a feature list every time.

This is the interview-ready, whiteboard-sized version of the eval gate: the smallest code that shows you understand how to stop a bad deploy. It's the same idea as section 5, trimmed to the essentials.

1. BASELINE = 0.90 is the minimum acceptable pass rate.
2. gate(pass_rate, tol=0.02) compares the new score to the baseline, allowing a small tolerance so ordinary noise doesn't block releases.
3. sys.exit(1) is the load-bearing line: a non-zero exit code tells the CI system the step failed, which blocks the deploy. Everything else is just the message.

Try this: Be ready to explain why sys.exit(1) matters: CI treats any non-zero exit as failure, so this one line is what turns a failing eval into a blocked deploy.