MLOps, Deployment & Experiment Tracking

MLOps for LLM apps — what's different motivation

Here's the moment this chapter exists to prevent. Your agent works beautifully in the notebook. You paste it into production, it helps real users for two weeks, and then someone quietly edits a prompt to "make answers friendlier." Overnight the escalation rate doubles and nobody notices for three days because there was no eval gate, no version history, and no dashboard. When you finally trace it, you can't even reproduce the old behaviour — the good prompt is gone. That whole disaster is an MLOps failure, and every section here is a piece of the machinery that would have caught it on day one.

Think of it like the difference between cooking dinner and running a restaurant kitchen. Anyone can make one good meal when they have all afternoon and only themselves to feed. A restaurant has to serve the same dish, at quality, hundreds of times a night, when the head chef is off sick and a supplier swapped the tomatoes — so it needs written recipes (versioned prompts), a pass at the line before food leaves (eval gates), a way to yank a bad dish (rollback), and a manager watching the tickets (monitoring). MLOps is the kitchen discipline that turns "I made a good agent once" into "this agent is reliably good for everyone, every day."

The common mistake — and it catches strong engineers — is assuming that because you're not training a model, the ops rigor doesn't apply to you. It absolutely does; it just points at different objects. Classic MLOps centers on training pipelines and model weights. For LLM apps you usually don't train — you call a model API. So your versioned, tested, monitored artifacts shift to prompts, tool definitions, retrieval configs, and eval suites. The rigor is the same (reproducibility, testing, monitoring, rollback); the objects are different. A prompt string is production logic every bit as load-bearing as a function — treat it casually and it will page you at 2am just like any other untested code path. This page is the operational wrapper around everything in Ch 5–6 and the capstone: it's what lets a working prototype become a system you'd stake a customer relationship on.

1 · The LLM-app lifecycle intermediate

Every serious AI system is a loop, not a line — and the teams that forget that ship once and then slowly rot. The tempting mental model is a straight arrow: build the thing, launch it, done. But an LLM app that isn't fed by what happens after launch is like a shop that never reads its own complaints box — it keeps making the same mistakes forever, confidently.

Picture instead the way a good hospital handles a near-miss. Something goes wrong, it gets logged, the case is reviewed, and a new checklist item is added so that exact failure can't recur. The lifecycle below is the same idea applied to software: you develop, evaluate, package, deploy, monitor — and then the interesting part, the monitor stage feeds back into evaluate. Every real production failure and every thumbs-down becomes a new test case, so your safety net grows tighter with every incident instead of staying frozen at launch.

The table maps each stage to the concrete artifact you produce and where in this course you learned to build it — read it as a checklist you'll return to on every project. The mistake to avoid is treating the stages as one-and-done departments handed off over the wall. A prompt tweak (Develop) with no new eval (Evaluate) is how quality silently drifts; a deploy (Deploy) with no monitoring (Monitor) is how you find out about outages from angry users instead of your own alerts. The stages are cheap when they're a habit and brutally expensive when they're skipped, because a gap anywhere breaks the feedback loop that keeps the whole system honest.

2 · Experiment tracking advanced

Ask yourself an honest question: three prompt tweaks ago, what was your pass rate? If the answer is "I think it was better?" you've just met the problem this section solves. Tuning an LLM app means running dozens of variations — this model or that one, temperature 0.2 or 0.7, retrieve 5 chunks or 8 — and human memory is spectacularly bad at holding a scoreboard of that many runs. You end up going in circles, re-testing things you already tried, and shipping a version because it "felt good" rather than because it won.

Experiment tracking is nothing more exotic than a lab notebook for your prompts. A scientist doesn't trust their memory of which reagent concentration gave the best yield — they write down every condition and every result so the comparison is undeniable and repeatable. You do the same: for each run you record the inputs (prompt version, model, temperature, retrieval settings) alongside the outputs (eval scores, cost, latency). Now "which was better?" stops being an argument and becomes a sort on a column.

Why the discipline matters so much here specifically: an LLM app's quality is multi-dimensional, so a change almost never makes everything better at once. Prompt v4 lifts your pass rate but also costs 30% more and adds 400ms of latency — is that a win? You genuinely cannot answer without the numbers side by side, because the tradeoff depends on your product. The classic mistake is tracking only the metric you're chasing and discovering, weeks later in the billing console, that your "better" prompt quietly tripled the bill. Log cost and latency from the very first run, even while you're only trying to improve quality — they're free to capture now and impossible to reconstruct later. Without tracking, you can't say which run was better or reproduce a win; with it, every experiment compounds into knowledge instead of evaporating.

pythonimport json, hashlib

def run_id(config):
    return hashlib.sha256(json.dumps(config, sort_keys=True).encode()).hexdigest()[:8]

def log_run(store, config, metrics):
    store.append({"id": run_id(config), "config": config, "metrics": metrics})

runs = []
log_run(runs, {"model": "claude-opus-4-8", "prompt": "v3", "k": 5},
        {"pass_rate": 0.92, "cost_per_1k": 1.80, "p50_latency": 1.2})
log_run(runs, {"model": "claude-opus-4-8", "prompt": "v4", "k": 8},
        {"pass_rate": 0.95, "cost_per_1k": 2.40, "p50_latency": 1.6})
best = max(runs, key=lambda r: r["metrics"]["pass_rate"])
print(best["id"], best["config"]["prompt"])   # the winning prompt version

# Real tools: MLflow (mlflow.log_params/log_metrics) or Weights & Biases (wandb.log)
# give you a UI, run comparison, and artifact storage over exactly this data.

3 · Prompt & model registry — version prompts like code advanced

Picture the 2am page. Quality has cratered, users are complaining, and you trace it to a prompt edit someone made this afternoon. The fix is obvious: put the old prompt back. So — what was the old prompt? If it lived as a string pasted into code and edited in place, the previous wording is gone, and now you're reconstructing production logic from memory under pressure. That is the nightmare a registry exists to abolish.

The core realization is that a prompt is code — it's the instruction that decides what your system does — so it deserves everything code gets: a history of every version, the ability to review a change before it ships, and a one-move undo. Think of it like the "published version" of a document versus the live draft. You keep every published edition immutably, and you keep a separate pointer that says "this is the one customers see right now." Fixing a bad release isn't re-typing the old text; it's just moving the pointer back to a known-good edition. That move is instant and can't introduce a new typo, because you're re-selecting text that already worked, not recreating it.

That separation — immutable versions plus a movable "active" tag — is what makes rollback a seconds-long, deterministic operation instead of an anxious rewrite. A deploy pins an exact version (diagnose@v2), so the behaviour in production is reproducible and a rollback lands somewhere you've actually tested. The mistake that bites teams is letting production point at "latest" or at a mutable string: now every prompt edit is an instant, unreviewed, unversioned production change, and you've lost the very history you need the day something breaks. The same discipline extends to the model ID and retrieval config — pin everything that affects the output, or your "reproducible" system quietly shifts under you when a vendor updates an alias.

pythonfrom dataclasses import dataclass

@dataclass(frozen=True)
class PromptVersion:
    name: str
    version: int
    template: str

class PromptRegistry:
    def __init__(self):
        self._store = {}          # (name, version) -> PromptVersion
        self._active = {}         # name -> active version
    def publish(self, name, template):
        version = 1 + max([v for (n, v) in self._store if n == name], default=0)
        self._store[(name, version)] = PromptVersion(name, version, template)
        self._active[name] = version
        return version
    def get(self, name, version=None):
        return self._store[(name, version or self._active[name])]
    def rollback(self, name, version):
        self._active[name] = version    # instant revert, no redeploy

reg = PromptRegistry()
reg.publish("diagnose", "You are an SRE... {ctx}")      # v1
reg.publish("diagnose", "You are a senior SRE... {ctx}")  # v2 (active)
reg.rollback("diagnose", 1)                          # back to v1 in one call

4 · Packaging & serving advanced

"It works on my machine" is where a thousand production incidents are born. Your agent runs perfectly in your notebook, with your Python version, your installed libraries, and your API key sitting in a shell variable. None of that travels. The moment someone else — a teammate, a CI runner, a Kubernetes pod — needs to run it, all those invisible assumptions surface as errors. Packaging and serving is the craft of turning "runs where I built it" into "runs anywhere, the same way, on demand."

There are two separate ideas here, and it helps to name them. Packaging is bundling your agent into a self-describing, installable unit — code plus its exact dependencies (P6's pyproject.toml) — so any machine can install and import it without guesswork. Think of it as vacuum-sealing a meal with the recipe taped to the lid: whoever opens it gets the same thing you made. Serving is the next step — wrapping that unit in a web API so other programs can call it over HTTP instead of importing your Python. That's the difference between a script you run and a service that's always listening; a front-end, a mobile app, or another microservice can all hit the same endpoint.

FastAPI is the standard tool for the serving half in Python, and it's standard for good reasons that all trace back to earlier chapters: it's async (A3), so one process can juggle many slow LLM calls at once instead of blocking on each; it's typed with Pydantic (A6), so malformed requests are rejected at the door with a clear error before your code ever runs; and it's streaming-friendly, so you can flush tokens to the user as they generate. The beginner mistake is stopping at a bare /ask route and calling it production. A real service also needs a /health endpoint so orchestrators know whether to route traffic to it or restart it — without one, Kubernetes has no way to tell a wedged pod from a healthy one, and users get served by a corpse. Serving is as much about the boring operational surface as it is about the answer.

python# pip install fastapi uvicorn
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class Query(BaseModel):        # request validated by Pydantic (A6)
    question: str
    prompt_version: int | None = None

@app.post("/ask")
async def ask(q: Query):        # async endpoint (A3)
    # pin the prompt version, call the agent, return structured output
    return {"answer": f"(stub) answering: {q.question}", "prompt": q.prompt_version}

@app.get("/health")
async def health():         # liveness probe for k8s (Ch 8)
    return {"status": "ok"}
# run:  uvicorn app:app --host 0.0.0.0 --port 8000
# containerize with a Dockerfile (Ch 8) and deploy to k8s/ECS.

5 · CI/CD with eval gates expert expert

If you remember one practice from this entire page, make it this one. The single thing that most separates a hobby LLM project from a production one is whether a quality regression can reach users without anyone deciding it should. On a mature system, it can't — because every change automatically runs the eval suite, and a deploy is blocked the moment quality drops. That gate is the difference between "we move fast and occasionally torch the customer experience" and "we move fast and the customer experience only ever gets better."

Here's the analogy that makes it click. An eval gate is the airport metal detector for your prompts. Everyone walks through the same scanner, every time, no exceptions — the CEO and the intern alike. It doesn't judge intent or trust anyone's assurance that "this change is fine"; it just measures, and if something trips the threshold, that passenger doesn't board. Wire that scanner into CI (the automation that runs on every push) and you've made high quality the default state of your main branch, enforced by a machine that never gets tired, rushed, or talked out of it.

The mechanism is beautifully simple and rests on one humble fact: a program that exits with a non-zero code is telling the operating system "I failed," and CI systems treat any failed step as a reason to halt the pipeline. So an eval script that computes your pass rate and calls sys.exit(1) on a regression will physically stop the deploy — no human in the loop required. Two design choices make it usable in practice. First, a small tolerance: LLM evals wobble run-to-run, so you allow a couple of points of noise or you'll block every release for nothing. Second — and this is the mistake that undermines otherwise-careful teams — gate on more than pass rate. A prompt that's 2% more accurate but 3x more expensive, or that quietly starts taking irreversible actions without approval, is not an improvement. Quality, cost, and safety are all release criteria; a gate that watches only one of them is a scanner that checks for metal but waves through the explosives.

pythonimport sys, json

BASELINE = {"pass_rate": 0.90}       # committed alongside the code

def gate(results, tolerance=0.02):
    pr = results["pass_rate"]
    if pr < BASELINE["pass_rate"] - tolerance:
        print(f"❌ regression: {pr:.3f} < baseline {BASELINE['pass_rate']:.3f}")
        sys.exit(1)                    # non-zero exit FAILS the CI job -> blocks deploy
    print(f"✅ evals pass: {pr:.3f}")

# gate(run_evals())   # in CI, after building the app

.github/workflows/ci.yml (sketch)jobs:
  test-and-eval:
    steps:
      - run: pytest -q                 # unit tests (P6)
      - run: python evals.py --json    # run golden-set evals (Ch 5)
      - run: python gate.py results.json   # BLOCK deploy on regression

6 · Deployment strategies advanced

The riskiest word in operations is "just." "Just push the new version to everyone" feels efficient right up until the new version has a subtle flaw — and now 100% of your users are hitting it simultaneously, your on-call is scrambling, and the blast radius is the entire customer base. Deployment strategies are the accumulated wisdom of everyone who's been burned by that flip and decided there had to be a safer way to introduce change.

The unifying idea across all of them is controlling exposure: instead of a binary old-then-new switch, you find a way to let a new version prove itself while limiting who can be hurt if it's wrong. A canary sends a small slice of traffic (5%) to the new version first — named after the caged birds miners carried underground, which showed distress before the humans did. Blue-green keeps two complete environments and flips traffic between them, so rollback is as fast as flipping back. Shadow is the cleverest: it mirrors real traffic to the new version but throws its answers away, letting you measure it against live load with zero user risk. And a feature flag toggles the new behaviour per user or cohort, giving you a gradual rollout with an instant kill switch. Read the table below as a menu, not a ranking — each buys safety in a different currency.

Which one you reach for is a judgment about how much you can validate before real users see the change, and that's where LLM apps have a genuine superpower worth internalizing. Because you have evals, shadow mode is far more powerful for you than for a typical web service: you can run a candidate prompt or a new model against a day of real traffic, score both offline with your golden set, and compare quality, cost, and latency before a single user is affected. That's how you de-risk something as scary as a model-version upgrade. The mistake here is defaulting to one strategy for everything — big-bang cutovers for high-risk model swaps (reckless), or elaborate shadow pipelines for a one-word prompt fix (wasteful). Match the ceremony to the risk.

7 · Monitoring & drift expert expert

The worst way to learn your AI system has degraded is from a user telling you. By then it's been broken for who-knows-how-long, the damage is done, and you're reacting instead of preventing. Shipping is not the finish line — it's the moment the system starts living in the wild, exposed to real inputs you never tested and a model provider who can change behaviour under you without warning. Monitoring is how you keep your eyes open after launch instead of shipping and hoping.

Monitoring an LLM app means watching two layers at once. The first is the ordinary service dashboard any backend needs — latency percentiles, error rates, throughput — the vital signs on the hospital monitor. But AI systems have a second, sneakier layer: quality and cost signals that a normal service simply doesn't have. Your app can be returning HTTP 200s in 200 milliseconds and still be quietly getting worse — refusing more often, escalating more, giving answers a judge would score lower. The service looks healthy while the product rots. That's why you track quality proxies (escalation rate, thumbs-down, low-confidence rate) and cost (token usage, cache-hit rate) right alongside the classic metrics.

The signal that trips people up most is drift — the slow, silent kind of failure with no error message attached. Nothing crashes; the world just shifts out from under your assumptions. Users start asking about a new product you have no context for, so retrieval quality quietly falls. Or the model provider ships an update and your carefully-tuned prompt now behaves subtly differently. Drift is like a photograph left in sunlight: no single day looks different from the last, but compare against last month and the colours have visibly faded. You catch it by tracking a rolling baseline and alerting when a metric leaves its expected band. The mistake is monitoring only outputs. Drift usually shows up in your inputs first — the distribution of questions shifting — so watching input patterns gives you an early warning before quality visibly cracks. The practical way to feed all of this is boring and powerful: emit one structured (JSON) log line per request, and let dashboards aggregate thousands of them into charts and threshold alerts.

pythonimport json, logging
log = logging.getLogger("agent.metrics")

def record(**fields):
    # one structured JSON line per request -> ship to Grafana/Kibana/CloudWatch
    log.info(json.dumps({"kind": "llm_call", **fields}))

record(model="claude-opus-4-8", prompt_version=2,
       input_tokens=1200, output_tokens=300,
       latency_ms=1180, cache_hit=True, escalated=False)
# Dashboards aggregate these; alerts fire on thresholds (p95 latency, cost/hr,
# escalation rate). This is the monitoring skill of the DevOps agent (Ch 8).

8 · Cost control & governance advanced

Every token you send and receive is metered, and the meter never sleeps. That's the fact that makes LLM apps unlike almost anything else you've deployed. A traditional service costs roughly the same whether it handles a clever request or a dumb one; an LLM app can bill you ten times more for one answer than another, and a single sloppy prompt shipped to production can turn a comfortable budget into a runaway invoice while you sleep. Cost isn't an afterthought you optimize later — for LLM apps it's a first-class production concern that belongs in the same conversation as latency and correctness.

The good news is that cost is controllable with a handful of well-understood levers, and it pays to know their rough order of impact. Prompt caching is usually the biggest single win (Ch 6): reused context — a long system prompt, a fixed set of instructions — gets charged at a fraction of the price on repeat calls. Model routing is the next lever and the one people most often skip: not every step deserves your most expensive model. Classification, routing, and easy lookups can go to a small cheap model (or the cache), while you reserve the frontier model (claude-opus-4-8) for the genuinely hard reasoning. Then token budgets per request cap the worst case, and batching amortizes overhead across many calls. The instinct to reach for is "cheapest tool that clears the quality bar," not "best tool for everything."

But there's a trap in cheapness, and it's why routing decisions must themselves be gated by evals: a cheaper path is only a saving if quality holds. Route a hard question to a weak model and you don't save money — you burn it on an answer that fails, gets escalated, and costs a human's time on top. That's why the right metric is cost-per-outcome, not cost-per-token; a "cheaper" prompt that halves your success rate is the most expensive change you could make. The other half of this section is governance — the auditability layer that answers "who deployed which prompt and model version, and why," and that puts a gate in front of every call for spend caps, model allowlists, and an incident kill switch. On a shared platform, owning cost and policy centrally is what keeps one team's experiment from draining everyone's budget or shipping an unvetted model into production.

Exercises expert

🎯 Interview practice interview

The interview questions this topic gets asked — worked, with code. For the full pattern catalog see A9 · Big Tech AI-engineering patterns.

python# SHIP behind an eval gate (golden set + judge; block deploy on regression)
# MONITOR latency / cost / quality / drift dashboards + alerts
# CONTROL cost: prompt caching, model routing, token budgets
# VERSION model + prompt pinned; canary / shadow deploy; instant rollback
# LOOP  prod failures -> new golden cases -> prevent regression

pythonimport sys
BASELINE = 0.90
def gate(pass_rate, tol=0.02):
    if pass_rate < BASELINE - tol:
        print(f"regression {pass_rate:.3f} < {BASELINE}")
        sys.exit(1)          # fail CI -> block deploy
    print("evals pass")

Checkpoint — you've completed the AI Engineering track when you can… expert

def llm_equivalent(classic_stage):
    return {
        "collect training data": "curate an eval set (question, answer, context)",
        "train a model":         "engineer prompts + choose/version a model + RAG context",
        "validate on test set":  "run eval metrics + LLM-as-judge on the eval set",
        "deploy weights":        "deploy prompts/config behind an API (weights are the vendor's)",
        "monitor accuracy":      "monitor quality, cost, latency, and prompt/data drift",
    }[classic_stage]

for s in ("train a model", "validate on test set", "monitor accuracy"):
    print(s, "->", llm_equivalent(s))

class PromptRegistry:
    def __init__(self):
        self.versions = {}                  # name -> {version: text}
        self.tags = {}                       # name -> {tag: version}
    def register(self, name, text):
        v = len(self.versions.setdefault(name, {})) + 1
        self.versions[name][v] = text
        self.tags.setdefault(name, {})["latest"] = v
        return v
    def promote(self, name, version, tag="prod"):
        self.tags[name][tag] = version       # move the tag (deploy / rollback)
    def resolve(self, name, tag="prod"):
        return self.versions[name][self.tags[name][tag]]

r = PromptRegistry()
r.register("triage", "Classify the ticket into billing/tech/other.")
v2 = r.register("triage", "Classify into billing/tech/other. Return one word.")
r.promote("triage", v2, "prod")
print(r.resolve("triage", "prod"))           # v2 text
r.promote("triage", 1, "prod")               # rollback to v1
print(r.resolve("triage", "prod"))           # v1 text

THRESH = {"faithfulness": 0.85, "answer_relevance": 0.75, "context_recall": 0.80}

def eval_gate(scores):
    failed = [f"{k} {scores[k]:.2f}<{THRESH[k]}" for k in THRESH if scores[k] < THRESH[k]]
    return ("PASS", []) if not failed else ("FAIL", failed)

baseline = {"faithfulness": 0.88, "answer_relevance": 0.79, "context_recall": 0.83}
candidate= {"faithfulness": 0.80, "answer_relevance": 0.81, "context_recall": 0.84}  # faith regressed
print(eval_gate(baseline))    # ('PASS', [])
print(eval_gate(candidate))   # ('FAIL', ['faithfulness 0.80<0.85'])

def rollout(risk, can_compare_offline, need_live_traffic_to_judge):
    if need_live_traffic_to_judge and not can_compare_offline:
        return "SHADOW — mirror real traffic to the new version, compare, serve old"
    if risk == "high":
        return "CANARY — 1%%->10%%->100%%, watch metrics, auto-rollback on regression"
    if risk == "low":
        return "BLUE-GREEN — cut over fully, keep old env warm for instant rollback"
    return "CANARY — default cautious rollout"

print(rollout("high", True,  False))   # CANARY
print(rollout("low",  True,  False))   # BLUE-GREEN
print(rollout("med",  False, True))    # SHADOW

from collections import deque

class DriftMonitor:
    def __init__(self, window=20, tol=0.05):
        self.window, self.tol = window, tol
        self.hist = deque(maxlen=window)
    def observe(self, score):
        if len(self.hist) < self.window:
            self.hist.append(score); return "warming up"
        baseline = sum(self.hist) / len(self.hist)
        self.hist.append(score)
        if abs(score - baseline) > self.tol:
            return f"DRIFT: {score:.2f} vs baseline {baseline:.2f}"
        return "ok"

m = DriftMonitor(window=5, tol=0.05)
for s in [0.90,0.91,0.89,0.90,0.90, 0.90, 0.72]:   # last score drops sharply
    print(m.observe(s))

class Governor:
    def __init__(self, monthly_cap, allowed_models):
        self.cap = monthly_cap; self.allowed = set(allowed_models)
        self.spent = 0.0; self.enabled = True
    def authorize(self, model, est_cost):
        if not self.enabled:            return "DENY — kill switch engaged"
        if model not in self.allowed:   return f"DENY — {model} not on allowlist"
        if self.spent + est_cost > self.cap:
            return f"DENY — would exceed ${self.cap} monthly cap"
        self.spent += est_cost
        return f"ALLOW — spent ${self.spent:.2f}/{self.cap}"

g = Governor(monthly_cap=100.0, allowed_models={"claude-haiku-4-5","claude-opus-4-8"})
print(g.authorize("claude-opus-4-8", 5.0))     # ALLOW
print(g.authorize("some-other-model", 1.0))    # DENY — not on allowlist
g.enabled = False
print(g.authorize("claude-opus-4-8", 1.0))     # DENY — kill switch

Knowledge check check yourself