Foundations of MLOps & LLMOps
Shipping a demo is easy; keeping an LLM system healthy, cheap, and trustworthy for a year is the hard part. This module is the operational discipline for that. It starts here: what MLOps is, how LLMOps differs, and the lifecycle every production LLM app moves through.
MLOps is the discipline of running machine-learning systems in production: versioning, deploying, monitoring, and improving them safely. LLMOps is the LLM-specific version — dealing with prompts, per-token cost, provider risk, and non-deterministic output. This section is the operational backbone under everything else.
The words you'll hear (in plain terms):
| Term | What it actually means |
|---|---|
| MLOps / LLMOps | the practices for running ML/LLM systems reliably in production. |
| deployment | getting your app running where users reach it, safely and repeatably. |
| monitoring | watching latency, errors, cost, and quality of a live system. |
| drift | when real-world inputs/behavior slowly diverge from what you built/tested for. |
| eval-gated rollout | only shipping a change if it passes automated quality checks. |
What you need before starting:
- Most useful after you've built something real (Ch 6, a project).
- General programming comfort; no specialized ops background assumed.
- The concepts read without infrastructure; some labs use cloud tools.
New to the topic? Read this box, then take the chapters in order — each section is tagged essential → expert so you always know the depth you're at.
Learning objectives
- Define MLOps and explain what problems it exists to solve.
- Name the ways LLMOps differs from classic MLOps — and where they overlap.
- Walk the LLM application lifecycle from idea to monitored production.
- Explain the three pillars: reproducibility, automation, and observability.
- Place prompts, evals, and models as the versioned artifacts they are.
What MLOps is (and why it exists) essential
MLOps is DevOps for machine-learning systems: the practices that take a model from a notebook to a reliable, repeatable, monitored production service. It exists because ML systems fail in ways ordinary software doesn't — they depend on data and models that drift, and "it worked on my machine" becomes "it worked on last month's data."
How LLMOps differs from classic MLOps essential
Most LLM apps don't train a model — they call one. That shifts what "ops" means. The lifecycle rhymes with MLOps, but the artifacts and risks are different.
| Dimension | Classic MLOps | LLMOps |
|---|---|---|
| Core artifact | A model you trained | A prompt + a model you call (usually via API) |
| "Training" | Fit weights on your data | Prompt engineering, RAG, sometimes fine-tuning (T4) |
| Cost driver | GPU training + serving | Per-token inference — every call costs money |
| Evaluation | Accuracy/F1 on a test set | LLM-as-judge, rubrics, groundedness — often no single number (Ch 5) |
| Latency | Usually milliseconds | Seconds; streaming matters (C2) |
| New failure modes | Data/concept drift | Hallucination, prompt injection, refusals, provider changes (T1) |
claude-opus-4-8, a model you don't control is in your critical path. Model updates, deprecations, rate limits, and pricing changes are operational events you must plan for — pin model IDs, watch deprecation notices, and keep evals that catch a behavior shift when a model version changes. This is a genuinely new axis MLOps didn't have.The LLM application lifecycle essential
Every production LLM system cycles through the same phases. The point of LLMOps is to make each one repeatable rather than heroic.
This is the whole life of a production LLM app drawn as a circle, not a straight line. You don't finish and walk away — you keep going around, and each lap makes the app a little better. Read the five circles clockwise, then notice the dashed arrow that closes the loop.
- Scope (start here): decide what you're building and what 'good enough' means — the task, the success bar, and the cheapest approach that could possibly work. Skipping this is why many projects wander.
- Prototype: build the smallest thing that shows value — usually a prompt, some retrieval (RAG), or a simple agent. It doesn't have to be pretty yet; it has to prove the idea.
- Evaluate: measure the prototype against a fixed set of test cases (a 'golden set') so you have real numbers, not a gut feeling, about whether it works.
- Deploy: put it where real users can reach it — but behind guardrails (safety checks), caching, and cost limits so a live mistake can't run wild.
- Monitor: once it's live, watch its quality, speed, cost, and safety. The solid arrows are the forward flow; each stage hands off to the next.
- The dashed red arrow ('findings feed the next iteration') is the important one: what you learn from monitoring flows back to Scope, and the loop starts again. That feedback is what turns a one-off demo into a system that keeps working.
In short: A production LLM app is never 'done' — it's a loop of scope → prototype → evaluate → deploy → monitor → and back. LLMOps is simply the tooling that makes each lap around this circle fast and safe.
| Phase | What happens | Course chapter |
|---|---|---|
| Scope | Define the task, success criteria, and the cheapest approach that could work | Ch 7 (FDE), T4 (build vs fine-tune) |
| Prototype | Prompt, RAG, or agent — the smallest thing that demonstrates value | Ch 2, 3, 4; L2–L5 |
| Evaluate | Golden set + automated evals; a regression gate | Ch 5 |
| Deploy | Ship behind guardrails, caching, cost controls | Ch 6; O3 |
| Monitor | Track quality, cost, latency, drift, safety in prod | O4 |
The three pillars intermediate
Strip LLMOps to its essence and it's three disciplines. Everything in O2–O4 serves one of them.
This is a picture of what holds a live LLM system up, drawn like a three-legged stool. The box on top is the goal; the three boxes below are the legs that support it. Read it bottom-to-top: three practices at the bottom all point up to the one thing they make possible.
- The top box, reliable production system, is the outcome you want — an app that keeps working, day after day, for real users.
- Leg 1 — Reproducibility ('version everything'): can you rebuild exactly what's running in production? Every prompt, model, and setting is saved and labelled, so nothing about your live app is a mystery.
- Leg 2 — Automation ('eval-gated CI/CD'): changes ship through automatic tests and quality checks, not by hand. A change only goes live if it passes the 'eval gate' — the automated grade on your test set.
- Leg 3 — Observability ('trace & measure prod'): you can actually see what the live system is doing — tracing each request and measuring quality, cost, and speed.
- The three arrows all point up to the top box: each leg is a separate support, and the caption's point is that a stool needs all three — miss any one leg and production becomes guesswork.
In short: Everything in the rest of this MLOps module (chapters O2–O4) is really just building one of these three legs: reproducibility, automation, or observability.
Prompts, evals & models are versioned artifacts intermediate
The single most important habit: treat the LLM-specific pieces with the same rigor as code. If it can change behavior, it gets a version and lives in source control.
| Artifact | Why version it | How |
|---|---|---|
| Prompts | A one-word change alters behavior; you must know which prompt produced which output | Prompt IDs + versions in git; log the version per request (Ch 2) |
| Model + params | Behavior shifts across model versions and effort levels | Pin the model ID; record model + params with every call (C1) |
| Eval sets | Your definition of "good" must be stable to detect regressions | Golden set in git; version it alongside the prompt (Ch 5) |
| RAG index / data | Answers change when the corpus changes | Version the index build; track embedding model & chunking config (Ch 3) |
Who does LLMOps? intermediate
Unlike classic MLOps (often a dedicated ML-platform team), LLMOps frequently lands on the application engineer — because there's no model to train, just an app to operate. That's you. The good news: your existing software-engineering instincts (version control, CI, monitoring, incident response) transfer almost directly. The new parts are the LLM-specific artifacts and failure modes above.
Common pitfalls advanced
| Pitfall | Fix |
|---|---|
| Treating a working demo as "done" | The lifecycle is a loop; deploy → monitor → iterate |
| Applying only classic MLOps thinking | Account for prompts, per-token cost, and provider risk |
| Unversioned prompts / no eval set | Version prompts, models, evals, and the RAG index |
| Ignoring the provider as an operational risk | Pin model IDs; watch deprecations; keep behavior evals |
| No observability until something breaks | Build tracing and dashboards before you need them (O4) |
| "MLOps is another team's job" | For LLM apps it's usually the app engineer's — own it |
Exercises advanced
Exercise O1.1 — Classify the artifacts
Context: You can only version what you can name, and an LLM feature has more moving parts than they first appear — prompt, model, params, eval set, retrieval config, and tools all change behaviour.
Your task: Take an LLM feature you have built in this course and list every piece that can change its behaviour, stating for each whether it is currently versioned and how you would version the ones that aren't.
Requirements:
- Enumerate all behaviour-affecting pieces, not just the prompt
- State the current versioning status of each piece honestly
- Propose a concrete versioning approach for each unversioned piece
- Cover prompt, model, params, eval set, retrieval config, and tools
💡 Hint: If a change to something can silently alter an answer, it belongs on the list — including the eval set itself.
Exercise O1.2 — MLOps vs LLMOps
Context: LLMOps is not uniformly harder than classic MLOps — it is easier in some places and harder in others. Naming which is which per lifecycle phase sharpens your operational intuition.
Your task: Pick a classic ML task and an LLM task, and for each lifecycle phase note one operational difference, then say where LLMOps is easier and where it is harder.
Requirements:
- Choose one concrete classic ML task and one concrete LLM task
- Walk every lifecycle phase, not just training
- Identify at least one place LLMOps is easier (e.g. no training pipeline)
- Identify at least one place it is harder (e.g. non-determinism, per-token cost, an uncontrolled third-party model)
💡 Hint: The absence of a training pipeline cuts both ways — it speeds prototyping but hands control of your critical-path model to a vendor.
Show a sample answer
Easier: no training pipeline, no GPU fleet to manage, faster prototyping. Harder: non-deterministic output, no single accuracy number, per-token cost at scale, and a third-party model you don't control in your critical path.
Exercise O1.3 — Map the pillars
Context: The three pillars only mean something when they turn into concrete work. Mapping each to one buildable thing gives you your own OP2–OP4 backlog.
Your task: For a system you know, write one concrete thing you would build for each pillar: a reproducibility practice, an automation gate, and an observability signal.
Requirements:
- Name a specific reproducibility practice (e.g. pinned model + versioned prompts)
- Name a specific automation gate (e.g. a CI eval gate)
- Name a specific observability signal (e.g. per-request cost logging)
- Keep each item concrete enough to start building this week
💡 Hint: These three items become your to-do list for the rest of the module — pick things you could actually ship, not aspirations.
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: Classic ML behaviour is code + data + model. LLMOps adds a fourth first-class artifact — the prompt — and you can only version what you can name.
Your task: Write a classifier that, given a change description, says which of the four artifacts (code, data, model/params, prompt) actually changed.
Requirements:
- Recognise prompt/instruction/system-message edits as the prompt artifact
- Recognise model, provider, temperature, or param edits as model/params
- Recognise corpus, index, chunk, or dataset edits as data
- Default everything else to code
- Run it over a few mixed change descriptions and print each verdict
💡 Hint: Keyword matching on the change text is enough — the lesson is that the prompt is a versioned artifact, not an anonymous string literal.
Show solution
The lesson's four artifacts as a classifier (pure stdlib, runnable):
def classify_artifact(change):
c = change.lower()
if "prompt" in c or "instruction" in c or "system message" in c:
return "prompt"
if "model" in c or "provider" in c or "temperature" in c or "params" in c:
return "model/params"
if "corpus" in c or "index" in c or "chunk" in c or "dataset" in c:
return "data"
return "code"
for ch in ["Reworded the system message", "Bumped temperature to 0.2",
"Re-chunked the FAQ corpus", "Refactored the retry loop"]:
print(f"{ch:35s} -> {classify_artifact(ch)}")Behavior in an LLM app is code + data + model + prompt. Naming which of the four moved tells you exactly what to pin in git and what to re-evaluate — the prompt is a first-class artifact, not an anonymous string literal.
Context: The LLM lifecycle is Scope → Prototype → Evaluate → Deploy → Monitor, and crucially it cycles — monitoring feeds back into scope. Treating it as a loop is what keeps an app improving after launch.
Your task: Given the five phases in the wrong order, sort them into the canonical loop and show that Monitor wraps back around to Scope.
Requirements:
- Encode the canonical phase order once
- Sort an arbitrary jumble of phases into that order
- Provide a
next_phasethat wraps Monitor back to Scope - Demonstrate both the sort and the wrap-around
- Keep it case-insensitive on the phase names
💡 Hint: Index each phase against a fixed ORDER list; the wrap-around is just modulo arithmetic on that index.
Show solution
Encode the lifecycle order and prove it is a cycle:
ORDER = ["scope", "prototype", "evaluate", "deploy", "monitor"]
def sort_phases(phases):
return sorted(phases, key=lambda p: ORDER.index(p.lower()))
def next_phase(p):
i = ORDER.index(p.lower())
return ORDER[(i + 1) % len(ORDER)] # monitor wraps back to scope
print(sort_phases(["Deploy", "Scope", "Monitor", "Evaluate", "Prototype"]))
print("after monitor comes:", next_phase("monitor")) # scope -- it loopsThe phases are not a one-way pipeline: monitoring production surfaces new topics and failures that re-scope the next iteration. Treating it as a loop is what keeps an LLM app improving instead of decaying after launch.
Context: LLMOps inherits MLOps but adds failure modes classic ML never had — hallucination, prompt injection, refusals, and a third-party provider you don't control. Knowing which bucket a symptom falls in tells you whether existing tooling already covers it.
Your task: Write a triage check that, given a symptom, says whether classic MLOps already covered it or it is genuinely LLM-specific.
Requirements:
- Maintain a set of LLM-specific concerns (hallucination, injection, refusal, provider deprecation, rate limit, non-determinism)
- Maintain a set of inherited concerns (data drift, feature skew, staleness)
- Return which bucket a symptom lands in
- Handle an unknown symptom explicitly rather than mislabelling it
- Run it over a mix of LLM-specific and classic symptoms
💡 Hint: Treat the provider as part of the system — deprecations, rate limits, and price changes are LLM-specific operational risks classic MLOps never modelled.
Show solution
Separate the inherited concerns from the genuinely new ones:
LLM_SPECIFIC = {
"hallucination", "prompt injection", "refusal",
"provider deprecation", "rate limit", "non-determinism",
}
CLASSIC = {"data drift", "feature skew", "model staleness", "label noise"}
def triage(symptom):
s = symptom.lower()
if s in LLM_SPECIFIC:
return "LLM-specific -- new in LLMOps"
if s in CLASSIC:
return "inherited from classic MLOps"
return "unknown -- classify it before you can monitor it"
for sym in ["hallucination", "data drift", "provider deprecation", "rate limit"]:
print(f"{sym:22s} -> {triage(sym)}")Classic MLOps handles drift and staleness; LLMOps must also treat the provider as part of the system (deprecations, rate limits, price changes) and defend against non-determinism, injection, and refusals. Knowing which bucket a symptom falls in tells you whether existing tooling covers it.
Context: Reproducibility, the first pillar, means every request records exactly what produced it — prompt version, model id, params, and eval-set version — so any answer can be regenerated months later.
Your task: Build a version_stamp that emits one immutable, deterministic log record capturing everything needed to reproduce an answer.
Requirements:
- Hash the prompt text so the exact prompt is pinned
- Record a pinned model id — never "latest"
- Record the params and the eval-set version
- Serialize with a stable key order so identical inputs give an identical record
- Prove reproducibility: the same inputs produce a byte-identical stamp
💡 Hint: A sorted-key JSON dump of a small dict is enough — determinism comes from the stable ordering and from hashing the prompt rather than storing it raw.
Show solution
The reproducibility stamp — deterministic and hashable (pure stdlib):
import hashlib, json
def version_stamp(prompt_text, model_id, params, evalset_version):
prompt_hash = hashlib.sha256(prompt_text.encode()).hexdigest()[:12]
rec = {
"prompt_sha": prompt_hash,
"model_id": model_id, # pinned, never "latest"
"params": params,
"evalset_version": evalset_version,
}
# stable key order so identical inputs -> identical record
return json.dumps(rec, sort_keys=True)
s1 = version_stamp("Classify the ticket.", "claude-opus-4-8",
{"temperature": 0, "max_tokens": 256}, "goldenset-v7")
s2 = version_stamp("Classify the ticket.", "claude-opus-4-8",
{"temperature": 0, "max_tokens": 256}, "goldenset-v7")
print(s1)
print("reproducible:", s1 == s2) # True -- same inputs, same stampPinning the model id (never "latest") and hashing the prompt means a logged answer can be regenerated months later. Without the stamp you cannot tell whether a regression came from a prompt edit, a param tweak, or a silent provider model update.
Context: The three pillars are Reproducibility, Automation, and Observability. A system can be strong on one and blind on another, so scoring each against concrete deliverables turns "we should do LLMOps" into a ranked to-do list.
Your task: Build a pillar scorecard that scores a system's config against concrete deliverables per pillar and names the weakest pillar to invest in first.
Requirements:
- Score reproducibility on pinned model + versioned prompts + versioned eval set
- Score automation on a CI eval gate + automated deploy
- Score observability on per-request cost, latency, and quality logging
- Return the per-pillar scores, not just a single number
- Identify and report the weakest pillar as the priority
💡 Hint: Sum booleans per pillar, then take the pillar with the minimum score — the output should name the gap, not just grade it.
Show solution
A pillar scorecard that names the gap, not just a number:
def pillar_audit(sys):
scores = {
"reproducibility": sum([sys.get("pins_model", False),
sys.get("versions_prompts", False),
sys.get("versions_evalset", False)]),
"automation": sum([sys.get("ci_eval_gate", False),
sys.get("automated_deploy", False)]),
"observability": sum([sys.get("logs_cost", False),
sys.get("logs_latency", False),
sys.get("logs_quality", False)]),
}
weakest = min(scores, key=scores.get)
return scores, f"weakest pillar: {weakest} -- invest here first"
sysA = dict(pins_model=True, versions_prompts=True, versions_evalset=True,
ci_eval_gate=False, automated_deploy=False,
logs_cost=True, logs_latency=True, logs_quality=False)
print(pillar_audit(sysA))A system can be strong on reproducibility yet blind operationally. Scoring each pillar against concrete deliverables turns "we should do LLMOps" into a ranked to-do list — here automation (no CI gate, no automated deploy) is the hole to fill first.
Context: A team ships a demo with prompts as inline literals, no eval set, and no logged model version. As the platform lead you don't rewrite it — you define the bar it must clear before it can carry production traffic.
Your task: Produce the minimum LLMOps intake checklist a prototype must pass, and gate on it so "not yet operable" becomes an objective verdict.
Requirements:
- List the smallest set of requirements (prompts in git, pinned model id, eval set, version-stamped logs, cost/latency logging, a kill switch)
- Check a candidate system against every requirement
- Return a clear READY / BLOCK verdict
- List exactly which requirements are missing when blocking
- Demonstrate the gate blocking an unready demo
💡 Hint: The intake gate is the three pillars turned into a pass/fail checklist — the lead's job is to enforce the bar, not to fix the demo.
Show solution
The intake gate — the smallest bar a prototype must clear to be operable:
INTAKE = [
("prompts_in_version_control", "prompts are files in git, not string literals"),
("model_id_pinned", "pinned model id recorded per call (no 'latest')"),
("has_eval_set", "a golden set exists to catch regressions"),
("logs_version_stamp", "each request logs prompt+model+params version"),
("logs_cost_and_latency", "cost and latency captured per request"),
("kill_switch", "feature flag to disable the LLM path instantly"),
]
def intake_gate(candidate):
missing = [why for key, why in INTAKE if not candidate.get(key)]
verdict = "READY for traffic" if not missing else "BLOCK -- fix first"
return verdict, missing
demo = dict(prompts_in_version_control=False, model_id_pinned=False,
has_eval_set=False, logs_version_stamp=False,
logs_cost_and_latency=True, kill_switch=False)
verdict, missing = intake_gate(demo)
print(verdict)
for m in missing:
print(" -", m)A lead does not rewrite the demo — they define the bar and enforce it. The intake gate converts the three pillars into a checklist a prototype must pass before it can hurt real users, and makes "not yet operable" an objective verdict instead of a hunch.
✓ Checkpoint — you can move on when you can…
- Define MLOps and the code+data+model problem it solves.
- List the key ways LLMOps differs from classic MLOps.
- Walk the five lifecycle phases and name the course chapter behind each.
- State the three pillars and what each guarantees.
- Name the four artifacts that must be versioned — and why.
Knowledge check check yourself
The lesson says classic MLOps behavior is defined by code + data + model. What third artifact does LLMOps add, and why does the 'provider as part of your system' create a genuinely new operational axis?
Show answer
Name the three pillars of LLMOps and what each one guarantees.