MLOps & LLMOps · Chapter O3

Deployment, Containerization & Scaling of LLM Systems

Your app calls an API — so why is deployment hard? Because LLM apps are I/O-bound, slow-per-request, and metered by the token. Those three facts change how you containerize, scale, and stay up. This chapter is the deployment playbook built around them.

⏱️ ~55 min🚢 Hands-on🎯 Intermediate

Learning objectives

Explain why LLM apps scale differently from CPU-bound services.
Containerize an LLM app cleanly (config, secrets, health checks).
Choose a deployment target and a scaling strategy that fits token-metered, I/O-bound work.
Apply safe rollout patterns — blue/green, canary — with an eval gate.
Design for graceful degradation when the model is slow, rate-limited, or down.

Reuses your DevOps foundationsDocker, Kubernetes, and CI you may know already (and the repo's own k8s/terraform work). This chapter is what's different about deploying an LLM app on top of that. Async/streaming mechanics are in A3; caching and guardrails in Chapter 6.

Why LLM apps scale differently intermediate

A normal web service is CPU-bound and answers in milliseconds. An LLM app spends most of each request waiting for a slow, token-metered upstream. Three facts flow from that and drive every deployment decision.

Fact	Consequence for deployment
I/O-bound — mostly waiting on the model	One process can handle many concurrent requests with async; you don't need a core per request (A3)
Slow per request — seconds, not ms	Long-lived connections; streaming; generous timeouts; graceful shutdown must drain in-flight calls
Metered per token	Scaling up multiplies cost linearly; caching & model-tiering matter more than raw replicas

The counter-intuitive partBecause the work is I/O-bound, throwing CPUs at it barely helps — your app is idle, waiting on the API. The levers that actually move the needle are concurrency (async), caching (don't pay for repeat calls), and model choice (route easy work to a cheaper tier). Adding replicas is the last lever, not the first.

Lab O3.1 · Containerize an LLM app intermediate

A container makes your app reproducible — the reproducibility pillar (O1) at the deployment layer. The LLM-specific parts are config, secrets, and a health check that means something.

Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.

Lab O3.1

DockerfileFROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
# API keys come from the environment at RUN time — NEVER baked into the image
EXPOSE 8000
# stream-friendly server; graceful shutdown drains in-flight requests
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

▶ How this works

A Dockerfile is a recipe for building an image — a self-contained box holding your app plus everything it needs to run. Ship that box anywhere and it behaves the same. Read it top to bottom: each line is one build step, and the steps run in order.

FROM python:3.12-slim picks the starting point: a small (slim) Linux image that already has Python 3.12. Everything you add layers on top of this. WORKDIR /app then says "do the rest of the work inside the /app folder."
COPY requirements.txt . copies just the dependency list in, then RUN pip install ... -r requirements.txt installs those libraries while building the image. Copying the list first (before the rest of the code) lets Docker reuse the cached install step when only your code changes — a big speed-up.
COPY . . copies the rest of your app into the image. The # comment lines aren't run — they're notes to humans. The key one says the API key comes from the environment at run time and is NEVER baked into the image (see the red box below for why that matters).
EXPOSE 8000 documents that the app listens on port 8000. CMD ["uvicorn", "app:app", ...] is the command that runs when the container starts — it launches uvicorn, an async web server that fits the slow, streaming, I/O-bound work of an LLM app.

What the output means: This file doesn't print anything on its own — it's a build recipe. Running docker build against it produces an image; docker run then starts a container from that image, launching uvicorn on port 8000.

Try this: Build and run it, passing the key at run time: docker build -t myapp . then docker run -e ANTHROPIC_API_KEY=... -p 8000:8000 myapp. The -e injects the secret now, so it never lives inside the image — exactly what the comment insists on.

Container concern	The LLM-app rule
Secrets	Inject the API key at runtime (env / secrets manager). Never bake it into the image or commit it (C2, T1)
Config	Model ID, timeouts, and limits from env — so you change them without rebuilding
Health check	A cheap `/health` that checks the app is up — don't call the paid model on every probe
Graceful shutdown	On SIGTERM, stop accepting new requests but let in-flight (slow!) calls finish
Image size	Slim base; no model weights inside (you're calling an API) → small, fast-pulling images

The classic leakBaking ANTHROPIC_API_KEY into a Docker image (via ENV or a copied .env) ships your key to anyone who can pull the image — and image layers are forever. Keys are injected at run time, full stop. This is the deployment-layer version of the T1 secrets rule.

Scaling strategy intermediate

Scale the levers in the order that respects the three facts — cheapest and most effective first.

Try the levers in order. Cache repeat calls (free hits). Max out concurrency per replica with async (I/O-bound work loves it). Route easy requests to a cheaper model tier. Only then add replicas — each one multiplies your token bill, so it's the last resort, not the reflex.

🗺️ How to read this diagram

This picture ranks the four ways to handle more traffic, cheapest and most effective at the top, most expensive at the bottom. The whole point: try them in this order, top to bottom.

Bar 1 · Cache — reuse answers you've already paid for. Repeat calls become free hits, so this is the first and biggest win.
Bar 2 · Concurrency (async) — because LLM work is mostly waiting on the API, one server can juggle many requests at once. You squeeze far more out of each replica before adding hardware.
Bar 3 · Route to a cheaper model — send easy requests to a smaller, faster, cheaper model tier, saving both cost and time.
Bar 4 · Add replicas (autoscale) — last. Notice it's the widest bar and marked "use last": each extra copy of your app multiplies your token bill, so it's the last resort. The labels on the right ("cheapest, biggest win" up top; "most cost, use last" at the bottom) restate the ranking.

In short: Work down the list, not up. Cache and concurrency are nearly free and go first; spinning up more replicas is the reflex to resist, because it's the only lever that scales your cost linearly with traffic.

Lever	Why it's high on the list	Course link
Caching	Cached calls cost ~nothing; prompt caching cuts input cost ~90%	Ch 6, O2 gateway
Concurrency (async)	One replica serves many waiting requests; fits I/O-bound work	A3
Model tiering	Haiku/Sonnet for easy work slashes cost & latency	C1
Horizontal replicas	Real capacity, but linear cost — autoscale on queue depth/latency	k8s HPA

Autoscale on the right signalCPU-based autoscaling misfires for LLM apps — your pods are near-idle while blocked on the API, so CPU stays low even when overloaded. Scale on request concurrency, queue depth, or p95 latency instead. And remember the real ceiling is often the provider rate limit, not your pod count — more replicas hitting a 429 wall just fail faster.

Lab O3.2 · Safe rollouts with an eval gate advanced

The scariest deploy in an LLM app isn't a code change — it's a prompt or model change, because behavior shifts invisibly. Safe-rollout patterns plus the eval gate (Ch 5, O1) catch regressions before all users see them.

Two gates before everyone sees it. First the offline eval gate — the change must not regress the golden set. Then a canary: route a small traffic slice to the new version and watch quality, cost, and latency before going 100% (or rolling back).

🗺️ How to read this diagram

This is the safe path a prompt or model change travels before every user sees it. Read it left to right, following the arrows — the change only advances if each stage passes.

Left box · eval gate — before anything ships, the change is scored against a fixed golden set of test cases (offline, no real users). If quality drops, it's blocked here.
Arrow → means "only if it passed." Middle box · canary 5% — the change goes live for a small slice of real traffic (5%) while you watch quality, cost, and latency. A problem now hits only a few users, not everyone.
Arrow → then right box · full rollout (or roll back) — if the canary looks healthy, ramp to 100%; if not, roll back and no one else is affected.
The line across the top says it plainly: a change ships only if it clears the offline evals AND the live canary — two independent gates.

In short: Treat a prompt tweak like a code deploy. "It's just a prompt, ship it" skips both gates — and a bad prompt can silently wreck quality just like a code bug. Two gates catch it cheaply.

Pattern	What it does	Good for
Eval gate	Block deploy if the golden set regresses	Every prompt/model change (Ch 5)
Blue/green	Full new version alongside old; flip traffic instantly	Fast rollback; infra changes
Canary	Small % to new version; ramp if healthy	Catching live regressions cheaply
Shadow	New version runs on real traffic, output discarded & compared	Validating a change with zero user risk

Treat prompt changes like code deploysA prompt edit can regress quality as badly as a code bug — and silently. Version it (O1), run it through the eval gate, canary it. "It's just a prompt tweak, ship it" is how teams wake up to a quality incident with no idea what changed.

Designing for graceful degradation advanced

Your app depends on a slow external service that will rate-limit, slow down, or have an outage. Plan the failure modes so a bad model day degrades service instead of taking you down.

Failure	Graceful response
429 rate limit	Retry with backoff (SDK does this); queue; shed load or fall back to a smaller model (C2, A6)
High latency	Stream partial output; set a timeout; show progress rather than a spinner that never ends
Provider outage	Circuit breaker; fail over to a secondary model/provider via the gateway (O2, A6)
Refusal / bad output	Detect `stop_reason`; return a safe fallback message, don't crash (C2)
Cost spike	Budget caps in the gateway; alert & throttle before the bill runs away (O4)

The circuit breaker earns its keep hereThe circuit breaker and retry/backoff you built in A6 exist precisely for this: when the model is failing, stop hammering it, fail fast, and recover automatically. In an LLM app that external dependency is your critical path — resiliency isn't optional polish, it's core.

Common pitfalls expert

Pitfall	Fix
Baking the API key into the image	Inject at runtime; never commit or `ENV` it
Autoscaling on CPU	Scale on concurrency / queue depth / p95 latency
Adding replicas as the first scaling move	Cache → concurrency → model tier → replicas last
Health check that calls the paid model	Cheap liveness check; probe the dependency separately & sparingly
Shipping a prompt change with no gate	Version it; run the eval gate; canary it
No plan for provider 429/outage	Backoff, circuit breaker, fallback, budget caps
Killing pods without draining	Graceful shutdown lets slow in-flight calls finish

Exercises expert

Exercise O3.1 — Containerize & run

Context: Shipping an LLM app means containerising it without baking the API key into the image — a secret in a layer is a secret leaked to anyone who pulls the image.

Your task: Wrap a small FastAPI-plus-Claude app in a Dockerfile, pass the API key at run time, and prove the key is not baked into the image.

Requirements:

Build the app into an image with a Dockerfile
Pass ANTHROPIC_API_KEY via -e at run time, not at build
Confirm the running container answers requests
Inspect docker history and confirm the key is absent from every layer

💡 Hint: Anything on a build-time ARG or ENV lands in the image history — inject secrets only at docker run.

Exercise O3.2 — Scaling plan

Context: A sudden 10× traffic spike is where the lever ordering pays off — and where you discover the provider rate limit is the real ceiling that no number of replicas can raise.

Your task: Write the order in which you would apply the four scaling levers under 10× traffic, what you would measure after each, and when the provider rate limit becomes the binding constraint.

Requirements:

Order the levers from cheapest to most expensive
State the metric you would watch after each lever (hit rate, p95, cost/req, 429s)
Identify the point where 429s mean you have hit the provider limit
Say what you do then (higher tier, fallback provider, queue/shed load)

💡 Hint: Replicas past the provider limit just fail faster — the fix at that point is capacity or a second provider, not more pods.

Show a sample answer

Cache (measure hit rate) → raise per-pod concurrency (measure p95, error rate) → route easy calls to Haiku (measure cost/req) → add replicas (watch for 429s). When 429s appear, you've hit the provider limit — request a higher tier, add a fallback provider, or queue/shed load. Replicas past that point just fail faster.

Exercise O3.3 — Degradation drill

Context: A graceful-degradation spec is what stands between a bad minute and an incident. None of your failure modes should surface as a stack trace or an infinite spinner.

Your task: For your own app, write exactly what the user sees in each failure mode: rate-limited, model slow, provider down, and refusal.

Requirements:

Cover all four failure modes
Specify a concrete user-visible response for each
Ensure none is a stack trace or an endless spinner
Frame the result as a reusable degradation spec

💡 Hint: Write it from the user's seat — every mode should map to a message or a fallback, not a raw error.

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · Rank the four scaling levers by costBeginner

Context: LLM apps scale differently from classic services. The levers, cheapest-win first, are cache, concurrency (async), model tier, and replicas — and replicas are the last resort because they multiply cost and still hit the provider rate limit.

Your task: Given a jumbled list of the four scaling levers, return them in the order you should reach for them.

Requirements:

Encode the canonical lever order (cache, concurrency, model tier, replicas)
Sort an arbitrary input list into that order
Keep it pure-stdlib and deterministic
Demonstrate on a shuffled list

💡 Hint: Caching reuses answers you already paid for, so it sits first; replicas are last because they scale cost linearly and don't lift the provider ceiling.

Show solution

Encode the ranked levers (pure stdlib):

ORDER = ["cache", "concurrency", "model tier", "replicas"]

def rank_levers(levers):
    return sorted(levers, key=lambda x: ORDER.index(x))

print(rank_levers(["replicas", "cache", "model tier", "concurrency"]))
# ['cache', 'concurrency', 'model tier', 'replicas']

Reach for the cheapest lever first: caching reuses answers you already paid for, async concurrency exploits the I/O-bound wait, model-tiering routes easy work to a cheap model, and only then do you add replicas — which multiply cost linearly and hit the provider rate limit anyway.

Exercise 2 · Cache hit-rate turns into dollars savedIntermediate

Context: Caching is the top lever because it reuses paid answers. Because LLM cost is metered per token, a good hit rate cuts spend before you touch tiers or replicas.

Your task: Given request volume, per-request cost, and a cache hit rate, compute the monthly spend with and without the cache and the percent saved.

Requirements:

Model cache hits as costing ~0
Compute monthly spend at a given hit rate
Compare against a zero-hit-rate baseline
Report the percent saved
Show that a 60% hit rate cuts spend ~60%

💡 Hint: Only the miss fraction incurs model cost — the saving tracks the hit rate almost directly.

Show solution

The cache savings model (pure arithmetic, runnable):

def monthly_cost(requests, cost_per_req, hit_rate):
    paid = requests * (1 - hit_rate)      # cache hits cost ~0
    return round(paid * cost_per_req, 2)

reqs, cost = 1_000_000, 0.01
no_cache = monthly_cost(reqs, cost, hit_rate=0.0)
cached   = monthly_cost(reqs, cost, hit_rate=0.6)
saved = round(100 * (no_cache - cached) / no_cache)
print(f"no cache: ${no_cache:,.2f}   60% hit: ${cached:,.2f}   saved: {saved}%")

A 60% hit rate is a 60% cut in model spend before you touch replicas or tiers. Because LLM cost is metered per token, reusing a paid answer is the single biggest scaling win — which is why it sits at the top of the lever list.

Exercise 3 · Size replicas with Little's Law (not CPU)Advanced

Context: Autoscaling on CPU is wrong for I/O-bound LLM apps — pods sit idle while blocked on the API. The right signal is request concurrency, and Little's Law converts arrival rate and latency into the concurrency you must serve.

Your task: Use Little's Law to size in-flight concurrency from arrival rate × latency, then compute the replicas needed given a per-pod concurrency.

Requirements:

Compute in-flight requests as arrival rate × average latency
Divide by per-pod concurrency and round up to whole replicas
Use seconds consistently for latency
Demonstrate with a concrete load (e.g. 50 req/s, 2s latency, 20 slots/pod)
Note that the provider rate limit is often the real ceiling above this number

💡 Hint: Little's Law is L = λ × W — arrival rate times time in system gives the concurrency; replicas follow by division.

Show solution

Little's Law sizes the fleet from load, not CPU (pure stdlib):

import math

def replicas_needed(req_per_sec, avg_latency_s, concurrency_per_pod):
    # Little's Law: in-flight requests L = arrival rate * time in system
    in_flight = req_per_sec * avg_latency_s
    return math.ceil(in_flight / concurrency_per_pod)

# 50 req/s, each request holds a connection ~2s (seconds, not ms), 20 async slots/pod
print("in-flight:", 50 * 2, "-> replicas:", replicas_needed(50, 2.0, 20))
# in-flight: 100 -> replicas: 5

The right autoscaling signal is request concurrency / queue depth / p95 latency — never CPU, which stays low while a pod waits on the model. Little's Law converts arrival rate and latency into the concurrency you must serve, and that divides into replicas. The provider rate limit is often the real ceiling above this number.

Exercise 4 · The p99 tail, not the mean, sets your SLOExpert

Context: Users feel the tail, not the mean. Chaining sequential model calls compounds each hop's tail, so a multi-hop agent's p99 balloons far past its average — and the SLO you promise is a p99.

Your task: Compute p50/p95/p99 over a latency sample and show why a 4-hop sequential path inflates the tail versus a 1-hop path.

Requirements:

Implement a percentile function over a latency list
Model one hop as mostly fast with an occasional slow tail
Model four sequential hops as the sum of four independent draws
Report p50/p95/p99 for both the 1-hop and 4-hop paths
Make it reproducible with a fixed seed

💡 Hint: Summing independent per-hop latencies is what compounds the tails — design against p99 by cutting hops, parallelising, or streaming partial output.

Show solution

Percentiles and tail compounding (pure stdlib, runnable):

def pctl(latencies, p):
    s = sorted(latencies)
    k = max(0, math.ceil(p/100 * len(s)) - 1)
    return s[k]

import math, random
random.seed(7)
# one model hop: mostly ~700ms, occasional 2200ms tail
one_hop = [random.choice([700]*9 + [2200]) for _ in range(1000)]
# four sequential hops: tails compound (sum of four independent draws)
four_hop = [sum(random.choice([700]*9 + [2200]) for _ in range(4)) for _ in range(1000)]

for name, data in [("1-hop", one_hop), ("4-hop", four_hop)]:
    print(f"{name}: p50={pctl(data,50)}  p95={pctl(data,95)}  p99={pctl(data,99)}")

Averages hide the pain: a 4-hop agent compounds each hop's tail, so the p99 balloons far past 4x the mean. Design against the tail — plan once and parallelize, stream partial output, or set per-hop timeouts — because the SLO you promise is a p99, not an average.

Exercise 5 · Gate a rollout: eval gate then canaryProfessional

Context: Safe rollout is two independent gates before 100%: an eval gate on the golden set that stops known regressions, then a canary on a slice of live traffic that catches the failures the golden set could not.

Your task: Implement the two-gate rollout: block on a golden-set regression, then require the canary metrics to hold within tolerance before ramping.

Requirements:

Block at the eval gate when the golden pass rate is below the gate
Compare canary quality and error rate against a live baseline
Allow a small tolerance on the canary metrics
Return RAMP / ROLLBACK / BLOCK as distinct outcomes
Demonstrate all three outcomes

💡 Hint: The two gates are independent — the eval gate runs before any user sees the change, the canary runs on real traffic; both must pass to ramp.

Show solution

The two-gate rollout as a decision function (pure stdlib):

def rollout_decision(golden_pass, golden_gate, canary, baseline, tol=0.02):
    if golden_pass < golden_gate:
        return "BLOCK at eval gate -- golden set regressed, do not deploy"
    # canary: quality must not drop, error rate must not rise beyond tolerance
    q_ok = canary["quality"] >= baseline["quality"] - tol
    e_ok = canary["error_rate"] <= baseline["error_rate"] + tol
    if q_ok and e_ok:
        return "RAMP -- canary healthy, proceed to full rollout"
    return "ROLLBACK -- canary regressed on live traffic"

base = {"quality": 0.90, "error_rate": 0.01}
good = {"quality": 0.90, "error_rate": 0.012}
bad  = {"quality": 0.83, "error_rate": 0.05}
print(rollout_decision(0.94, 0.90, good, base))   # RAMP
print(rollout_decision(0.94, 0.90, bad,  base))   # ROLLBACK
print(rollout_decision(0.85, 0.90, good, base))   # BLOCK at eval gate

The eval gate stops known regressions before any user sees them; the canary catches the live-only failures the golden set could not. Two cheap, independent gates in front of every deploy is what makes shipping LLM changes routine instead of scary.

Exercise 6 · Author the graceful-degradation playbook as the on-call leadIndustry scenario

Context: Production will hit rate limits, latency spikes, provider outages, refusals, and cost spikes. As on-call lead your job is to write, before the incident, the map from each failure to a user-visible degradation — never a hard error.

Your task: Author the graceful-degradation playbook: map each failure mode to the degraded response the system should show, routed through the gateway.

Requirements:

Cover rate limits, high latency, provider outage, refusals, and cost spikes
Map each to a concrete degradation (backoff/queue, stream partial, circuit breaker/failover, safe fallback, budget cap)
Route responses through the gateway (fallback, circuit breaker, budget cap)
Handle an unknown failure explicitly
Print the full playbook

💡 Hint: Every foreseeable failure should degrade to something a user can live with — pair the map with a pre-built kill switch so an outage is a downgrade, not a 500.

Show solution

The failure-to-response map from the lesson (pure logic):

PLAYBOOK = {
    "rate_limit_429":  "retry w/ backoff; queue; shed load; fall back to cheaper model",
    "high_latency":    "stream partial output; set timeout; show progress",
    "provider_outage": "circuit breaker; fail over to secondary provider via gateway",
    "refusal_bad_out": "detect stop_reason; return safe fallback message",
    "cost_spike":      "budget cap in gateway; alert & throttle",
}

def degrade(failure):
    return PLAYBOOK.get(failure, "unknown failure -- add it to the playbook after")

for f in PLAYBOOK:
    print(f"{f:16s}: {degrade(f)}")

Every foreseeable failure should degrade to something a user can live with, routed through the gateway (fallback, circuit breaker, budget cap). The lead's job is to write this map before the incident and wire a kill switch, so an outage is a graceful downgrade, not a 500.

✓ Checkpoint — you can move on when you can…

Explain why LLM apps are I/O-bound and what that changes.
Containerize an app with runtime secrets and a real health check.
Order the scaling levers and justify replicas being last.
Roll out a prompt/model change through an eval gate and canary.
Specify graceful behavior for rate limits, latency, and outages.

🏗️ Toward the capstoneThe AI DevOps Engineer ships as a container with its key injected at runtime, autoscales on concurrency (not CPU), and every change to its prompt or safety policy clears the eval gate (Chapter 8d) and a canary before full rollout. When Claude is rate-limited, it degrades — queues work, falls back — rather than dropping incidents. That's this chapter, applied to the capstone. See the evals + go-real build →

Knowledge check check yourself

✓ Knowledge check

LLM apps are I/O-bound, so what does the lesson say is the correct order of scaling levers, and why are replicas last?

Show answer

Cache repeat calls, then max out concurrency with async, then route easy work to a cheaper model tier, and only then add replicas. Replicas are last because each one multiplies the per-token bill linearly, whereas caching/concurrency are nearly free.

✓ Knowledge check

Why is CPU-based autoscaling the wrong signal for an LLM app, and what should you scale on instead?

Show answer

The pods are near-idle while blocked waiting on the model API, so CPU stays low even when the app is overloaded — CPU autoscaling misfires. Scale on request concurrency, queue depth, or p95 latency; and note the real ceiling is often the provider rate limit, not pod count.

Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.

Learning objectives

Why LLM apps scale differently intermediate

Lab O3.1 · Containerize an LLM app intermediate

Scaling strategy intermediate

Lab O3.2 · Safe rollouts with an eval gate advanced

Designing for graceful degradation advanced

Common pitfalls expert

Exercises expert

Exercise O3.1 — Containerize & run

Exercise O3.2 — Scaling plan

Exercise O3.3 — Degradation drill

🪜 Practice ladder beginner → industry

✓ Checkpoint — you can move on when you can…

Knowledge check check yourself

More in MLOps & LLMOps