Config, health & autoscaling
A Deployment that runs isn't a Deployment that's production-ready. This lesson is the hardening layer: liveness / readiness / startup probes so the cluster knows when a Pod is healthy and when to send it traffic; resource requests & limits so the scheduler can pack nodes and one Pod can't starve the rest; rolling updates & rollbacks for zero-downtime deploys; the HorizontalPodAutoscaler to add replicas under load; and a PodDisruptionBudget so maintenance never takes you below quorum.
Learning objectives
- Configure liveness, readiness, and startup probes and say what each one gates.
- Set resource requests (for scheduling) and limits (for capping), and explain QoS.
- Perform a zero-downtime rolling update and a rollback.
- Add a HorizontalPodAutoscaler that scales on CPU (and understand custom metrics).
- Protect availability during maintenance with a PodDisruptionBudget.
1 · The three probes — and what each gates
Kubernetes can't read your app's mind; you tell it health via probes. The three do different jobs and confusing them causes real outages:
| Probe | Question it answers | What failing it does |
|---|---|---|
| readiness | "Can I serve traffic right now?" | Removed from Service endpoints (no restart) |
| liveness | "Am I alive, or wedged?" | Container is restarted |
| startup | "Have I finished booting?" | Holds off the other probes until boot is done |
probes.yaml readinessProbe: # gates traffic: pull from LB until ready
httpGet: { path: /healthz/ready, port: 8000 }
periodSeconds: 5
livenessProbe: # gates restart: kill+restart if wedged
httpGet: { path: /healthz/live, port: 8000 }
periodSeconds: 10
failureThreshold: 3
startupProbe: # slow starters: don't let liveness kill boot
httpGet: { path: /healthz/live, port: 8000 }
failureThreshold: 30
periodSeconds: 5 # allows up to ~150s to start
2 · Resource requests and limits
Two numbers per resource, and they mean different things. The request is what the scheduler reserves to place the Pod (and what other Pods can't use). The limit is the hard cap while running. Getting these right is most of good cluster economics.
resources.yaml resources:
requests: # scheduler reserves this; used for bin-packing
cpu: "250m" # 0.25 of a core
memory: "256Mi"
limits: # hard cap at runtime
cpu: "1" # throttled above 1 core
memory: "512Mi" # OOM-killed above 512Mi
| Behavior | CPU (compressible) | Memory (incompressible) |
|---|---|---|
| Over the limit | Throttled (slowed, not killed) | OOM-killed — the container dies |
| Set request only | Can burst above request if node has room | Same — no cap |
| No request/limit | "BestEffort" — first to be evicted | First to be evicted under pressure |
3 · Rolling updates and rollbacks
Changing the image on a Deployment triggers a rolling update by default: new Pods come up and pass readiness before old ones are removed, so traffic never drops. If it goes wrong, rollback is one command because the old ReplicaSet is still there.
rollout.shkubectl set image deploy/web app=ghcr.io/example/web:1.3.0
kubectl rollout status deploy/web # blocks until healthy or fails
kubectl rollout history deploy/web # revisions you can return to
kubectl rollout undo deploy/web # instant rollback to previous revision
kubectl rollout undo deploy/web --to-revision=2
deployment "web" successfully rolled out4 · HorizontalPodAutoscaler
The HPA adjusts a Deployment's replica count to hit a target metric — classically CPU. It needs a metrics source (metrics-server for CPU/memory, or an adapter for custom metrics). It scales up quickly under load and down conservatively to avoid flapping.
hpa.yamlapiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: web
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: web
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 60 # % of the CPU *request*
autoscaling/v2 API and available metric types evolve — verify the current API version and options against the Kubernetes docs. For bursty, spiky traffic (like LLM inference, K7), CPU may be the wrong signal — consider queue depth or a custom metric.5 · PodDisruptionBudget
There are two kinds of disruption: involuntary (a node crashes — you can't prevent it) and voluntary (you drain a node for an upgrade). A PodDisruptionBudget constrains the voluntary kind so routine maintenance can't take you below a safe number of Pods.
pdb.yamlapiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: web
spec:
minAvailable: 2 # or: maxUnavailable: 1
selector:
matchLabels:
app: web
kubectl drain, a cluster autoscaler scale-down, a node upgrade) wait until it can respect the budget — it slows or blocks the disruption rather than losing availability. It does not protect against involuntary loss (a hard node crash); that's what replicas across nodes and readiness are for. Verify current PDB semantics against the Kubernetes docs.✓ Checkpoint — you can move on when you can…
- Say what each probe gates: readiness → traffic, liveness → restart, startup → boot window.
- Explain the difference between a resource request and a limit, and what happens over each.
- Perform a rolling update and roll it back with
kubectl rollout. - Write an HPA on CPU and explain why it needs a CPU request set.
- Explain what a PodDisruptionBudget protects and what it doesn't.
An app takes ~90 seconds to warm a model cache on boot. With a plain liveness probe (10s period, 3 failures) it enters CrashLoopBackOff and never starts. What's wrong and how do you fix it?
Show answer
failureThreshold × periodSeconds window (e.g. 30 × 5s = ~150s). While the startup probe is still passing/pending, liveness and readiness are suspended, so the slow warm-up is allowed to finish; only after startup succeeds does liveness begin guarding against wedging.You set an HPA to keep CPU at 60% but it never adds replicas even under obvious load. The Deployment has no resources.requests. Why is the HPA inert?
Show answer
resources.requests.cpu on the container; then "60%" becomes 60% of that request and the HPA can react. (Also confirm metrics-server is installed, or the HPA has no metrics at all.)🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: Probes are the first thing separating a demo Deployment from a real one.
Your task: Add a readiness and a liveness probe to a web container and state, in one line each, what failing each probe causes.
Requirements:
- Readiness on a cheap /ready endpoint
- Liveness on a cheap /live endpoint (no dependency calls)
- State the effect of each failing
💡 Hint: Liveness must be cheap and in-process; readiness may check whether it can serve.
Show solution
readinessProbe:
httpGet: { path: /healthz/ready, port: 8000 }
periodSeconds: 5
livenessProbe:
httpGet: { path: /healthz/live, port: 8000 }
periodSeconds: 10
failureThreshold: 3Effects: a failing readiness probe removes the Pod from the Service's endpoints so it stops receiving traffic (no restart) — used to shed a temporarily-busy Pod. A failing liveness probe restarts the container — used to recover a wedged process. Keep liveness cheap and dependency-free so a slow database can't trigger needless restarts.
Context: Right-sizing resources is most of cluster cost control.
Your task: Given an app that idles ~200m CPU / 180Mi and peaks 900m / 480Mi, choose requests and limits and justify each number.
Requirements:
- Set the request from typical usage, the memory limit from peak + headroom
- Explain why CPU over-limit and memory over-limit behave differently
- Note what QoS class request==limit would give
💡 Hint: Memory over-limit kills; CPU over-limit only throttles — size memory more carefully.
Show solution
resources:
requests: { cpu: "250m", memory: "256Mi" }
limits: { cpu: "1", memory: "640Mi" }Reasoning: the request (250m / 256Mi) sits at or just above typical usage so the scheduler packs nodes efficiently and the Pod is guaranteed its baseline. The CPU limit (1 core) lets it burst for spikes; exceeding it only throttles the container, so a generous cap is safe. The memory limit (640Mi) is peak (480Mi) plus headroom, because exceeding a memory limit OOM-kills the container — you never want a normal peak to kill it. Setting request == limit on both would give the Guaranteed QoS class (least likely to be evicted), at the cost of no bursting.
Context: A rollout that stalls safely on a bad version is the whole point of readiness-gated deploys.
Your task: Roll out a new image, then explain what happens if the new version fails its readiness probe, and how you'd recover.
Requirements:
- Give the rollout and status commands
- Explain why a failing readiness probe stalls rather than breaks the rollout
- Give the rollback command
💡 Hint: Old Pods are only removed once new ones are Ready — so a never-Ready new version can't take the app down.
Show solution
kubectl set image deploy/web app=ghcr.io/example/web:1.3.0
kubectl rollout status deploy/web # watch it progress or stallIf v1.3.0's readiness probe never passes: the rolling update creates a new Pod but won't mark it Ready, so it won't remove an old (working) Pod to make room beyond maxUnavailable. The rollout stalls — old Pods keep serving traffic, and rollout status reports it isn't progressing. The bad version never takes the app down on its own.
Recover: kubectl rollout undo deploy/web scales the previous ReplicaSet back up and the broken one down — an instant return to v1.2.0 because the old ReplicaSet was never deleted.
Context: Choosing the right autoscaling signal is what separates a working HPA from a flapping one.
Your task: Add a CPU HPA to a web Deployment, then argue whether CPU is the right metric for a latency-sensitive, bursty API and what you'd use instead.
Requirements:
- Write the HPA (2–10 replicas, 60% CPU)
- Note the CPU-request prerequisite
- Argue for/against CPU vs a custom metric for bursty inference-style load
💡 Hint: For request/response latency, in-flight requests or queue depth track user pain better than CPU.
Show solution
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: web }
spec:
scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: web }
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource: { name: cpu, target: { type: Utilization, averageUtilization: 60 } }Prerequisite: a CPU request must be set, since utilization is measured against it, plus a running metrics-server.
Is CPU the right signal? For a CPU-bound service, yes. But for a latency-sensitive, bursty API — or LLM inference (K7) — CPU can lag the actual pain: requests queue up before CPU saturates, so users see latency while the HPA sits idle. Better signals are in-flight requests, queue depth, or p95 latency, exposed as custom/external metrics via an adapter. Verify the current custom-metrics API against the docs before relying on it.
Context: Voluntary disruptions (node upgrades, autoscaler scale-down) will take your Pods down unless a budget constrains them.
Your task: Add a PodDisruptionBudget for a 3-replica service and explain exactly what it does — and doesn't do — during a rolling node upgrade.
Requirements:
- Write a PDB keeping at least 2 of 3 available
- Explain how it interacts with
kubectl drain - State clearly what it does NOT protect against
💡 Hint: A PDB gates voluntary evictions only; involuntary loss still needs replicas across nodes.
Show solution
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: web }
spec:
minAvailable: 2
selector: { matchLabels: { app: web } }During a node upgrade: the upgrade drains nodes one at a time. When a drain would evict a web Pod, the PDB checks whether at least 2 would remain available; if not, the eviction blocks and waits until a replacement Pod is Ready elsewhere. The upgrade proceeds only as fast as availability allows — so you never drop below 2 serving Pods.
What it does NOT do: a PDB is powerless against involuntary disruption — if a node crashes, both its Pods are gone regardless of the budget. Real resilience needs replicas spread across nodes/AZs (anti-affinity / topology spread) so no single node failure can breach your target. The PDB only governs the disruptions Kubernetes initiates.
Context: Representative scenario: a service passed review functionally but the SRE team blocks its prod promotion until it meets a reliability checklist.
Your task: Produce the full container spec and supporting objects that take a bare Deployment to the production bar: probes, resources, rollout strategy, HPA, and PDB, each with a one-line justification.
Requirements:
- All three probe types where appropriate, correctly scoped
- Requests and limits set from a profile, memory limit above peak
- Rolling strategy with maxUnavailable: 0 for a critical service
- HPA on an appropriate metric and a PDB
- One line on what each control buys you
💡 Hint: Tie each control to a specific failure it prevents — that's how SRE reviews think.
Show solution
Container spec (probes + resources):
readinessProbe: { httpGet: { path: /healthz/ready, port: 8000 }, periodSeconds: 5 }
livenessProbe: { httpGet: { path: /healthz/live, port: 8000 }, periodSeconds: 10, failureThreshold: 3 }
startupProbe: { httpGet: { path: /healthz/live, port: 8000 }, failureThreshold: 30, periodSeconds: 5 }
resources:
requests: { cpu: "250m", memory: "256Mi" }
limits: { cpu: "1", memory: "640Mi" }Deployment strategy (critical service — never dip below capacity):
strategy:
type: RollingUpdate
rollingUpdate: { maxUnavailable: 0, maxSurge: 1 }HPA + PDB: a CPU (or custom-metric) HPA from minReplicas: 3 to 12, and a PDB with minAvailable: 2.
What each control buys: readiness → no traffic to a not-ready Pod; liveness → auto-recovery from wedging; startup → slow boots don't crash-loop; requests → the scheduler reserves capacity; memory limit above peak → no surprise OOM-kills; maxUnavailable: 0 → zero-downtime deploys; HPA → capacity follows load; PDB → maintenance can't breach quorum. Spread replicas across AZs (topology spread) so involuntary node loss can't either. Every line maps to a failure it prevents — which is exactly what the SRE checklist is asking for.