Config, health & autoscaling

1 · The three probes — and what each gates

Here's the outage nobody forgets. Your app is fine — a little slow to boot because it warms a model cache — but Kubernetes decides it's dead, kills it, and restarts it. It's slow to boot again, so Kubernetes kills it again. You've built a machine that murders your app on a loop, and the app was never actually broken. That whole disaster comes from getting probes wrong, which is why we start here.

Think of probes as the questions a hospital nurse asks a patient, not a single "are you OK?" A nurse doesn't collapse three very different checks into one. "Are you conscious?" (if not, we intervene) is a different question from "Are you ready to be discharged and see visitors?" (if not, we just keep you resting) — and both differ from "Have you woken up from the anaesthetic yet?" (don't judge the other two until they have). Kubernetes has exactly these three questions, and each triggers a different response. The single most common beginner mistake is treating them as interchangeable — copy-pasting the same endpoint into all three — which is precisely how a briefly-slow database turns into a restart storm.

Why does this matter more than it looks? Because a probe isn't passive monitoring — it's a control loop that takes action. Answer the wrong question and Kubernetes takes the wrong action: it pulls a healthy Pod out of rotation, or restarts one that just needed a moment. The three probes are readiness (should you get traffic?), liveness (should you be restarted?), and startup (has boot finished, so hold off the other two?). Readiness gates traffic — failing it quietly removes the Pod from the load balancer with no restart. Liveness gates restarts — failing it kills and recreates the container. Startup is the referee that suspends the first two until a slow-booting app has finished coming up. Now the precise version:

Kubernetes can't read your app's mind; you tell it health via probes. The three do different jobs and confusing them causes real outages:

probes.yaml        readinessProbe:            # gates traffic: pull from LB until ready
          httpGet: { path: /healthz/ready, port: 8000 }
          periodSeconds: 5
        livenessProbe:             # gates restart: kill+restart if wedged
          httpGet: { path: /healthz/live, port: 8000 }
          periodSeconds: 10
          failureThreshold: 3
        startupProbe:              # slow starters: don't let liveness kill boot
          httpGet: { path: /healthz/live, port: 8000 }
          failureThreshold: 30
          periodSeconds: 5         # allows up to ~150s to start

2 · Resource requests and limits

This is where cloud bills quietly explode — or where apps mysteriously die at 3 a.m. Set these two numbers too high and you pay for reserved capacity nobody uses; set them too low and Kubernetes packs your Pods onto a node that can't actually feed them, and they start getting throttled or killed under load. Almost every "why is our cluster so expensive?" and every "why does this Pod keep restarting with no error in the logs?" conversation traces back to requests and limits.

Here's an analogy that makes the two numbers click. Booking a restaurant, you tell them "party of four" — that's your request: the restaurant reserves a table for four and won't give it away, even if only two of you show up early. Separately, the fire marshal posts a limit: this room holds at most twenty people, full stop. The request is a guarantee to you (this much is set aside); the limit is a wall around you (you may never exceed it). They are answering completely different questions — "how much do I plan to use?" versus "what's the absolute ceiling?" — and beginners constantly conflate them.

The subtlety that bites hardest is that CPU and memory behave differently when you cross the limit, because one is compressible and the other isn't. CPU is like clock time: if you want more than your cap, Kubernetes simply hands you less — you get throttled, slowed down, but nobody dies. Memory is like physical shelf space: once it's full, you can't be "slowed down" into using less, so the kernel does the only thing it can and OOM-kills the container outright. That asymmetry is the whole reason the guidance differs — you can be a little aggressive with CPU limits (worst case: slow), but a too-low memory limit is a loaded gun (worst case: dead). And the request has a second job people forget: it's the only thing the scheduler looks at when deciding which node a Pod fits on. Get the request wrong and Pods land on nodes that are already overcommitted. Now the precise version:

Two numbers per resource, and they mean different things. The request is what the scheduler reserves to place the Pod (and what other Pods can't use). The limit is the hard cap while running. Getting these right is most of good cluster economics.

resources.yaml        resources:
          requests:            # scheduler reserves this; used for bin-packing
            cpu: "250m"        # 0.25 of a core
            memory: "256Mi"
          limits:              # hard cap at runtime
            cpu: "1"           # throttled above 1 core
            memory: "512Mi"    # OOM-killed above 512Mi

3 · Rolling updates and rollbacks

Deploying a new version used to mean a maintenance window and a held breath. You'd take the app down, swap the code, bring it back up, and pray it came back healthy — with users staring at an error page the whole time. Kubernetes makes that ritual obsolete: it can swap versions while the app keeps serving, and if the new one is broken it stops and backs out on its own. That capability is the rolling update, and it's a big part of why teams deploy ten times a day instead of once a quarter.

Picture repainting a long hallway that people are still walking through. You don't close the whole hallway — you rope off one section, paint it, let it dry and reopen it, then move to the next section. At every moment most of the hallway is open, so foot traffic never stops. A rolling update does exactly this with your Pods: it brings up a Pod on the new version, waits until that Pod is actually ready to serve, and only then retires one old Pod — shifting your replicas from all-old to all-new a slice at a time. The number of Pods answering requests barely dips, so users notice nothing.

The load-bearing detail — and the one beginners skip — is the phrase "waits until it's actually ready." That readiness check from §1 is what makes the whole thing safe: a new Pod is added to the load balancer only after it passes readiness, and an old Pod is removed only after the new one is in. The common mistake is shipping a new version with a broken or missing readiness probe. Then Kubernetes thinks each new Pod is ready the instant it starts, tears down the old ones, and cheerfully replaces a working app with a broken one at full speed. Wire readiness correctly and the opposite happens: a bad version can't pass readiness, so the rollout stalls with the old Pods still serving — a failed deploy instead of an outage. And because the previous ReplicaSet is kept around, undoing a bad rollout is a single command, not a frantic redeploy. Now the precise version:

Changing the image on a Deployment triggers a rolling update by default: new Pods come up and pass readiness before old ones are removed, so traffic never drops. If it goes wrong, rollback is one command because the old ReplicaSet is still there.

rollout.shkubectl set image deploy/web app=ghcr.io/example/web:1.3.0
kubectl rollout status deploy/web      # blocks until healthy or fails
kubectl rollout history deploy/web     # revisions you can return to
kubectl rollout undo deploy/web        # instant rollback to previous revision
kubectl rollout undo deploy/web --to-revision=2

deployment "web" successfully rolled out

4 · HorizontalPodAutoscaler

Traffic isn't flat, but a fixed replica count is. Provision for your busiest hour and you burn money all night running idle Pods; provision for the average and you fall over the moment a spike hits. Sizing by hand means either overpaying or getting paged — and you can't sit there watching a dashboard and typing kubectl scale at 2 a.m. anyway. The HorizontalPodAutoscaler is the robot that watches the dashboard for you and adds or removes Pods to match demand.

The mental model is a thermostat, and it's worth holding onto because it explains the HPA's whole personality. You set a target ("keep the room at 20°C"), the thermostat reads the actual temperature, and it acts to close the gap — more heat when it's cold, less when it's warm. The HPA does the same with a metric: you say "keep average CPU at 60%," it reads the current average across your Pods, and it computes how many replicas would bring the reading back to target. Too hot (CPU above 60%) → add Pods; comfortably cool → remove some. It's a feedback loop, not a fixed rule.

Two things about that loop trip people up. First, it scales up fast and down slow on purpose: adding capacity late means dropped requests, but removing it too eagerly causes flapping — a jittery add-remove-add cycle that's worse than doing nothing — so the HPA is deliberately cautious about scaling in. Second, and this is the mistake almost everyone makes once: "60% CPU" means 60% of the CPU request you set in §2, not 60% of a whole core. The request is the denominator. If you never set a CPU request, there's no denominator, the HPA can't compute a percentage, and it just sits there inert while your app melts — no error, no scaling, nothing. The HPA also needs a metrics source feeding it numbers (metrics-server for CPU and memory, or an adapter for custom metrics like queue depth). Note too that the autoscaling API and its metric options evolve across Kubernetes versions, so verify the current shape against the docs rather than trusting a snippet's age. Now the precise version:

The HPA adjusts a Deployment's replica count to hit a target metric — classically CPU. It needs a metrics source (metrics-server for CPU/memory, or an adapter for custom metrics). It scales up quickly under load and down conservatively to avoid flapping.

hpa.yamlapiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60   # % of the CPU *request*

5 · PodDisruptionBudget

The outage you cause yourself is the most embarrassing kind. You have three replicas spread across three nodes — plenty of redundancy — and then a routine node upgrade drains all three nodes at once because the cluster was in a hurry. For a few seconds you have zero Pods serving, and the pager goes off during scheduled maintenance you thought was safe. Your replicas were fine; the timing of the maintenance wasn't. A PodDisruptionBudget exists to stop precisely this.

The key idea is that not all Pod deaths are the same, and you can only govern one kind. Think of a 24-hour shop with a staff of three. Some absences you simply can't plan for — someone calls in sick, a bus breaks down; that's involuntary disruption (a node crashes), and no policy prevents it. But scheduled things — sending people on training, closing for a deep-clean — those are voluntary, and for those a sensible manager sets a floor: "never send more than one person off at a time; at least two must be on the floor." A PodDisruptionBudget is that floor, applied to voluntary disruptions like node drains, cluster autoscaler scale-downs, and rolling node upgrades.

What makes it work is that it doesn't stop maintenance — it makes maintenance wait its turn. When something tries to voluntarily evict a Pod, Kubernetes checks the budget first: if evicting now would drop you below minAvailable, the eviction blocks until enough Pods are healthy again, then proceeds one at a time. The mistake to avoid is expecting a PDB to save you from a hard node crash — it can't, because a crash never asks permission. Involuntary loss is what spreading replicas across nodes and wiring readiness are for; the PDB is strictly a brake on the planned disruptions you'd otherwise let run too fast. (PDB semantics also shift between versions, so confirm the current behaviour against the docs.) Now the precise version:

There are two kinds of disruption: involuntary (a node crashes — you can't prevent it) and voluntary (you drain a node for an upgrade). A PodDisruptionBudget constrains the voluntary kind so routine maintenance can't take you below a safe number of Pods.

pdb.yamlapiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web
spec:
  minAvailable: 2          # or: maxUnavailable: 1
  selector:
    matchLabels:
      app: web

        readinessProbe:
          httpGet: { path: /healthz/ready, port: 8000 }
          periodSeconds: 5
        livenessProbe:
          httpGet: { path: /healthz/live, port: 8000 }
          periodSeconds: 10
          failureThreshold: 3

        resources:
          requests: { cpu: "250m", memory: "256Mi" }
          limits:   { cpu: "1",    memory: "640Mi" }

kubectl set image deploy/web app=ghcr.io/example/web:1.3.0
kubectl rollout status deploy/web       # watch it progress or stall

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: web }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: web }
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 60 } }

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: web }
spec:
  minAvailable: 2
  selector: { matchLabels: { app: web } }

        readinessProbe:  { httpGet: { path: /healthz/ready, port: 8000 }, periodSeconds: 5 }
        livenessProbe:   { httpGet: { path: /healthz/live,  port: 8000 }, periodSeconds: 10, failureThreshold: 3 }
        startupProbe:    { httpGet: { path: /healthz/live,  port: 8000 }, failureThreshold: 30, periodSeconds: 5 }
        resources:
          requests: { cpu: "250m", memory: "256Mi" }
          limits:   { cpu: "1",    memory: "640Mi" }

  strategy:
    type: RollingUpdate
    rollingUpdate: { maxUnavailable: 0, maxSurge: 1 }