Production ops, DR & cost

1 · The troubleshooting playbook (decision table)

It's 3am, your phone is buzzing, and a Pod is red. This is the moment that separates a calm ten-minute fix from a two-hour flail — and the difference is almost never how clever you are. It's whether you have a procedure. Under stress, freshly woken, you don't want to be inventing a debugging strategy; you want a checklist your fingers already know. That's what this section gives you: a way to turn "something's broken" into "I recognise this exact signature."

Here's the reassuring truth that makes on-call survivable: Kubernetes doesn't fail in a million creative ways. It fails in about five. A container that keeps crashing, one killed for using too much memory, one whose image can't be pulled, one that never gets scheduled, and one evicted because its node ran out of room. Each of those leaves a distinctive fingerprint — a specific status string, a specific exit code, a specific line in the events. Think of it like a doctor reading symptoms: chest pain plus shortness of breath points somewhere specific, and you confirm with a test rather than guessing. The decision table below is your differential diagnosis, and the "Confirm with" column is the test that turns a hunch into certainty before you touch anything.

The mistake that costs everyone their first year: fixing before confirming. You see a crashing Pod, you assume it's the memory limit, you bump the limit, redeploy, wait five minutes… and it crashes again, because the real cause was a typo in a config file. Every guess you act on costs a full deploy-and-observe cycle, and you can burn an hour that way. The discipline the table enforces is match the fingerprint first, change one thing second — read the status, read the exit code, read the events, then act. Slow is smooth, and smooth is fast, especially when you're barely awake.

Most Pod failures fall into a handful of buckets with distinctive signatures. Match the symptom to the cause, then confirm with the evidence column — this is the table to keep open on-call:

triage.shkubectl get pods -n web -o wide                 # phase, restarts, node, age
kubectl describe pod <pod> -n web               # events, last state, probe/schedule reasons
kubectl logs <pod> -n web --previous            # logs from the CRASHED container, not the new one
kubectl get events -n web --sort-by=.lastTimestamp   # what k8s did recently
kubectl top pod <pod> -n web                    # live CPU/mem (needs metrics-server)

NAME          READY   STATUS             RESTARTS      AGE   NODE
chat-7d9...   0/1     CrashLoopBackOff   6 (2m ago)    14m   ip-10-0-3-21
# describe -> Last State: Terminated, Reason: OOMKilled, Exit Code: 137

2 · Pending, eviction & QoS under node pressure

Two of the five failures aren't your app's fault at all — they're arguments about real estate. A Pod that won't schedule and a Pod that gets kicked off a node are both about the same scarce thing: room on a machine. Get these wrong and you'll waste an afternoon reading application logs that are perfectly fine, because the problem was never in the code — it was in how you asked for space.

Picture the Kubernetes scheduler as a host seating parties at a restaurant. When you submit a Pod you hand over a reservation slip — "I need a table for this much CPU and this much memory." That slip is your requests. The host walks the floor looking for a table that fits; if none does, your party waits at the door, and that waiting is Pending. The crucial, counter-intuitive part: the host seats you based on the slip, not on how much you'll actually eat. Reserve a table for eight when two of you show up, and you've blocked six seats nobody uses — which is exactly the cost bug section 5 comes back to.

Eviction is the darker cousin. When a node genuinely runs low on memory or disk, the kubelet can't politely wait — it has to throw someone out to save the machine, and it needs a rule for who goes first. That rule is QoS class, and here's the mistake almost everyone makes: they think QoS is a setting you choose. It isn't. It's derived from how you wrote your requests and limits. Set them equal and Kubernetes silently grants you Guaranteed (the last to be sacrificed); set none at all and you're BestEffort (first out the door). You're picking your Pod's survival odds without realising it every time you write a manifest. The lesson: for anything you can't afford to lose, make the request/limit relationship deliberate.

Two of the table's rows deserve depth because they're about scheduling and resources, not app bugs. A Pending Pod almost always means the scheduler can't find a node that fits — read the FailedScheduling message: 'Insufficient memory', 'had untolerated taint', 'didn't match node affinity'. Under node pressure (memory/disk low), the kubelet evicts Pods, and the order is decided by QoS class, which is set by how you wrote requests/limits:

3 · Backup, restore & disaster recovery

The worst day of an operator's career usually starts with a command that worked exactly as asked. Someone runs kubectl delete namespace against the wrong context, or a bad migration wipes a database, or a cluster simply falls over — and in that instant the only thing standing between you and a résumé-updating disaster is whether you can restore. Not whether you have backups. Whether you can restore. The gap between those two words is where careers quietly end.

The seductive trap here is a phrase you'll hear from smart people: "everything's in Git, so we're covered." Think about what that actually promises. Git holds your recipes — the manifests that describe what should exist. It does not hold the meal: the rows in your database, the files on a PersistentVolume, the vector embeddings your RAG index spent hours building, the state that accumulated while the app ran. Re-applying your manifests recreates empty pots and pans. It's like keeping every blueprint for your house but none of the furniture, then calling that a fire-safety plan. GitOps restores structure; it cannot restore data, and it cannot restore the cluster's own brain — the etcd datastore that remembers what actually exists.

So real disaster recovery comes in layers, each protecting against a different kind of loss, and the diagram below lays them out: your manifests in Git, your live objects and volume data via a tool like Velero, and the control-plane state in etcd. But the single most important idea in this whole section isn't any tool — it's a discipline. A backup you have never restored is not a backup; it's a hope. Backups fail silently in the most heartbreaking ways: they captured the objects but not the volume snapshots, or the scope was wrong, or the restore only works against a version you no longer run. You find out at the exact moment you can least afford to. The professionals test their restores on a schedule, into a scratch cluster, and watch the app come up with its data — because a restore that has actually run is the only backup that counts.

'The cluster is declarative and in Git, so I don't need backups' is a dangerous half-truth. GitOps restores your manifests, but not PersistentVolume data, not dynamically-created state, and not the etcd datastore itself. DR has two layers:

velero.sh# Back up one namespace, including its PVs, on a schedule:
velero backup create web-daily --include-namespaces web
velero schedule create web-nightly --schedule="0 2 * * *" --include-namespaces web

# Restore after an accidental delete / into a new cluster (DR):
velero restore create --from-backup web-daily
velero backup describe web-daily        # verify it actually captured what you think

4 · Cluster upgrades & multi-cluster HA

Here's the cruel irony of cluster upgrades: the procedure meant to keep you safe is the one most likely to cause the outage. Kubernetes moves fast and only supports a handful of recent versions, so falling behind isn't an option — an unsupported cluster stops getting security patches and eventually can't be helped when it breaks. Upgrading is a chore you will do, repeatedly, forever. The question is only whether you do it like a surgeon or like someone yanking a tablecloth.

Two rules make it surgical, and both come from the same simple fact: a Kubernetes cluster is not one thing, it's a control plane (the brain: API server, scheduler, etcd) plus a fleet of worker nodes (the muscle running your Pods). Rule one is order: upgrade the brain before the muscle, one minor version at a time. Why? A newer worker talking to an older brain is like an employee following next year's policy manual while their manager still works from last year's — they'll disagree in ways nobody tested. The API server must always be at least as new as the kubelets talking to it. Skipping versions (1.28 straight to 1.31) is the same gamble: each version's migrations assume the previous one ran.

Rule two is how you evacuate a node. To upgrade a worker you drain it — evict its Pods so they reschedule elsewhere. The naive way evicts everything at once, and if all your replicas happened to live on that node, you just took the service down to "upgrade" it. The fix is cordon then drain (stop new Pods landing, then gracefully evict) combined with a PodDisruptionBudget — a promise Kubernetes enforces that says "never voluntarily drop below N healthy replicas." The PDB is what turns a drain from a coin-flip into a guarantee. And when one cluster isn't resilient enough, the honest-but-expensive answer is running several — covered below, along with why "just run multi-region" is a decision to make on purpose, not by reflex.

Kubernetes releases often and supports only a few recent minors, so upgrading is routine, not optional. The safe order is control plane first, then node groups, one minor version at a time, draining nodes so Pods reschedule gracefully:

drain.sh# 1) Upgrade the control plane one minor at a time (provider-specific; EKS via console/eksctl/IaC).
# 2) For each node: cordon (no new pods), drain (evict gracefully), then replace/upgrade:
kubectl cordon ip-10-0-3-21
kubectl drain ip-10-0-3-21 --ignore-daemonsets --delete-emptydir-data --timeout=5m
# PodDisruptionBudgets keep enough replicas up during the drain.
kubectl uncordon ip-10-0-4-15                 # after the new node is Ready

pdb.yamlapiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: llm-api-pdb, namespace: web }
spec:
  minAvailable: 2                 # never voluntarily evict below 2 ready pods
  selector: { matchLabels: { app: llm-api } }

Multi-cluster / HA. One cluster is a single failure/blast-radius domain. Higher tiers run multiple clusters (per-region or per-blast-radius) behind global load balancing, replicate data across them, and manage them with a fleet tool (Argo CD app-of-apps, cluster APIs). It buys resilience and blast-radius isolation at a real cost in complexity — adopt when the availability requirement (or regulatory isolation) justifies it, not by default.

5 · Right-sizing requests — the biggest cost lever

Somewhere in your company there is a cloud bill three times larger than it should be, and the cause is almost boringly simple. When finance asks why the Kubernetes cluster costs so much while the dashboards show nodes sitting at 30% utilisation, the answer is nearly always the same one word: over-requesting. It's the single highest-leverage cost fix in this entire chapter, and it requires no new tools — just the courage to ask for what you actually use.

Recall the restaurant from section 2: the scheduler seats you by your reservation slip, not your appetite. Now scale that up. Every Pod reserves, say, 2 GiB of memory but truly uses 700 MiB. The scheduler dutifully packs nodes full of reservations, sees no room left, and the autoscaler buys more machines — all while those machines run near-idle. You're paying for a restaurant booked solid with tables of eight where everyone ordered a side salad. The utilisation graph looks embarrassingly low and the cluster keeps growing, which feels contradictory until you realise the scheduler was never looking at usage. It was looking at your slip.

The fix is to make the slip honest: set requests from observed usage — typically the p95 (the level 95% of samples stay under, so you're covered on all but the spikiest moments) plus some headroom for safety. That one change frees enormous amounts of "reserved but unused" capacity to be packed with real work. But there's a genuine trade-off to respect, and it's where beginners over-correct into a new disaster: memory and CPU behave differently under pressure. Memory is incompressible — blow past your limit and the kernel kills you (the OOMKilled you met in section 1), so you want memory requests set close to a real, generous ceiling. CPU is compressible — it just gets throttled, slowing you down invisibly rather than crashing you — so a too-tight CPU limit shows up as a baffling latency mystery, not an error. Cut fat, yes; but cut it knowing which resource forgives you and which one doesn't. The math below makes the waste concrete and offline-runnable.

The single most common cluster cost bug is over-requesting: Pods reserve far more CPU/memory than they use, so nodes fill up on reservations while sitting near-idle, and you pay for nodes you don't need. Requests should be set from observed usage (K8 metrics), with headroom. The gap is easy to quantify offline:

right_size.pydef right_size(observed_p95_mib, headroom=0.30):
    """Set the memory request from observed p95 usage plus headroom."""
    return round(observed_p95_mib * (1 + headroom))

def waste(requested_mib, observed_p95_mib, replicas):
    """Reserved-but-unused memory across all replicas (what you pay for, unused)."""
    return max(0, requested_mib - observed_p95_mib) * replicas

# A pod requests 2048 MiB but its p95 usage is only 700 MiB, x20 replicas:
print(right_size(700))                 # -> 910 MiB  (a sane request)
print(waste(2048, 700, 20), "MiB")     # -> 26960 MiB (~26 GiB reserved and wasted)
# Cutting the request from 2048 -> 910 frees ~22 GiB of schedulable memory to bin-pack.

910
26960 MiB

6 · Karpenter, Spot & bin-packing

Right-sizing made your reservations honest; now you get to spend that honesty. Once Pods ask for what they truly need, you can pack far more of them onto each machine and buy fewer, cheaper machines to hold them — and that's the second half of the cost story. If section 5 was about not booking tables for eight, this section is about seating the restaurant efficiently and choosing when to rent the room by the hour instead of the year.

Start with bin-packing, because it's the intuition everything else rests on. Imagine boxes of different sizes and a supply of identical shipping crates: your job is to fit all the boxes into as few crates as possible. Pods are the boxes, nodes are the crates, and the classic greedy trick — sort boxes largest-first, drop each into the first crate it fits — is exactly what the offline model below does. Here's the payoff that ties the two sections together: honest requests are what make the boxes their true size, so they pack tight. Over-sized reservations are boxes stuffed with packing foam — they waste crate space no algorithm can reclaim. Right-sizing pays off twice: once by freeing reservations, again by letting the packer do its job.

Karpenter is the piece that automates this on AWS, and it's smarter than a fixed pool of pre-bought machines. It watches for Pods stuck Pending (nobody could seat them) and launches a node shaped to fit — often one well-chosen instance rather than another cookie-cutter box — then, as load drops, consolidates: it rebin-packs the survivors onto fewer nodes and deletes the empties. Layer on Spot instances — spare capacity the cloud rents at a steep discount — and the savings compound. But Spot comes with an honest catch you must design around: the provider can reclaim it with about two minutes' notice. That's fine for interruptible, stateless work (batch jobs, background inference) and reckless for a stateful database. The reliability guard is the same PodDisruptionBudget from section 4 — it keeps both Spot reclamation and Karpenter's own consolidation from ever dropping you below a safe number of replicas. Cheap and reliable aren't opposites here; they're two dials you tune together.

Once requests are honest, node provisioning is the next lever. Karpenter (on AWS) watches for unschedulable (Pending) Pods and launches right-sized nodes to fit them — often a single well-chosen instance instead of a fixed node group — then consolidates (bin-packs workloads onto fewer nodes and removes empty/underused ones) as load drops. Combined with Spot instances for interruptible work, this is where the big savings live.

nodepool.yaml# Karpenter CRDs/fields change across versions — verify against the current Karpenter docs.
apiVersion: karpenter.sh/v1
kind: NodePool
metadata: { name: default }
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: [spot, on-demand]    # prefer Spot, fall back to on-demand
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized   # bin-pack + remove waste
    consolidateAfter: 1m

Bin-packing is why right-sizing (section 5) pays off twice: honest requests let the scheduler and Karpenter pack more Pods per node. The idea is a first-fit-decreasing packing — modeled offline:

binpack.pydef pack(pod_mib, node_mib):
    """Greedy first-fit-decreasing: how many nodes to fit these pods (memory only)."""
    nodes = []                       # each entry = remaining free MiB on that node
    for size in sorted(pod_mib, reverse=True):
        for i, free in enumerate(nodes):
            if free >= size:
                nodes[i] -= size
                break
        else:
            nodes.append(node_mib - size)   # open a new node
    return len(nodes)

pods_over  = [2048] * 20            # over-requested
pods_right = [910]  * 20            # right-sized (from section 5)
print(pack(pods_over,  node_mib=8192))    # -> 5 nodes
print(pack(pods_right, node_mib=8192))    # -> 3 nodes  (~40% fewer for the same work)

5
3

kubectl get pods -n web -o wide          # phase, RESTARTS, node, age
kubectl describe pod  -n web        # Events + Last State (why it died)
kubectl logs  -n web --previous     # logs from the CRASHED container
kubectl get events -n web --sort-by=.lastTimestamp

velero schedule create web-nightly --schedule="0 2 * * *" \
  --include-namespaces web            # objects + PV snapshots
velero restore create --from-backup web-nightly   # into a scratch cluster to TEST

kind: PodDisruptionBudget
spec: { minAvailable: 2, selector: { matchLabels: { app: llm-api } } }

right_size = lambda p95, h=0.30: round(p95*(1+h))
right_size(700)            # 910 MiB request (was 2048)
# waste per replica = 2048-700 = 1348 MiB; x20 = ~26 GiB reserved unused
# bin-pack 20 pods on 8192-MiB nodes: 2048-sized -> 5 nodes; 910-sized -> 3 nodes