Internals & debugging

1 · What a container actually is advanced

Start with the bug this section prevents. Someone's container keeps getting killed. They see "137" and "OOMKilled," assume the app threw an exception, and wrap the whole thing in a try/except. It doesn't help — because they're carrying the wrong mental model. They think a container is a tiny virtual machine, so they reason about it like one. It isn't, and almost every confusing container failure traces back to that one misconception. Fix the model and the failures stop being mysterious.

Here's the model that actually works: a container is just a normal process the kernel has agreed to lie to. Picture a new hire on their first day. The company gives them a badge, a desk, and a login, and carefully arranges things so they only ever see their files, their tools, their corner of the office — it feels like they have the whole building to themselves. But there's no separate building. They're in the same office as everyone else; the company just curated what they can see. That curation is a namespace. And the office also caps how much they can consume — desk space, printer quota, bandwidth. That budget is a cgroup.

Hold those two words apart, because conflating them is the single most common mistake: namespaces control what a process can see; cgroups control what it can use. A namespace is about visibility — the container looks around and sees only its own processes, its own /, its own network card, and it genuinely believes it's PID 1 of a fresh machine. A cgroup is about consumption — it's the meter that says "you get 256 MB and half a CPU, no more." When your container "can't see the host's other processes," that's a namespace. When it gets throttled or killed at a memory ceiling nobody in your code set, that's a cgroup. Two different mechanisms, two different classes of bug.

The third piece is the filesystem. A container's disk is a stack of transparencies: read-only image layers on the bottom (shared, never touched) and one thin writable layer laid on top — that's the union filesystem (overlayfs). Your writes only ever land on that top sheet, and the top sheet is thrown away when the container is removed. That's why "I wrote a file, restarted the container, and it's gone" is not a bug — it's the design. And because all of this is just kernel bookkeeping around a normal process — the same kernel the host runs — a container is cheap to start (no OS to boot) but also not a security wall: an escape puts an attacker straight onto the host, which is exactly why the hardening in CD6 matters. A container is not a lightweight VM. It is an ordinary Linux process that the kernel has been told to lie to: namespaces give it a private view of PIDs, mounts, network, users, and hostname; cgroups cap how much CPU/memory/IO it can use; and a union filesystem (overlayfs) stacks read-only image layers under a thin writable layer. Same kernel as the host — that is why containers are cheap and why an escape lands you on the host.

2 · The OCI / containerd stack under Docker advanced

You've probably said "Docker started my container" — but "Docker" didn't do most of that work. The word hides a relay team of four separate programs handing the job down a line, and knowing who's who is the difference between staring at the wrong log for an hour and going straight to the one that has your answer. When a container won't start, "check the Docker logs" is useless advice until you know which layer failed.

Think of ordering food through a delivery app. You tap "order" in the app (the Docker CLI) — but the app doesn't cook. It sends your request to the restaurant's back-office system (dockerd, the daemon), which passes the ticket to the kitchen manager (containerd), who assigns it to the line cook that actually stands at the stove (runc). Only that last cook touches the food — runc is the one that talks to the kernel, carves out the namespaces and cgroups from section 1, and execs your process. Everyone above it is coordination and paperwork. So when the meal is wrong, you ask which station dropped it: a hang in docker run often lives in dockerd or containerd, not in your image at all.

The mistake this untangles: people think Kubernetes "runs on Docker," so they apply Docker daemon advice to a cluster and it silently doesn't apply. In a modern cluster there is no dockerd — Kubernetes' kubelet talks to containerd directly through a standard socket called the CRI (Container Runtime Interface), skipping the top two layers entirely. That's possible because of the OCI standard: the Open Container Initiative defines an image format and a runtime format so any compliant tool can run any compliant image. It's the shipping-container standard of software — Docker builds the box, and containerd, Podman, or CRI-O can all pick it up and run it unchanged. Learn the layers once and every "why won't this start" question gets a much shorter list of suspects. "Docker" is several layers. The Docker CLI talks to the dockerd daemon, which delegates to containerd (the container lifecycle manager), which uses runc (the low-level OCI runtime) to actually create the namespaces/cgroups and exec your process. Kubernetes skips Docker entirely and talks to containerd via the CRI. Knowing the layers tells you which logs to read when something breaks.

3 · Resource limits & OOMKilled (exit 137) professional

This is the single most misdiagnosed container failure, and it costs teams days. The container dies. The exit code is 137, the status says OOMKilled, and the natural instinct — the wrong instinct — is to treat it like an application bug: add logging, wrap the risky code in try/except, redeploy. It dies again, identically. The reason it won't die away is that your app never had a chance to run any of that code. It wasn't your program that decided to stop; it was the kernel that decided to execute it.

Remember the cgroup from section 1 — the meter that says "you get 256 MB." When your process tries to allocate past that ceiling, the kernel's OOM killer (out-of-memory killer) steps in and shoots the process with SIGKILL, signal 9. Think of a landlord who set a strict weight limit on a shelf: the moment you stack one book too many, they don't send a warning letter — they walk in and sweep the whole shelf onto the floor, mid-sentence, no chance to tidy up. SIGKILL is that sweep. It is the one signal a program is forbidden from catching, by design — you cannot install a handler for it, cannot log it, cannot clean up. That is precisely why try/except is useless here: there is no exception, no stack unwind, no Python frame left to run. The process is simply gone, and the container reports exit 137 — which is the convention 128 + signal number, and 128 + 9 = 137.

So the mistake to name out loud: 137 / OOMKilled is a resource problem, never a code-path problem. Chasing it in your application logic is looking for your keys under the streetlight. The two real fixes both live at the memory boundary. Either your limit is genuinely too small for honest usage — then raise the limit (and, on Kubernetes, set a memory request too, so the scheduler reserves the RAM up front) — or your app is using more memory than it should: a leak, an unbounded cache, buffering a whole file when you could stream it, loading a model twice. The way you tell which is by measuring real usage (docker stats) and sizing the limit to reality plus headroom, rather than guessing. The labs below turn that exit code into an action. cgroups enforce a hard memory ceiling. When a process tries to exceed it, the kernel's OOM killer terminates it with SIGKILL — the container exits 137 (128 + signal 9) and shows OOMKilled. This is the most misread container failure: the app didn't crash, it was killed for using too much memory. The fix is a right-sized limit and a request, not a bigger try/except.

oom.sh# Cap memory and CPU. Exceeding --memory gets the process SIGKILL'd (exit 137).
docker run --memory=256m --cpus=0.5 myapp:1.4.2

# After it dies, read WHY — OOMKilled and the exit code are recorded here:
docker inspect --format '{{.State.OOMKilled}} {{.State.ExitCode}} {{.State.Error}}' <container>
# -> true 137
# See real-time usage to right-size the limit:
docker stats --no-stream <container>
# Verify field paths/flags against current Docker docs.

exit_triage.pydef diagnose(exit_code, oom_killed):
    if oom_killed or exit_code == 137:
        return "OOMKilled — hit the cgroup memory limit; raise limit or cut memory use"
    if exit_code == 0:
        return "clean exit (PID 1 returned 0) — check restart policy if it shouldn't stop"
    if exit_code == 143:
        return "SIGTERM (128+15) — graceful stop requested; fine if shutdown was clean"
    if exit_code == 139:
        return "SIGSEGV (128+11) — native crash / segfault in the process"
    if exit_code == 1:
        return "app error — read the app logs; the program itself returned non-zero"
    if exit_code == 126 or exit_code == 127:
        return "entrypoint not executable / not found — check the CMD path & +x bit"
    return f"exit {exit_code} — read logs; map 128+N to signal N if > 128"

for ec, oom in [(137, True), (143, False), (1, False), (127, False)]:
    print(ec, "->", diagnose(ec, oom))

137 -> OOMKilled — hit the cgroup memory limit; raise limit or cut memory use
143 -> SIGTERM (128+15) — graceful stop requested; fine if shutdown was clean
1 -> app error — read the app logs; the program itself returned non-zero
127 -> entrypoint not executable / not found — check the CMD path & +x bit

4 · Debugging a crashing container professional

A crash-looping container is a crime scene where the body keeps disappearing. The container starts, dies, restarts, dies again — and each dead instance takes its evidence with it. Beginners lose here by reaching for the one habit that doesn't work: they try to exec a shell "to poke around," but the container is already dead by the time they hit Enter, or (if it's a hardened distroless image) there's no shell to exec into at all. Then they're stuck, watching the restart counter climb, with no plan. The fix isn't a cleverer command — it's an order of operations that gets the most evidence with the least access.

Here's the reframe that makes it calm instead of panicky: you don't need the container alive, and you don't need a shell, to learn why it died. Think like an accident investigator, not a paramedic. The paramedic wants the patient breathing right now; the investigator reads the black box after the fact. Two facts do most of the work for you. First, the logs of the previous, already-dead instance are still on disk — docker logs, or kubectl logs --previous — so the last words your app spoke before it fell over are recoverable even though the process is gone. Second, inspect hands you the container's post-mortem report without any shell at all: the exit code, the OOM flag, the last state, the mounts, the environment. Logs plus inspect answer the overwhelming majority of "why is it dying" questions, and neither needs the container to be running.

Only after those do you escalate to touching a live container. If the image has a shell and an instance stays up long enough, exec in and look around. And for the hard case — a distroless, crash-looping container with no shell, which is exactly what your CD6 hardening produced — the tool is an ephemeral debug container. This is the clever move worth understanding: instead of adding tools back into your hardened image (which would un-do the hardening), you launch a separate throwaway image that's full of debugging tools and tell it to join the target's namespaces — the same PID and network view from section 1. It's like sending a specialist into the sealed room through a connecting door, carrying their own kit, rather than stocking the sealed room with tools it should never contain. You keep the minimal image and still get a full toolbox. The table below maps common symptoms straight to the first place to look. A container that crash-loops gives you a shrinking window to inspect it. The toolkit, in order: read the logs (including the previous crashed instance), inspect the config and last state, exec a shell if the image has one, and — for a distroless or crash-looping container with no shell — attach an ephemeral debug container that brings its own tools into the target's namespaces.

debug.sh# 1) Logs — and the logs of the PREVIOUS, already-dead instance:
docker logs <container>
kubectl logs <pod> --previous          # the crashed instance's logs, K8s

# 2) Inspect — exit code, OOM flag, last state, mounts, env:
docker inspect <container> --format '{{json .State}}'

# 3) Exec a shell IF the image has one (won't work on distroless):
docker exec -it <container> sh

# 4) No shell / crash-looping? Attach an EPHEMERAL debug container that shares
#    the target's process + network namespaces and brings its own tools:
kubectl debug -it <pod> --image=busybox --target=<container>
# Command surfaces evolve (esp. kubectl debug) — verify against current docs.

5 · Healthchecks — liveness vs readiness professional

The scariest outages aren't caused by the platform failing — they're caused by the platform doing exactly what you told it to. A team wires up a health probe, a downstream database hiccups for thirty seconds, and the platform responds by restarting every pod at once — turning a minor blip that would have healed on its own into a full self-inflicted outage. Nobody wrote a bug. They just answered the wrong question with their probe. Getting healthchecks right is less about syntax and more about knowing which question you're answering.

Because there are two completely different questions, and the whole trick is not to confuse them. Picture a shop with two staff decisions. One is: "Should this cashier be open for customers right now?" — maybe they're on a call with a supplier, so you rope off their lane and send shoppers to other tills, but you'd never fire them for it. The other is: "Is this cashier unconscious on the floor?" — then you call for a replacement. The first question is readiness: should traffic go here? The answer just adds or removes the container from the load balancer, no restart. The second is liveness: is this process wedged beyond recovery? A "no" gets it restarted. Same container, two orthogonal judgments — and the platform will act with full force on whichever you ask.

Now the mistake becomes obvious, and it's the one that caused the outage above: putting a dependency check inside the liveness probe. If your liveness endpoint reaches out to the database, the cache, and three downstream APIs, then a transient blip in any of them reports "this container is dead" — and the platform dutifully restarts a container that was perfectly healthy, over and over, in lockstep across every replica. That's answering "should I send traffic here?" (a dependency question) with the "should I restart this?" hammer. The rule that keeps you safe: liveness stays cheap and local — it should only fail if the process itself is genuinely stuck, and it deserves an initialDelaySeconds grace period so a slow-but-healthy startup isn't executed for being slow. Dependency health belongs in readiness, where a bad dependency simply pulls the pod out of rotation and lets it flow back in the moment things recover — no restart storm, no amplification. The lab shows both probes side by side so the split is concrete. A healthcheck lets the platform know whether to send traffic (readiness) or restart the container (liveness). Getting them wrong causes two classic outages: a too-aggressive liveness probe restarts a container that was merely slow (a restart storm), and a missing readiness probe sends traffic to a container that isn't warmed up yet.

health.yml# Dockerfile HEALTHCHECK (Docker/Compose):
HEALTHCHECK --interval=30s --timeout=3s --start-period=20s --retries=3 \
  CMD python -c "import urllib.request;urllib.request.urlopen('http://localhost:8000/healthz')"

# Kubernetes separates the two intents (put in the Pod spec):
# readinessProbe -> remove from load balancer until ready (don't restart)
# livenessProbe  -> restart the container if it stays unhealthy
readinessProbe:
  httpGet: { path: /healthz, port: 8000 }
  periodSeconds: 5
livenessProbe:
  httpGet: { path: /livez, port: 8000 }
  initialDelaySeconds: 20      # give slow starts room before restarting
  periodSeconds: 10
  failureThreshold: 3

6 · PID 1, signals & zombie reaping tech-lead

Here's a symptom that drives people up the wall: every single deploy takes exactly thirty seconds to stop, and in-flight requests get dropped every time. Not thirty-one, not five — always the same suspiciously round number. That number is a clock running out. The platform politely asked your app to shut down, your app never answered, and after the grace period the platform gave up and killed it. The app didn't hang because of your code; it hung because of where it sits in the process tree. This is the section that explains that eerily consistent thirty seconds — and it's a favourite senior-engineer interview question precisely because it separates people who know how containers really work from people who don't.

Inside a container, your process is PID 1 — and PID 1 isn't an ordinary process, it's the captain of the ship. On a normal Linux machine, PID 1 is init/systemd, a purpose-built program with two jobs most developers never think about. First, it receives the shutdown signals and decides what to do with them. Second, it's the "parent of last resort" that cleans up orphaned children so they don't pile up as zombies. The kernel treats PID 1 specially: it does not install the default signal handlers it gives every other process. So when your Python or Node app is suddenly promoted to captain, it inherits duties it was never written to perform — and if it doesn't perform them, things quietly rot.

That produces two distinct bites, and it's worth keeping them separate because they have separate fixes. The signal bite: docker stop and Kubernetes send SIGTERM first — "please wind down." Because PID 1 has no default handler, if your app doesn't explicitly listen for SIGTERM it simply ignores it; the platform waits out the grace period and then sends the uncatchable SIGKILL. That's your mysterious thirty seconds and your dropped requests. There's a sneakier version, too: writing CMD python app.py in shell form makes /bin/sh the captain instead of your app, and the shell typically won't pass the signal down — so even a perfectly written SIGTERM handler never fires. The fix is exec form, CMD ["python", "app.py"], which makes your process PID 1 directly, plus actually installing a handler that drains in-flight work and exits 0. The zombie bite: if your app spawns child processes, PID 1 must reap them when they finish, and most app runtimes don't — so defunct processes accumulate. The fix is to run a tiny purpose-built init (Docker's --init flag bakes in tini) as the real captain, delegating the reaping to something built for it. Two duties, two fixes; the labs show both. Your process runs as PID 1 inside the container, and PID 1 is special: the kernel does not install default signal handlers for it, and it is responsible for reaping orphaned child processes (zombies). Two consequences bite in production: (1) if PID 1 ignores SIGTERM, docker stop waits then SIGKILLs it (slow, unclean shutdown); (2) a shell-form CMD makes the shell PID 1, which doesn't forward signals to your app.

Dockerfile.pid1# WRONG: shell-form makes /bin/sh PID 1; it doesn't forward SIGTERM to python.
# CMD python app.py

# RIGHT: exec-form makes YOUR process PID 1 so it receives signals directly.
CMD ["python", "app.py"]

# If your app spawns children (workers, subprocesses), PID 1 must reap zombies.
# Either handle it in-app, or add a tiny init as PID 1:
#   docker run --init myapp        # Docker's built-in init (tini)
# or bake tini into the image and use it as the entrypoint. Verify current syntax.

graceful_shutdown.pyimport signal, sys, time

shutting_down = False

def handle_term(signum, frame):
    global shutting_down
    shutting_down = True
    print(f"received signal {signum} -> draining, then exit", flush=True)

# PID 1 gets NO default handlers — you MUST install them or SIGTERM is ignored.
signal.signal(signal.SIGTERM, handle_term)   # docker stop / k8s sends SIGTERM first
signal.signal(signal.SIGINT, handle_term)    # Ctrl-C

def serve_one_request():
    time.sleep(0.1)   # pretend to handle in-flight work

while not shutting_down:
    serve_one_request()

print("finished in-flight work, exiting 0", flush=True)
sys.exit(0)   # clean exit -> container exit code 0, fast stop (no SIGKILL wait)

received signal 15 -> draining, then exit
finished in-flight work, exiting 0

Knowledge check check yourself

def diagnose(exit_code, oom_killed):
    if oom_killed or exit_code == 137:
        return "OOMKilled — cgroup memory limit; raise limit or cut memory use"
    if exit_code == 0:   return "clean exit — check restart policy"
    if exit_code == 143: return "SIGTERM (128+15) — graceful stop"
    if exit_code == 139: return "SIGSEGV (128+11) — native crash"
    if exit_code == 1:   return "app error — read the app logs"
    if exit_code in (126, 127): return "entrypoint not executable/not found"
    return f"exit {exit_code} — map 128+N to signal N if > 128"

# 1) crashed instance's logs:
kubectl logs <pod> --previous
# 2) exit code / OOM / last state — no shell needed:
docker inspect <container> --format '{{json .State}}'
# 3) exec fails on distroless (no /bin/sh):
docker exec -it <container> sh        # -> not found
# 4) bring your own tools into the target's namespaces:
kubectl debug -it <pod> --image=busybox --target=<container>

readinessProbe:            # remove from LB if deps unusable (no restart)
  httpGet: { path: /healthz, port: 8000 }
  periodSeconds: 5
livenessProbe:             # restart only if the process itself is wedged
  httpGet: { path: /livez, port: 8000 }
  initialDelaySeconds: 20
  periodSeconds: 10
  failureThreshold: 3

CMD ["python", "app.py"]

import signal, sys
shutting_down = False
def handle_term(sig, frame):
    global shutting_down; shutting_down = True
signal.signal(signal.SIGTERM, handle_term)
while not shutting_down:
    serve_one_request()
sys.exit(0)   # clean, fast stop — no SIGKILL wait

docker run --init myapp      # Docker's built-in tini as PID 1