Security & supply chain

1 · The container supply-chain threat model advanced

Start with the day it goes wrong. Your code passed review. Your tests are green. You docker push, the deploy goes out, and three weeks later you learn a package your app never called — buried four levels deep in a transitive dependency — was quietly exfiltrating environment variables the whole time. Nobody wrote a bug. Nobody was careless. The attacker never touched your code — they touched something your code trusts. That is the shape of almost every real container breach, and it's why this chapter exists.

Here is the mental shift that makes the rest of the lesson click. Most of your career you've asked one question of code: "is it correct?" Security asks a completely different pair of questions, and you have to hold both at once: "can I trust every byte that ends up in this image?" and "if I'm wrong, how much can the running container wreck?" Those are not the same question, and — this is the part people miss — fixing one does nothing for the other. A perfectly-vetted, signed image still hands an attacker the whole host if it runs as root. A locked-down, unprivileged container is still a disaster if the code inside it was poisoned before it ever started.

The analogy that keeps it straight: think of shipping a sealed package to a customer. Provenance is the tamper-evident seal and the chain-of-custody paperwork — who packed this, and has anyone opened it since? Blast radius is what happens if a bomb did slip past the seal — is it in a blast-proof container, or sitting on the customer's kitchen table? A serious shipper does both: verify the seal and assume something might get through anyway. That "assume you'll be wrong somewhere" posture is the whole idea behind defense in depth — you don't bet the system on any single control holding, because sooner or later one won't.

The common mistake is treating security as a single gate — usually the vulnerability scanner — and calling it done once that gate is green. A clean scan feels like a passing grade, so teams ship on it and stop thinking. But a scan only answers a sliver of one axis (known CVEs in packages it could identify, at the moment it ran); it is silent on whether the image gets swapped at the registry after the scan, and silent on how much damage the container can do if a brand-new vulnerability lands tomorrow. Security review does not ask "is the code safe?" — it asks "can I trust every byte in the image, and how much damage can it do if it runs hostile?" Those are two separate axes: provenance (where did the image and its dependencies come from) and blast radius (what can the running container touch). Every control in this lesson maps to one of them — as you read each section, keep asking yourself which axis is this defending?

2 · Least privilege at runtime — non-root, read-only, drop caps advanced

Assume the attacker is already inside. That sounds defeatist, but it's the only honest starting point — you cannot prove your dependencies are clean, so you plan for the day one isn't. When that day comes, the only thing standing between "one compromised library" and "the attacker owns the machine" is how much power you gave the container in the first place. This section is the entire blast-radius axis in three settings, and every one of them is about taking things away before you're attacked, not detecting the attack.

Picture a hotel. A container running as root with a writable filesystem and a full set of Linux capabilities is a guest handed the master key — they can walk into any room, the kitchen, the safe, the manager's office. Least privilege is the opposite policy: hand every guest a key that opens only their own room and nothing else. If that guest turns out to be a thief, the damage stops at one room. You're not trying to spot the thief at the door — you're making sure that even a thief can't get anywhere. That's why these controls matter even when — especially when — you're confident the code is fine: they're insurance against the confidence being wrong.

There are three doors to lock, and they're independent — leaving any one open undoes a lot of the others. Who you are: the default container runs as root (UID 0), so a container escape hands an attacker root on the host; run as an unprivileged numeric user instead. What you can write: a writable root filesystem lets an intruder drop a binary, tamper with your app, or persist across restarts; make the root filesystem read-only and open only the specific scratch paths that genuinely need writing. What you can do: Linux capabilities are the fine-grained slices of root's power (change file ownership, open raw sockets, load kernel modules) — most apps need none of them, so drop them all and add back only the rare one you can name a reason for.

The mistake almost every team makes is thinking "we're not running as root, so we're fine" and stopping there. Non-root is necessary but nowhere near sufficient: a non-root process on a writable filesystem can still be tampered with and made to persist, and a non-root process holding NET_RAW or SETUID can still do real harm. The three controls multiply, they don't substitute — the Python model later in this section literally scores that, so you can see how much risk each open door adds back. The default container runs as root (UID 0) with a writable filesystem and a bundle of Linux capabilities. That is the single most common finding in a review. Fix all three: a numeric non-root USER, a read-only root filesystem with explicit writable tmpfs mounts, and every capability dropped.

Dockerfile# Create an unprivileged user at build time and run as its NUMERIC id.
# A numeric USER lets the orchestrator enforce runAsNonRoot (it can't resolve names).
FROM python:3.12-slim
RUN useradd --uid 10001 --create-home --shell /usr/sbin/nologin appuser
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY --chown=10001:10001 . .
USER 10001
CMD ["python", "app.py"]

run-hardened.sh# Read-only root FS: app can't write anywhere except the tmpfs you allow.
# --cap-drop=ALL removes every Linux capability; add back only what you truly need.
docker run --rm \
  --read-only \
  --tmpfs /tmp:rw,noexec,nosuid,size=64m \
  --cap-drop=ALL \
  --security-opt no-new-privileges \
  --user 10001 \
  myapp:1.4.2

# If the app truly needs to bind port 80 as non-root, add the ONE cap:
#   --cap-drop=ALL --cap-add=NET_BIND_SERVICE
# Verify flag names/behavior against current Docker docs — they evolve.

runtime_risk.pydef runtime_risk(cfg):
    """Higher score = more blast radius if the container is compromised."""
    score = 0
    if cfg["user"] == "root":         score += 50
    if cfg["writable_rootfs"]:        score += 20
    score += 5 * len(cfg["caps"])          # each retained capability adds risk
    if cfg["new_privileges"]:         score += 15   # setuid escalation possible
    return score

default = dict(user="root", writable_rootfs=True,
               caps=["CHOWN","SETUID","SETGID","NET_RAW","MKNOD","..."], new_privileges=True)
hardened = dict(user="10001", writable_rootfs=False,
                caps=[], new_privileges=False)
print("default runtime risk :", runtime_risk(default))
print("hardened runtime risk:", runtime_risk(hardened))

default runtime risk : 115
hardened runtime risk: 0

3 · Minimal & distroless bases — shrink the attack surface advanced

Here's the uncomfortable truth about your base image. When you write FROM python:3.12, you are not just getting Python — you are inheriting an entire Linux distribution: a shell, a package manager, curl, wget, compilers, dozens of libraries your app never imports. You didn't ask for any of it, you'll never use most of it, and every single piece of it is now yours to defend. An attacker who lands a foothold in your app doesn't have to bring their own tools — you've helpfully pre-installed them.

The idea that reframes this is attack surface: the sum of everything an attacker could touch or turn against you. Think of your image as a house. Every extra tool in it is another unlocked window. A shell lets an intruder run commands interactively; a package manager lets them download and install more; curl lets them phone home or pull down a second-stage payload. None of these help your app run — a Python service just needs the interpreter and its libraries — but each one is a rung on the ladder an attacker climbs after the first break-in. The fewer rungs you leave lying around, the harder it is to get anywhere. Distroless images take this to its logical end: they ship your app and its language runtime and nothing else — no shell, no package manager, no ls, nothing to pivot with. A bonus falls out for free: fewer packages means fewer CVEs, so your weekly scan is quieter and your patch treadmill is shorter.

The analogy to hold onto is the difference between a fully-furnished house and an empty vault. The furnished house (python:3.12) is comfortable to live and debug in, but a burglar who gets in has a kitchen full of knives. The empty vault (distroless) is austere and awkward — but a burglar who gets in finds bare walls and no tools. That austerity is the whole point, and it's also the price: debuggability. You cannot docker exec … sh into a distroless container to poke around, because there is no sh to run — a genuine trade-off you'll learn to work around with ephemeral debug containers in CD8.

The mistake to avoid is believing distroless is a security silver bullet — that once you're distroless, you're "secure." It is not, and the framing matters: distroless is one control on the blast-radius axis only. It shrinks what an attacker can do after a break-in, but it does absolutely nothing about how they got in. It doesn't scan your dependencies for known holes, and it doesn't prove the image running is the one you built — those are the provenance controls in the next two sections. A distroless image full of vulnerable, unsigned dependencies is still a breach waiting to happen; it's just a breach with fewer tools lying around. Every binary in the image is something an attacker can use — a shell, a package manager, curl, even ls. Distroless images ship only your app and its runtime: no shell, no package manager, nothing to pivot with. Fewer packages also means fewer CVEs to patch. The cost is debuggability — you cannot docker exec … sh into a distroless container (covered in CD8).

Dockerfile.distroless# Build with a full toolchain, then ship into distroless (no shell, no pkg mgr).
FROM python:3.12 AS build
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --target=/deps -r requirements.txt

# gcr.io/distroless/python3-debian12 ships CPython + libs, nothing else.
# Verify the exact tag/name against current distroless docs (they update per-Debian).
FROM gcr.io/distroless/python3-debian12
WORKDIR /app
COPY --from=build /deps /app/deps
COPY . .
ENV PYTHONPATH=/app/deps
USER 65532                    # 'nonroot' user baked into distroless images
ENTRYPOINT ["python", "app.py"]

4 · Vulnerability scanning & SBOM as a CI gate professional

Imagine the phone call. A critical vulnerability just dropped in a wildly popular library — the kind that makes the news, where a single crafted request lets an attacker run code on your server. Your CISO messages the whole team the same three words every engineer dreads: "are we affected?" The teams that can answer in minutes and the teams that spend the next three days grepping through Dockerfiles and paging half the org are separated by exactly two habits, and this section is both of them.

The first habit is knowing what's inside. You cannot defend, patch, or even reason about software you can't enumerate, and a container image is a black box the moment it's built — hundreds of OS packages and language libraries fused into layers. An SBOM (Software Bill of Materials) is the fix: a machine-readable, itemized packing list of exactly what went into the image and at what version. The name is the analogy — it's the same "bill of materials" a manufacturer keeps so that when a batch of a component is recalled, they know instantly which products contain it. When the next Log4Shell lands, "are we affected?" collapses from a frantic investigation into a one-line query against a list you already generated.

The second habit is checking that list against what's known to be dangerous. A vulnerability scan takes the packages in your image (or its SBOM) and matches them against public CVE databases — the running, numbered catalog of publicly disclosed security holes. Think of it as running your grocery haul past a recall notice: the tool doesn't judge whether the food is good, only whether any item is on the "known bad" list right now. Tools like Trivy and Grype do the matching; Syft produces the SBOM. The pattern is: generate the SBOM once, scan it, and gate on the result — wire the scan into CI so a build with serious findings fails the pipeline automatically, instead of relying on someone remembering to look.

Now the mistake that undoes all of it — and it comes in two flavors. The first is believing a green scan means "secure, full stop." It does not: a scan only knows about known vulnerabilities (CVEs disclosed before the scan ran) in packages it could identify, and it says nothing about provenance or blast radius. A clean scan today can be a critical finding tomorrow when a new CVE is published against a package you never changed — which is exactly why you scan continuously, not once. The second flavor is the opposite failure: drowning in low-severity noise, so the team starts ignoring the scanner entirely. The cure is to gate hard on what's serious and actionable (High/Critical with a fix available) and track the rest deliberately rather than muting it. A vulnerability scan matches the packages in your image against CVE databases. An SBOM (Software Bill of Materials) is the machine-readable list of exactly what is in the image — so when the next Log4Shell drops, you can answer "are we affected?" in seconds instead of days. Generate the SBOM once, scan it, gate on it.

scan.sh# Scan a built image for OS + language CVEs. Fail CI on High/Critical only.
trivy image --severity HIGH,CRITICAL --exit-code 1 myapp:1.4.2
# Grype does the same job (different DB); many teams run both:
grype myapp:1.4.2 --fail-on high

# Generate an SBOM in the standard SPDX or CycloneDX format:
syft myapp:1.4.2 -o spdx-json > sbom.spdx.json
# Then scan the SBOM itself (fast, no re-pull) and attach it to the image later:
grype sbom:sbom.spdx.json --fail-on critical

# Flag names (--exit-code, --fail-on, -o) drift between versions — verify current docs.

scan_gate.pydef scan_gate(findings, allowlist=()):
    """findings: [{"id","severity","pkg","fixed_in"}]. Block on unfixed High/Critical
    that are not explicitly, temporarily allowlisted (with an expiry, in real life)."""
    blocking = []
    for f in findings:
        if f["severity"] in ("HIGH", "CRITICAL") and f["id"] not in allowlist:
            reason = "no fix yet" if not f["fixed_in"] else f"fix: {f['fixed_in']}"
            blocking.append((f["id"], f["severity"], f["pkg"], reason))
    return ("FAIL" if blocking else "PASS"), blocking

findings = [
    {"id": "CVE-2024-1111", "severity": "CRITICAL", "pkg": "libxyz", "fixed_in": "1.2.4"},
    {"id": "CVE-2024-2222", "severity": "LOW",      "pkg": "zlib",   "fixed_in": ""},
    {"id": "CVE-2023-9999", "severity": "HIGH",     "pkg": "openssl","fixed_in": "3.0.14"},
]
verdict, blocking = scan_gate(findings, allowlist={"CVE-2023-9999"})  # risk-accepted, tracked
print(verdict)
for b in blocking: print("  BLOCK", *b)

FAIL
  BLOCK CVE-2024-1111 CRITICAL libxyz fix: 1.2.4

5 · Signing & verifying images — cosign / sigstore professional

Here's a gap that a clean scan quietly leaves wide open. You built the image, scanned it, it passed — everything green. Then it sits in a registry for hours or days before it's pulled and run. In that window, anyone with push access to the registry — a compromised CI token, a rogue insider, an attacker who phished a teammate — can quietly replace the bytes behind your tag with something malicious. Your deploy pulls the poisoned image, runs it, and every guarantee your scan gave you is worthless, because you're no longer running the thing you scanned. Scanning proves the image was clean when you built it; it says nothing about whether the image you're about to run is that same image.

The idea that closes the gap is the digital signature, and the everyday analogy is a tamper-evident seal on a medicine bottle. The seal doesn't make the pills good — that's the scanner's job — it proves the bottle hasn't been opened since it left the factory. A signature does the same for an image: the builder cryptographically signs it, and the deploy step verifies that signature before running. If anyone altered the image after signing, or if it was signed by someone you don't trust, verification fails and the image is refused. cosign (part of the sigstore project) is the standard tool; on deploy you check not just that it's signed but who signed it and where their identity came from.

Two details make this genuinely robust, and both are worth internalizing. First, keyless signing: traditional signing needs a long-lived private key, and the ironclad rule of security is that long-lived secrets eventually leak. Keyless signing sidesteps the whole problem — cosign gets a short-lived certificate bound to your CI's workload identity (an OIDC token, the same kind of "prove who you are" token you met in the deploy chapters) and records the signature in a public transparency log (Rekor) that anyone can audit. There is no private key sitting somewhere waiting to be stolen. Second, and this is the subtle one everyone gets wrong the first time: sign the digest, not the tag.

That mistake deserves spelling out, because it looks fine and fails silently. A tag like :latest or :1.4.2 is just a mutable pointer — a sticky note that says "the current build" — and anyone with push rights can peel it off and stick it on a different image. A @sha256:… digest is the opposite: it's a cryptographic fingerprint of the exact bytes, so it cannot point at anything else without changing. Signing a tag is like signing a sticky note; the note can be moved to a forgery and your signature comes along for the ride. So you sign the immutable digest, verify the digest, and deploy by digest — that way the bytes you verified are provably the bytes that run. A scan proves the image was clean when you built it. A signature proves the image you are about to run is the same one you built and has not been swapped at the registry. cosign (part of sigstore) signs an image by its digest and stores the signature next to it; keyless signing binds the signature to a workload identity (an OIDC token from your CI) instead of a long-lived private key.

cosign.sh# Keyless signing: cosign gets a short-lived cert from Fulcio bound to the CI's
# OIDC identity, logs the signature to the Rekor transparency log. No key to leak.
cosign sign myrepo/myapp@sha256:<digest>      # sign the DIGEST, never a mutable tag

# On deploy, verify BOTH the identity that signed it and where the token came from:
cosign verify myrepo/myapp@sha256:<digest> \
  --certificate-identity-regexp 'https://github.com/acme/.*' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com'

# Attach the SBOM from section 4 as an attestation, then verify it too:
cosign attest --predicate sbom.spdx.json --type spdxjson myrepo/myapp@sha256:<digest>

# Command surface changes across cosign releases — verify against current sigstore docs.

admission.py# Models what an admission controller / deploy gate does: trust nothing unverified.
TRUSTED_ISSUER = "https://token.actions.githubusercontent.com"
TRUSTED_IDENTITY = "https://github.com/acme/"

def admit(image):
    if "@sha256:" not in image["ref"]:
        return False, "deploying a mutable tag, not a digest"
    if not image["signed"]:
        return False, "no signature found"
    if image["oidc_issuer"] != TRUSTED_ISSUER:
        return False, f"untrusted issuer {image['oidc_issuer']}"
    if not image["signer_identity"].startswith(TRUSTED_IDENTITY):
        return False, f"untrusted signer {image['signer_identity']}"
    return True, "admitted"

good = dict(ref="myapp@sha256:abcd", signed=True, oidc_issuer=TRUSTED_ISSUER,
            signer_identity="https://github.com/acme/build")
tag  = dict(ref="myapp:latest", signed=True, oidc_issuer=TRUSTED_ISSUER,
            signer_identity="https://github.com/acme/build")
evil = dict(ref="myapp@sha256:beef", signed=True, oidc_issuer=TRUSTED_ISSUER,
            signer_identity="https://github.com/attacker/x")
for img in (good, tag, evil):
    print(admit(img))

(True, 'admitted')
(False, 'deploying a mutable tag, not a digest')
(False, 'untrusted signer https://github.com/attacker/x')

6 · Secrets — never in layers, never in env tech-lead

The bug in this section is one keystroke, and it has leaked more credentials than any exploit. An engineer, in a hurry, writes COPY .env /app/ or ENV API_KEY=sk-live-... to make the build "just work." It does work. It ships. And now that API key, database password, or cloud credential is welded into the image forever — copied to every registry, pulled onto every machine, and recoverable by anyone who runs docker history. This isn't a rare foot-gun; it's one of the most common ways real secrets end up in the wild.

To see why deleting it later doesn't save you, you need the one fact about images that everything here rests on: an image is not a single blob — it's a stack of immutable layers, and every layer is permanent. Think of it like a ledger written in ink, not a whiteboard. If layer 3 does COPY .env and layer 5 does rm .env, the file is gone from the final filesystem — but layer 3 still exists in the image's history, secret intact, waiting for anyone who peels the layers back. You cannot un-write ink by writing more ink on top. A secret that ever touched a layer is compromised the moment that image leaves your machine, and rotating it is the only real remedy.

The ENV trap is the same lesson wearing different clothes, plus a second one. First, an ENV instruction is baked into a layer just like a COPY, so docker history hands it right over. But even a secret injected at runtime as an environment variable is shakier than it feels: env values leak through docker inspect, crash dumps, error-reporting tools, and the environment of every child process the app spawns. Environment variables were designed for configuration — the port to listen on, the log level — not for secrets, and treating them as a vault is a mistake that quietly widens your exposure. And watch the build-time footgun too: a plain ARG feels temporary, but its value is visible in the image history all the same.

So what's the right move? Split the problem by when the secret is needed. For secrets you need during the build (a private package token, say), use a BuildKit secret mount: the secret is mounted into a single RUN step, used, and vanishes — it never becomes a layer and never shows in history. For secrets the app needs at runtime, don't put them in the image at all; inject them when the container starts, as files, from a real secrets manager (Vault, a cloud secret manager, or the orchestrator's secret volume), and read them at startup. The principle underneath both: a secret's lifetime should be as short as the moment it's used, and it should never be written anywhere permanent. A secret COPY'd into a layer or baked with ENV is committed forever and ships to every registry that pulls the image — a later rm or override does not remove it from the layer history. Even build-time secrets leak if you use a plain ARG. Use BuildKit secret mounts at build time and inject real secrets at runtime from a secrets manager.

Dockerfile.secret# syntax=docker/dockerfile:1
# The secret is mounted only for THIS run and never lands in a layer or history.
FROM python:3.12-slim
RUN --mount=type=secret,id=pip_token \
    PIP_TOKEN="$(cat /run/secrets/pip_token)" \
    pip install --no-cache-dir -r requirements.txt
# Build with:
#   DOCKER_BUILDKIT=1 docker build --secret id=pip_token,src=./pip_token.txt -t myapp .
# Do NOT use ARG for secrets: ARG values are visible in `docker history`.

secret_lint.pyimport re
SECRET_RX = re.compile(r"(AKIA[0-9A-Z]{16}|-----BEGIN|password\s*=|token\s*=)", re.I)

def scan_layers(dockerfile_lines, env):
    problems = []
    for i, line in enumerate(dockerfile_lines, 1):
        s = line.strip()
        if s.startswith("COPY") and ".env" in s:
            problems.append((i, "COPY of .env bakes secrets into a layer"))
        if s.startswith(("ENV", "ARG")) and SECRET_RX.search(s):
            problems.append((i, "secret literal in ENV/ARG (visible in history)"))
    for k, v in env.items():
        if SECRET_RX.search(f"{k}={v}"):
            problems.append((0, f"secret-shaped runtime env: {k}"))
    return problems

df = ["FROM python:3.12-slim", "COPY .env /app/.env",
      "ENV API_TOKEN=token=sk-live-abc123", 'CMD ["python","app.py"]']
for p_ in scan_layers(df, {"DB_PASSWORD": "password=hunter2"}):
    print("LEAK", p_)

LEAK (2, 'COPY of .env bakes secrets into a layer')
LEAK (3, 'secret literal in ENV/ARG (visible in history)')
LEAK (0, 'secret-shaped runtime env: DB_PASSWORD')

Knowledge check check yourself

FROM python:3.12-slim
RUN useradd --uid 10001 --create-home --shell /usr/sbin/nologin appuser
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY --chown=10001:10001 . .
USER 10001
CMD ["python", "app.py"]

docker run --rm \
  --read-only \
  --tmpfs /tmp:rw,noexec,nosuid,size=64m \
  --cap-drop=ALL \
  --security-opt no-new-privileges \
  --user 10001 \
  myapp:1.4.2

def runtime_risk(cfg):
    score = 0
    if cfg["user"] == "root":  score += 50
    if cfg["writable_rootfs"]: score += 20
    score += 5 * len(cfg["caps"])
    if cfg["new_privileges"]: score += 15
    return score

default = dict(user="root", writable_rootfs=True,
               caps=["CHOWN","SETUID","NET_RAW"], new_privileges=True)
hardened = dict(user="10001", writable_rootfs=False, caps=[], new_privileges=False)
print(runtime_risk(default), runtime_risk(hardened))

105 0

def scan_gate(findings, allowlist=()):
    blocking = []
    for f in findings:
        if f["severity"] in ("HIGH","CRITICAL") and f["id"] not in allowlist:
            reason = "no fix yet" if not f["fixed_in"] else f"fix: {f['fixed_in']}"
            blocking.append((f["id"], f["severity"], f["pkg"], reason))
    return ("FAIL" if blocking else "PASS"), blocking

cosign sign myrepo/myapp@sha256:<digest>

cosign verify myrepo/myapp@sha256:<digest> \
  --certificate-identity-regexp 'https://github.com/acme/.*' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com'

TRUSTED_ISSUER = "https://token.actions.githubusercontent.com"
TRUSTED_IDENTITY = "https://github.com/acme/"

def admit(image):
    if "@sha256:" not in image["ref"]:
        return False, "deploying a mutable tag, not a digest"
    if not image["signed"]:
        return False, "no signature found"
    if image["oidc_issuer"] != TRUSTED_ISSUER:
        return False, f"untrusted issuer {image['oidc_issuer']}"
    if not image["signer_identity"].startswith(TRUSTED_IDENTITY):
        return False, f"untrusted signer {image['signer_identity']}"
    return True, "admitted"