Security & supply chain
The layer security review blocks on: run as non-root with a read-only rootfs, ship a distroless base, drop Linux capabilities, scan images and generate an SBOM, then sign and verify what you ship — the container supply-chain threat model, end to end.
Learning objectives
- Run containers as a non-root user with a read-only root filesystem and dropped capabilities.
- Choose a minimal or distroless base and explain the attack-surface trade-off.
- Scan images for CVEs (Trivy/Grype) and generate an SBOM (syft) as a CI gate.
- Sign images and verify signatures with cosign/sigstore in an admission-style check.
- Keep secrets out of layers and env, and reason about the full supply-chain threat model.
1 · The container supply-chain threat model advanced
Security review does not ask "is the code safe?" — it asks "can I trust every byte in the image, and how much damage can it do if it runs hostile?" Those are two separate axes: provenance (where did the image and its dependencies come from) and blast radius (what can the running container touch). Every control in this lesson maps to one of them.
| Stage | Threat | Control in this lesson |
|---|---|---|
| Source | Malicious commit / typosquatted image name | Pin bases by digest; review |
| Dependencies | Compromised transitive package | SBOM + vuln scan gate |
| Build | Poisoned CI leaks secrets or injects code | No secrets in layers; hermetic build |
| Registry | Tag re-pointed to a different image | Sign on push, verify by digest on pull |
| Runtime | Container escape / lateral movement | Non-root, read-only FS, drop caps |
2 · Least privilege at runtime — non-root, read-only, drop caps advanced
The default container runs as root (UID 0) with a writable filesystem and a bundle of Linux capabilities. That is the single most common finding in a review. Fix all three: a numeric non-root USER, a read-only root filesystem with explicit writable tmpfs mounts, and every capability dropped.
Dockerfile# Create an unprivileged user at build time and run as its NUMERIC id.
# A numeric USER lets the orchestrator enforce runAsNonRoot (it can't resolve names).
FROM python:3.12-slim
RUN useradd --uid 10001 --create-home --shell /usr/sbin/nologin appuser
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY --chown=10001:10001 . .
USER 10001
CMD ["python", "app.py"]
run-hardened.sh# Read-only root FS: app can't write anywhere except the tmpfs you allow.
# --cap-drop=ALL removes every Linux capability; add back only what you truly need.
docker run --rm \
--read-only \
--tmpfs /tmp:rw,noexec,nosuid,size=64m \
--cap-drop=ALL \
--security-opt no-new-privileges \
--user 10001 \
myapp:1.4.2
# If the app truly needs to bind port 80 as non-root, add the ONE cap:
# --cap-drop=ALL --cap-add=NET_BIND_SERVICE
# Verify flag names/behavior against current Docker docs — they evolve.
--read-only. The fix is not to drop the flag — it is to mount a small tmpfs at exactly the paths that need writing (e.g. /tmp) and keep the rest immutable.runtime_risk.pydef runtime_risk(cfg):
"""Higher score = more blast radius if the container is compromised."""
score = 0
if cfg["user"] == "root": score += 50
if cfg["writable_rootfs"]: score += 20
score += 5 * len(cfg["caps"]) # each retained capability adds risk
if cfg["new_privileges"]: score += 15 # setuid escalation possible
return score
default = dict(user="root", writable_rootfs=True,
caps=["CHOWN","SETUID","SETGID","NET_RAW","MKNOD","..."], new_privileges=True)
hardened = dict(user="10001", writable_rootfs=False,
caps=[], new_privileges=False)
print("default runtime risk :", runtime_risk(default))
print("hardened runtime risk:", runtime_risk(hardened))
default runtime risk : 115
hardened runtime risk: 0
3 · Minimal & distroless bases — shrink the attack surface advanced
Every binary in the image is something an attacker can use — a shell, a package manager, curl, even ls. Distroless images ship only your app and its runtime: no shell, no package manager, nothing to pivot with. Fewer packages also means fewer CVEs to patch. The cost is debuggability — you cannot docker exec … sh into a distroless container (covered in CD8).
| Base | Shell / pkg mgr | Typical CVEs | Debug | Use when |
|---|---|---|---|---|
| python:3.12 | yes | many | easy | never in prod |
| python:3.12-slim | yes | moderate | easy | sensible default |
| -alpine | yes (busybox) | few | ok | musl-safe wheels only |
| distroless/python3 | none | fewest | hard | hardened prod |
Dockerfile.distroless# Build with a full toolchain, then ship into distroless (no shell, no pkg mgr).
FROM python:3.12 AS build
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --target=/deps -r requirements.txt
# gcr.io/distroless/python3-debian12 ships CPython + libs, nothing else.
# Verify the exact tag/name against current distroless docs (they update per-Debian).
FROM gcr.io/distroless/python3-debian12
WORKDIR /app
COPY --from=build /deps /app/deps
COPY . .
ENV PYTHONPATH=/app/deps
USER 65532 # 'nonroot' user baked into distroless images
ENTRYPOINT ["python", "app.py"]
4 · Vulnerability scanning & SBOM as a CI gate professional
A vulnerability scan matches the packages in your image against CVE databases. An SBOM (Software Bill of Materials) is the machine-readable list of exactly what is in the image — so when the next Log4Shell drops, you can answer "are we affected?" in seconds instead of days. Generate the SBOM once, scan it, gate on it.
scan.sh# Scan a built image for OS + language CVEs. Fail CI on High/Critical only.
trivy image --severity HIGH,CRITICAL --exit-code 1 myapp:1.4.2
# Grype does the same job (different DB); many teams run both:
grype myapp:1.4.2 --fail-on high
# Generate an SBOM in the standard SPDX or CycloneDX format:
syft myapp:1.4.2 -o spdx-json > sbom.spdx.json
# Then scan the SBOM itself (fast, no re-pull) and attach it to the image later:
grype sbom:sbom.spdx.json --fail-on critical
# Flag names (--exit-code, --fail-on, -o) drift between versions — verify current docs.
scan_gate.pydef scan_gate(findings, allowlist=()):
"""findings: [{"id","severity","pkg","fixed_in"}]. Block on unfixed High/Critical
that are not explicitly, temporarily allowlisted (with an expiry, in real life)."""
blocking = []
for f in findings:
if f["severity"] in ("HIGH", "CRITICAL") and f["id"] not in allowlist:
reason = "no fix yet" if not f["fixed_in"] else f"fix: {f['fixed_in']}"
blocking.append((f["id"], f["severity"], f["pkg"], reason))
return ("FAIL" if blocking else "PASS"), blocking
findings = [
{"id": "CVE-2024-1111", "severity": "CRITICAL", "pkg": "libxyz", "fixed_in": "1.2.4"},
{"id": "CVE-2024-2222", "severity": "LOW", "pkg": "zlib", "fixed_in": ""},
{"id": "CVE-2023-9999", "severity": "HIGH", "pkg": "openssl","fixed_in": "3.0.14"},
]
verdict, blocking = scan_gate(findings, allowlist={"CVE-2023-9999"}) # risk-accepted, tracked
print(verdict)
for b in blocking: print(" BLOCK", *b)
FAIL
BLOCK CVE-2024-1111 CRITICAL libxyz fix: 1.2.4
5 · Signing & verifying images — cosign / sigstore professional
A scan proves the image was clean when you built it. A signature proves the image you are about to run is the same one you built and has not been swapped at the registry. cosign (part of sigstore) signs an image by its digest and stores the signature next to it; keyless signing binds the signature to a workload identity (an OIDC token from your CI) instead of a long-lived private key.
cosign.sh# Keyless signing: cosign gets a short-lived cert from Fulcio bound to the CI's
# OIDC identity, logs the signature to the Rekor transparency log. No key to leak.
cosign sign myrepo/myapp@sha256:<digest> # sign the DIGEST, never a mutable tag
# On deploy, verify BOTH the identity that signed it and where the token came from:
cosign verify myrepo/myapp@sha256:<digest> \
--certificate-identity-regexp 'https://github.com/acme/.*' \
--certificate-oidc-issuer 'https://token.actions.githubusercontent.com'
# Attach the SBOM from section 4 as an attestation, then verify it too:
cosign attest --predicate sbom.spdx.json --type spdxjson myrepo/myapp@sha256:<digest>
# Command surface changes across cosign releases — verify against current sigstore docs.
:latest or :1.4.2 are mutable — anyone with push rights can re-point them. A signature over a mutable tag proves nothing. Always sign and verify the immutable @sha256:… digest, and deploy by digest.admission.py# Models what an admission controller / deploy gate does: trust nothing unverified.
TRUSTED_ISSUER = "https://token.actions.githubusercontent.com"
TRUSTED_IDENTITY = "https://github.com/acme/"
def admit(image):
if "@sha256:" not in image["ref"]:
return False, "deploying a mutable tag, not a digest"
if not image["signed"]:
return False, "no signature found"
if image["oidc_issuer"] != TRUSTED_ISSUER:
return False, f"untrusted issuer {image['oidc_issuer']}"
if not image["signer_identity"].startswith(TRUSTED_IDENTITY):
return False, f"untrusted signer {image['signer_identity']}"
return True, "admitted"
good = dict(ref="myapp@sha256:abcd", signed=True, oidc_issuer=TRUSTED_ISSUER,
signer_identity="https://github.com/acme/build")
tag = dict(ref="myapp:latest", signed=True, oidc_issuer=TRUSTED_ISSUER,
signer_identity="https://github.com/acme/build")
evil = dict(ref="myapp@sha256:beef", signed=True, oidc_issuer=TRUSTED_ISSUER,
signer_identity="https://github.com/attacker/x")
for img in (good, tag, evil):
print(admit(img))
(True, 'admitted')
(False, 'deploying a mutable tag, not a digest')
(False, 'untrusted signer https://github.com/attacker/x')
6 · Secrets — never in layers, never in env tech-lead
A secret COPY'd into a layer or baked with ENV is committed forever and ships to every registry that pulls the image — a later rm or override does not remove it from the layer history. Even build-time secrets leak if you use a plain ARG. Use BuildKit secret mounts at build time and inject real secrets at runtime from a secrets manager.
Dockerfile.secret# syntax=docker/dockerfile:1
# The secret is mounted only for THIS run and never lands in a layer or history.
FROM python:3.12-slim
RUN --mount=type=secret,id=pip_token \
PIP_TOKEN="$(cat /run/secrets/pip_token)" \
pip install --no-cache-dir -r requirements.txt
# Build with:
# DOCKER_BUILDKIT=1 docker build --secret id=pip_token,src=./pip_token.txt -t myapp .
# Do NOT use ARG for secrets: ARG values are visible in `docker history`.
secret_lint.pyimport re
SECRET_RX = re.compile(r"(AKIA[0-9A-Z]{16}|-----BEGIN|password\s*=|token\s*=)", re.I)
def scan_layers(dockerfile_lines, env):
problems = []
for i, line in enumerate(dockerfile_lines, 1):
s = line.strip()
if s.startswith("COPY") and ".env" in s:
problems.append((i, "COPY of .env bakes secrets into a layer"))
if s.startswith(("ENV", "ARG")) and SECRET_RX.search(s):
problems.append((i, "secret literal in ENV/ARG (visible in history)"))
for k, v in env.items():
if SECRET_RX.search(f"{k}={v}"):
problems.append((0, f"secret-shaped runtime env: {k}"))
return problems
df = ["FROM python:3.12-slim", "COPY .env /app/.env",
"ENV API_TOKEN=token=sk-live-abc123", 'CMD ["python","app.py"]']
for p_ in scan_layers(df, {"DB_PASSWORD": "password=hunter2"}):
print("LEAK", p_)
LEAK (2, 'COPY of .env bakes secrets into a layer')
LEAK (3, 'secret literal in ENV/ARG (visible in history)')
LEAK (0, 'secret-shaped runtime env: DB_PASSWORD')
ENV values show up in docker inspect, crash dumps, and child process listings. Mount secrets as files from a secrets manager (Vault, AWS/GCP secret manager, or the orchestrator's secret volume) and read them at startup.✓ Checkpoint — you can move on when you can…
- Run a container non-root, read-only, with all capabilities dropped.
- Justify a distroless base and name the debuggability trade-off.
- Gate CI on a Trivy/Grype scan and produce an SBOM with Syft.
- Sign an image by digest and verify signer identity + OIDC issuer.
- Point to every place a secret can leak and state the runtime-injection fix.
Knowledge check check yourself
You verify a cosign signature over the tag myapp:1.4.2 and it passes. Why is this still insecure, and what should you verify instead?
Show answer
1.4.2 to a different image after you signed it, and a signature over a mutable tag proves nothing about what actually runs. Sign and verify the immutable @sha256: digest, and deploy by digest so the bytes that were verified are exactly the bytes that run.A scan of your image passes with zero High/Critical CVEs. A teammate says "we're secure, ship it." Name two independent risks a clean scan does not address.
Show answer
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: The most common security-review finding is a container running as root. Fixing it is a two-line change most teams still skip.
Your task: Modify a Dockerfile so the app runs as a numeric non-root user, and explain why a numeric UID matters to the orchestrator.
Requirements:
- Create an unprivileged user at build time (e.g. UID 10001)
- Set
USERto the numeric id, not the name - Give that user ownership of the app with
COPY --chown - Explain that a numeric UID lets the platform enforce
runAsNonRoot
💡 Hint: The orchestrator cannot resolve a username to a UID at admission time, so a named USER can silently still be root.
Show solution
Add a user at build time and switch to its numeric id; own the code with --chown.
FROM python:3.12-slim
RUN useradd --uid 10001 --create-home --shell /usr/sbin/nologin appuser
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY --chown=10001:10001 . .
USER 10001
CMD ["python", "app.py"]A numeric USER 10001 lets Kubernetes/Docker enforce runAsNonRoot; a named user cannot be resolved to a UID at admission, so it may still be root.
Context: Even a non-root container does damage if its filesystem is writable and it keeps every Linux capability. Least privilege at runtime shrinks the blast radius.
Your task: Write the docker run command that runs the image read-only, drops all capabilities, blocks privilege escalation, and still lets an app write to /tmp.
Requirements:
- Use
--read-onlyfor an immutable root filesystem - Mount a small
tmpfsat/tmpfor scratch writes - Drop all capabilities with
--cap-drop=ALL - Add
--security-opt no-new-privilegesand a non-root--user
💡 Hint: Do not drop --read-only when the app needs to write — mount tmpfs at exactly the path that needs writing.
Show solution
Immutable root FS plus a scratch tmpfs, no caps, no escalation, non-root user:
docker run --rm \
--read-only \
--tmpfs /tmp:rw,noexec,nosuid,size=64m \
--cap-drop=ALL \
--security-opt no-new-privileges \
--user 10001 \
myapp:1.4.2If the app must bind a low port, add back the single capability --cap-add=NET_BIND_SERVICE rather than restoring them all. Verify flags against current Docker docs.
Context: Reviews argue about 'how risky is this container' subjectively. A model turns the runtime config into a number you can compare and gate on.
Your task: Write runtime_risk(cfg) that scores blast radius: +50 for root, +20 for a writable rootfs, +5 per retained capability, +15 if new privileges are allowed.
Requirements:
- Take a config with
user,writable_rootfs,caps,new_privileges - Apply the weighted penalties above
- Show a default (root, writable, many caps) versus a hardened config
- Confirm the hardened config scores 0
💡 Hint: A pure function over a dict — no Docker needed; this is the offline model of what a policy engine enforces.
Show solution
def runtime_risk(cfg):
score = 0
if cfg["user"] == "root": score += 50
if cfg["writable_rootfs"]: score += 20
score += 5 * len(cfg["caps"])
if cfg["new_privileges"]: score += 15
return score
default = dict(user="root", writable_rootfs=True,
caps=["CHOWN","SETUID","NET_RAW"], new_privileges=True)
hardened = dict(user="10001", writable_rootfs=False, caps=[], new_privileges=False)
print(runtime_risk(default), runtime_risk(hardened))105 0The hardened config scores 0 — every point of risk maps to a specific control you removed.
Context: CI must block on serious CVEs, but occasionally you ship with a known, unreachable High CVE. The gate has to support risk-accepted exceptions without silently muting them.
Your task: Write scan_gate(findings, allowlist) that blocks on unfixed High/Critical CVEs unless they are explicitly allowlisted, returning the blocking findings.
Requirements:
- Block only on
HIGH/CRITICALseverities - Skip an id that is in the allowlist (a tracked, risk-accepted exception)
- Report whether a fix exists (
fixed_in) in the reason - Return
PASS/FAILplus the list of blocking findings
💡 Hint: An allowlist entry is a promise with an owner and expiry — model it as a set of ids here, but never delete the finding itself.
Show solution
def scan_gate(findings, allowlist=()):
blocking = []
for f in findings:
if f["severity"] in ("HIGH","CRITICAL") and f["id"] not in allowlist:
reason = "no fix yet" if not f["fixed_in"] else f"fix: {f['fixed_in']}"
blocking.append((f["id"], f["severity"], f["pkg"], reason))
return ("FAIL" if blocking else "PASS"), blockingAn allowlisted id is skipped so a risk-accepted, tracked CVE does not block the pipeline — but it stays in the report with an owner and expiry so it resurfaces, rather than being deleted.
Context: A clean scan proves the image was clean at build; a signature proves the image you run is the one you built. Keyless signing removes the long-lived private key that always leaks.
Your task: Write the cosign commands to keyless-sign an image by digest and verify the signer identity and OIDC issuer on deploy, and explain why signing a tag is worthless.
Requirements:
- Sign the immutable
@sha256:digest, not a tag - Verify with
--certificate-identity-regexpand--certificate-oidc-issuer - Explain keyless: a short-lived cert bound to the CI's OIDC identity, logged to Rekor
- State why a signature over a mutable tag proves nothing
💡 Hint: The registry tag is the mutable pointer; the digest is the content address. Verify the thing that cannot change.
Show solution
cosign sign myrepo/myapp@sha256:<digest>
cosign verify myrepo/myapp@sha256:<digest> \
--certificate-identity-regexp 'https://github.com/acme/.*' \
--certificate-oidc-issuer 'https://token.actions.githubusercontent.com'Keyless signing gets a short-lived Fulcio cert bound to the CI's OIDC token and records the signature in the Rekor transparency log — no private key to leak. A signature over a mutable tag is worthless because the tag can be re-pointed after signing; only the digest is immutable. Verify commands against current sigstore docs.
Context: In a regulated org, the deploy step is the last line of defense. It must refuse anything that is not a signed, verified, immutable image from a trusted builder — no exceptions, no humans in the loop.
Your task: Write an admit(image) gate that only admits images referenced by digest, signed, from a trusted OIDC issuer and a trusted signer identity; reject everything else with a reason.
Requirements:
- Reject any reference that is a tag, not a
@sha256:digest - Reject unsigned images
- Reject an untrusted OIDC issuer
- Reject a signer identity outside the trusted org prefix
- Return an admit/deny decision with a human-readable reason for each case
💡 Hint: This mirrors a Kubernetes admission controller / policy engine — trust nothing that is not both verified and immutable.
Show solution
TRUSTED_ISSUER = "https://token.actions.githubusercontent.com"
TRUSTED_IDENTITY = "https://github.com/acme/"
def admit(image):
if "@sha256:" not in image["ref"]:
return False, "deploying a mutable tag, not a digest"
if not image["signed"]:
return False, "no signature found"
if image["oidc_issuer"] != TRUSTED_ISSUER:
return False, f"untrusted issuer {image['oidc_issuer']}"
if not image["signer_identity"].startswith(TRUSTED_IDENTITY):
return False, f"untrusted signer {image['signer_identity']}"
return True, "admitted"This is the offline model of a Kubernetes admission controller (e.g. a policy engine calling cosign verify): a tag reference, a missing signature, a foreign issuer, or a signer outside the org are each denied with a specific reason — so only a signed, verified, immutable image from a trusted builder ever reaches the cluster.