Images, layers & optimization

1 · Images are stacks of layers essential

Here's the moment this chapter pays for itself. You change one line of your app, run docker build, and wait two minutes while Docker re-installs every dependency from scratch — again — even though you never touched the requirements. A colleague on the same project changes a line and their build finishes in three seconds. Same code, same tools; the only difference is that they understand how an image is assembled, and you don't yet. This section fixes that, and everything else in the chapter — cache tricks, size cuts, multi-stage builds — is just consequences of the one idea below.

Picture an image the way you'd picture a stack of transparent sheets on an overhead projector. You lay down a first sheet (the operating system), then a second on top (your dependencies), then a third (your code). Each sheet only draws the changes it adds; stack them and you see the finished picture. That's a Docker image: not one solid blob, but a pile of thin, ordered, read-only layers, each one recording the filesystem change made by a single Dockerfile instruction. The image you run is simply "all these sheets, in this order, flattened."

Two properties of this design are what make Docker fast, and they're worth naming before we go on. First, layers are cached: once Docker has built a layer, it keeps the result and reuses it verbatim next time — no work repeated. Second, layers are shared: if ten images all start FROM python:3.12-slim, that base layer is stored on disk once and every image points at the same copy. This is why pulling your second Python image is nearly instant, and why a hundred images don't cost a hundred times the disk. The container you actually run then adds one more sheet on top — a thin writable layer — so your program can create temp files without ever mutating the read-only stack beneath it.

The mistake nearly everyone makes at first is treating a Dockerfile as an ordinary shell script where line order is just personal style. It isn't. Because each line is a cached layer stacked on the ones before it, the order of your instructions silently decides how much work every future rebuild does. Put the thing you edit hourly (your code) near the bottom and the thing you touch monthly (dependencies) near the top, and 95% of your rebuilds skip the slow part entirely. Get that order backwards and you pay full price every time. Hold that thought — the diagram below shows the stack, and §2 turns this ordering insight into a hard, measurable rule.

Every Dockerfile instruction creates a read-only layer; the image is those layers stacked. A running container adds a thin writable layer on top. Layers are cached and shared between images — the key to both speed and small storage.

2 · The build cache — order matters enormously essential

This is the single highest-leverage habit in all of Docker, and it costs you nothing but paying attention to line order. A team that gets it right rebuilds in seconds and iterates fast; a team that gets it wrong burns a coffee break on every trivial edit and slowly learns to dread deploying. The difference is not tooling or hardware — it's whether the slow step in their Dockerfile sits above or below the fast-changing one.

Think of the cache like a checklist you re-run top to bottom. Docker walks your instructions in order and, for each one, asks: "Is this step — and everything above it — exactly as it was last time?" As long as the answer is yes, it reuses the finished layer for free and moves on. The instant it hits a step that changed, the honeymoon is over: that layer and every single layer below it must be rebuilt, no matter how expensive. There's no skipping ahead and no reaching back — the cache invalidates from the first changed layer straight down to the end.

A kitchen analogy makes the rule stick. Imagine prepping a dish where step 2 is "simmer the stock for forty minutes" and step 5 is "sprinkle parsley on top." If you decide to change the parsley, you obviously don't re-simmer the stock — the slow work near the start is already done and untouched. But if you change the stock (an early step), everything downstream that was built on it has to be redone. Docker is exactly this literal: expensive, stable steps belong early so they stay done; cheap, volatile steps belong late so a change to them is cheap to redo.

The classic self-inflicted wound is writing COPY . . (which grabs all your source, and changes on nearly every commit) before the RUN pip install that takes forty seconds. Now every one-character edit to your code invalidates the copy layer, which drags the whole dependency install down with it — you re-install NumPy and Pandas to fix a typo. The fix is almost insultingly simple: copy only requirements.txt first, install, and copy the rest of your code last. Because your dependency list changes far less often than your code, the slow install layer stays cached across the vast majority of builds. That's the golden rule — rarely-changing things first, often-changing things last — and the lab below turns it into a stopwatch you can read.

Docker reuses a layer's cache if that instruction and everything before it are unchanged. So the golden rule: put things that change rarely (dependencies) before things that change often (your code).

cache_model.pydef rebuild_time(layers, changed):
    """layers: [(name, seconds)]. A layer rebuilds if it or anything before it changed."""
    total, broken = 0, False
    for name, secs in layers:
        if name in changed: broken = True
        if broken: total += secs
    return total

good = [("FROM",2), ("COPY reqs",1), ("RUN pip install",40), ("COPY code",1)]
bad  = [("FROM",2), ("COPY code",1), ("RUN pip install",40)]   # code before install
print("edit code, GOOD order:", rebuild_time(good, {"COPY code"}), "s")   # 1s
print("edit code, BAD order: ", rebuild_time(bad,  {"COPY code"}), "s")   # 41s
print("same edit, 40x slower rebuild just from instruction order")

edit code, GOOD order: 1 s
edit code, BAD order:  41 s
same edit, 40x slower rebuild just from instruction order

Dockerfile.cacheFROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .            # changes rarely -> this layer stays cached
RUN pip install --no-cache-dir -r requirements.txt
COPY . .                           # changes often -> only THIS layer rebuilds
CMD ["python", "app.py"]

3 · Shrinking images — base image choice intermediate

A bloated image isn't just untidy — it costs you money and minutes every single day. Picture the whole lifecycle: your CI pipeline pushes the image to a registry (upload the fat gigabyte), then every server or Kubernetes node that runs it pulls the gigabyte back down before the container can even start. Multiply that by dozens of deploys and hundreds of nodes and a careless base-image choice turns into slow rollouts, sluggish autoscaling, and a fatter cloud bill — for an app whose actual code is a few hundred kilobytes.

The intuition to carry: your image is only as small as the foundation you build it on, and the foundation is the FROM line. Think of it like renting a truck to move a single armchair. The default python:3.12 image is the full moving van — it ships an entire build toolchain, compilers, and OS utilities you'll almost never touch at runtime, close to a gigabyte before you add a line of your own. A -slim variant is the sensible hatchback: the Python you need and little else. Since everything you COPY and RUN stacks on top of that base, starting lean is the one decision that shrinks the total more than any other.

There's a second, quieter reason smaller is better, and it matters more than speed: attack surface. Every package baked into the base is code an attacker could potentially exploit and a vulnerability scanner will flag. A slim or distroless image with no shell, no package manager, and no compilers simply has less for anything to go wrong with. The table below lays out the honest trade-offs — slim is the right default for almost everyone; alpine is tiny but its musl C library occasionally breaks pre-built Python wheels and forces slow source compiles; distroless is the most locked-down but gives you no shell to exec into when you need to debug.

The mistake to avoid is optimizing the wrong thing first. Beginners often reach straight for alpine chasing the smallest number on the chart, then lose an afternoon fighting cryptic wheel-build failures — a poor trade when -slim would have gotten them 85% of the saving with zero drama. And the base image is only half the size story: the other half is what you needlessly stuff into the build, which is exactly what the .dockerignore lab below is about. Choose a lean base, keep junk out of the context, and you've handled the two biggest levers before ever reaching for the advanced tricks in §4.

Image size drives push/pull time, cold-start latency, and attack surface. The single biggest lever is the base image.

.dockerignore# .dockerignore  (like .gitignore, for the build context)
.venv/
__pycache__/
*.pyc
.git/
tests/
.env
*.md
# smaller context = faster builds + nothing secret/bulky sneaks into the image

4 · Multi-stage builds — ship only what runs advanced

Here's the frustration that multi-stage builds were invented to end. To install many Python packages you need compilers and development headers — the C toolchain that turns source into working code. But once the packages are built, you never need those tools again to run the app. Without multi-stage builds you're stuck: either you ship all those compilers to production (a fat, insecure image) or you contort your Dockerfile with fragile apt-get install ... && ... && apt-get purge gymnastics trying to delete them afterward — which, as §5 shows, doesn't even shrink the image. Multi-stage builds make the whole dilemma disappear.

The mental model is a professional kitchen versus the plate that reaches the table. The kitchen is chaos — knives, mixers, flour everywhere, pots that will never leave the building. The plate carries only the finished dish. A multi-stage build gives you exactly that separation: a first "builder" stage is your messy kitchen, stuffed with compilers and dev headers, where the real work of installing dependencies happens. A second, pristine "runtime" stage is the plate — it starts fresh from a tiny base and receives only the finished result copied across. The diners (production) never see the kitchen.

The mechanism that makes this work is one keyword and one flag. You name the first stage — FROM python:3.12 AS builder — and later, in the slim final stage, you reach back into it with COPY --from=builder /path /path to lift out just the built artifacts. Everything else in the builder — every compiler, every dev header, the hundreds of megabytes of toolchain — is simply left behind, discarded when the build finishes. The final image contains a clean base, your dependencies, and your code, and nothing that was only needed to assemble it. Smaller to push and pull, and a far smaller attack surface because there's no shell-scripting toolchain sitting in production for an intruder to abuse.

The common misconception is that multi-stage builds are an exotic, expert-only technique you'll reach for someday. In reality they're the default for any real Python service, and the pattern is short enough to memorize: build in a fat stage, copy the result into a slim stage, ship the slim stage. The one subtlety to respect is that the runtime stage is a genuine clean slate — nothing from the builder tags along unless you explicitly COPY --from it, so if you forget to copy a needed file the runtime will fail to find it. The lab below shows the pattern, and the size-estimate lab right after puts a hard percentage on the payoff.

A multi-stage build uses a fat "builder" stage (compilers, dev headers) and copies only the result into a tiny final image. Build tools never ship to production — smaller and safer.

Dockerfile.multi# Stage 1 — build (has compilers/dev tools)
FROM python:3.12 AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --target=/deps -r requirements.txt

# Stage 2 — runtime (tiny; only deps + code, no build tools)
FROM python:3.12-slim
WORKDIR /app
COPY --from=builder /deps /usr/local/lib/python3.12/site-packages
COPY . .
USER 1000                          # non-root
CMD ["python", "app.py"]

size_estimate.pydef image_size(base_mb, deps_mb, build_tools_mb, multistage):
    # multi-stage drops the build tools from the final image
    return base_mb + deps_mb + (0 if multistage else build_tools_mb)

single = image_size(150, 120, 400, multistage=False)
multi  = image_size(150, 120, 400, multistage=True)
print(f"single-stage: {single} MB")
print(f"multi-stage:  {multi} MB  ({round(100*(single-multi)/single)}% smaller)")

single-stage: 670 MB
multi-stage:  270 MB  (60% smaller)

5 · Layer inspection & caching busting advanced

Sooner or later you'll stare at a 900 MB image and have no idea where the weight came from. Your code is tiny, your requirements look reasonable, and yet the thing is enormous. Guessing wastes hours. The good news is that an image is not a black box — because it's a stack of layers and each layer records exactly what it added, you can open it up and read the size of every slice. Debugging bloat becomes a lookup, not a hunt.

The tool for this is docker history, and the right way to think about it is a receipt, itemized line by line. Just as a grocery receipt shows what each item cost so you can spot the one absurd charge, docker history shows how many megabytes each instruction added, top to bottom. When one line reads 400 MB and the rest read a few kilobytes, you've found your culprit in seconds — usually a RUN that downloaded something huge or an apt-get that pulled in half an OS. You optimize by reading the receipt and attacking the fattest line first, not by rewriting the whole Dockerfile on a hunch.

The flip side of the cache is knowing what busts it — what forces a rebuild whether you wanted one or not. A layer's cache is invalidated when the instruction text changes, when a file it copies changes, when a build argument it depends on changes, or when you explicitly pass --no-cache. And remember the invalidation cascades: bust one layer and everything below it rebuilds too. Most of the time the cache is your friend, but occasionally you need to force a clean build to prove the image assembles from scratch or to shake loose a stale dependency — that's what --no-cache is for, used deliberately rather than reflexively.

The subtle trap that catches even experienced people is assuming that if a file isn't in the final image, it isn't costing you. Layers are append-only: once bytes are committed to a layer they live there forever, and a later instruction can only hide them, never reclaim the space. So RUN wget big.tar followed by a separate RUN rm big.tar gives you an image that still carries the full download, plus a later layer that merely marks it deleted — the worst of both worlds. The cure, shown in the lab and the callout below, is to download, use, and delete a file within a single RUN … && … && rm … so the bytes are never committed to a layer in the first place.

Diagnose bloated images: docker history shows each layer's size, so you can find the instruction that added 400MB. And know the cache busters — a changed file, a new arg, or --no-cache forces a rebuild from that point.

history.shdocker history myapp --human           # size added by each layer
docker image inspect myapp --format '{{.Size}}'
docker build --no-cache -t myapp .      # force full rebuild (ignore cache)
# a RUN that downloads then deletes in SEPARATE layers still keeps the download
# in the earlier layer -> combine with && in ONE RUN to keep it out

6 · Professional — the production hardening checklist professional

An image that merely "works" and an image that's ready for production are two very different things — and the gap is where incidents come from. The image that works on your laptop happily runs as the all-powerful root user, bakes a stray API key into a layer, and gives the orchestrator no way to tell whether the app inside is actually alive. Ship that and you've handed an attacker root, leaked a credential to every registry that pulls the image, and left your platform restarting a hung container by luck rather than design. Hardening is the discipline of closing those gaps before they become a 2 a.m. page.

The mindset to adopt is that a container is a box you're handing to strangers — other teams, a CI system, a public cloud node — so you assume the worst about what happens inside it. If the process is compromised, how much damage can it do? Run it as root and the answer is "everything the host allows"; run it as an unprivileged user and a break-in is trapped in a sandbox with almost no power. This is the principle of least privilege made concrete: the app gets exactly the permissions it needs to do its job and not one more. A useradd plus a USER line is the cheapest security win in the whole chapter.

Two more practices round out the checklist, and each answers a real operational question. A health check gives the platform a heartbeat: instead of assuming a running process means a working app, Docker periodically pokes an endpoint and marks the container unhealthy — so it can be restarted or pulled from the load balancer — when the app has silently wedged. And keeping secrets out of the image is non-negotiable: a key copied in via COPY .env or written in the Dockerfile is committed to a layer permanently and travels with the image everywhere, so secrets must be injected at runtime (environment variables, mounted files, a secrets manager) and never at build time.

The mistake that undoes all of this is treating hardening as a final polish step you'll "get to later." Later never comes, and the insecure image quietly becomes the template everyone copies. The fix is to make the hardened Dockerfile your starting point, not your cleanup — which is exactly why the next section turns this checklist into an automated scorer that fails the build when any box is unchecked. The lab below is that starting point: a single file that folds in the slim base, cache-friendly order, a non-root user, ownership via --chown, and a HEALTHCHECK, so you never have to remember the list by hand.

Dockerfile.prodFROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
RUN useradd --create-home appuser         # a non-root user
USER appuser                              # never run as root in prod
COPY --chown=appuser . .
HEALTHCHECK --interval=30s --timeout=3s \
  CMD python -c "import urllib.request;urllib.request.urlopen('http://localhost:8000/healthz')"
CMD ["uvicorn","main:app","--host","0.0.0.0","--port","8000"]

7 · Tech-lead — score images & gate CI tech-lead

A checklist that lives in someone's head is a checklist that gets skipped under deadline pressure. You can write the most beautiful hardening guide, but on the Friday-afternoon deploy the tired engineer forgets the non-root user, nobody catches it in review, and the insecure image ships. Relying on human diligence to enforce standards is how standards quietly erode. The tech-lead move is to stop asking people to remember and start making the pipeline refuse anything that falls short.

The idea is to turn the checklist into a credit score for your image. Just as a lender doesn't debate your character but reads a number, CI shouldn't host an opinion-fight about whether an image is "good enough" — it should read a score and act. You start every image at a perfect 100 and subtract points for each sin: too big, runs as root, an unpinned base, no health check, and — the fatal one — a baked-in secret. Weighting matters here, and it encodes real judgment: a slightly oversized image is a minor ding, but a leaked credential is catastrophic on its own, so it alone should sink the score below any passing line.

The second half of the pattern is the gate: a hard threshold (say, score ≥ 70) that turns the number into a pass or fail the build must respect. This is the difference between a linter that warns and a gate that blocks — warnings get ignored, gates get fixed. Returning the human-readable reasons alongside the score is what makes it humane rather than tyrannical: when the build fails, the engineer sees exactly which boxes are unchecked and what to do, instead of a bare red X. Policy replaces politics, and the same standard applies to everyone including you.

The trap to sidestep is building a scorer so strict or so noisy that the team learns to route around it — copying the "skip CI" incantation until the gate is theater. A good gate is calibrated: it blocks the things that genuinely matter (secrets, root, gross bloat) and stays quiet about the rest, so a failure always means something real. The lab below is a compact, honest version of exactly this scorer — small enough to read in a minute, weighted the way a real team would weight it, and wired to a ≥ 70 gate you can drop straight into a pipeline.

Make image quality objective and enforced. A scorer turns the checklist into a CI pass/fail — no opinions, just policy.

image_score.pydef score_image(facts):
    score, notes = 100, []
    if facts["size_mb"] > 400:      score -= 25; notes.append("image > 400MB")
    if facts["runs_as_root"]:       score -= 40; notes.append("runs as root")
    if facts["has_secrets"]:        score -= 100; notes.append("secrets baked in!")
    if not facts["pinned_base"]:    score -= 20; notes.append("unpinned base")
    if not facts["has_healthcheck"]:score -= 10; notes.append("no HEALTHCHECK")
    return max(score, 0), notes

good = score_image(dict(size_mb=180,runs_as_root=False,has_secrets=False,pinned_base=True,has_healthcheck=True))
bad  = score_image(dict(size_mb=900,runs_as_root=True, has_secrets=True, pinned_base=False,has_healthcheck=False))
print("good:", good)
print("bad: ", bad)
print("CI gate (>=70):", "PASS" if good[0]>=70 else "FAIL", "/", "PASS" if bad[0]>=70 else "FAIL")

good: (100, [])
bad:  (0, ['image > 400MB', 'runs as root', 'secrets baked in!', 'unpinned base', 'no HEALTHCHECK'])
CI gate (>=70): PASS / FAIL

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .            # changes rarely -> this layer stays cached
RUN pip install --no-cache-dir -r requirements.txt
COPY . .                          # changes often -> only THIS layer rebuilds
CMD ["python", "app.py"]

# .dockerignore  (for the build context)
.venv/
__pycache__/
*.pyc
.git/
tests/
.env
*.md

def rebuild_time(layers, changed):
    """layers: [(name, seconds)]. A layer rebuilds if it or anything before it changed."""
    total, broken = 0, False
    for name, secs in layers:
        if name in changed:
            broken = True
        if broken:
            total += secs
    return total

good = [("FROM",2), ("COPY reqs",1), ("RUN pip install",40), ("COPY code",1)]
bad  = [("FROM",2), ("COPY code",1), ("RUN pip install",40)]
print("edit code, GOOD:", rebuild_time(good, {"COPY code"}), "s")   # 1s
print("edit code, BAD :", rebuild_time(bad,  {"COPY code"}), "s")   # 41s

# Stage 1 — build (has compilers/dev tools)
FROM python:3.12 AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir --target=/deps -r requirements.txt

# Stage 2 — runtime (tiny; only deps + code, no build tools)
FROM python:3.12-slim
WORKDIR /app
COPY --from=builder /deps /usr/local/lib/python3.12/site-packages
COPY . .
USER 1000                          # non-root
CMD ["python", "app.py"]

def image_size(base_mb, deps_mb, build_tools_mb, multistage):
    return base_mb + deps_mb + (0 if multistage else build_tools_mb)

single = image_size(150, 120, 400, multistage=False)
multi  = image_size(150, 120, 400, multistage=True)
print(f"single: {single} MB")
print(f"multi:  {multi} MB ({round(100*(single-multi)/single)}% smaller)")

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
RUN useradd --create-home appuser         # a non-root user
USER appuser                              # never run as root in prod
COPY --chown=appuser . .
HEALTHCHECK --interval=30s --timeout=3s \
  CMD python -c "import urllib.request;urllib.request.urlopen('http://localhost:8000/healthz')"
CMD ["uvicorn","main:app","--host","0.0.0.0","--port","8000"]

def score_image(facts):
    score, notes = 100, []
    if facts["size_mb"] > 400:       score -= 25;  notes.append("image > 400MB")
    if facts["runs_as_root"]:        score -= 40;  notes.append("runs as root")
    if facts["has_secrets"]:         score -= 100; notes.append("secrets baked in!")
    if not facts["pinned_base"]:     score -= 20;  notes.append("unpinned base")
    if not facts["has_healthcheck"]: score -= 10;  notes.append("no HEALTHCHECK")
    return max(score, 0), notes

good = score_image(dict(size_mb=180, runs_as_root=False, has_secrets=False,
                        pinned_base=True, has_healthcheck=True))
bad  = score_image(dict(size_mb=900, runs_as_root=True, has_secrets=True,
                        pinned_base=False, has_healthcheck=False))
print("good:", good)
print("bad :", bad)
gate = lambda s: "PASS" if s >= 70 else "FAIL"
print("CI gate:", gate(good[0]), "/", gate(bad[0]))

Knowledge check check yourself