Advanced builds & multi-arch

1 · BuildKit — the modern build engine advanced

Picture the moment your CI dashboard turns red for the dumbest possible reason. The tests pass, the code is fine — but the build step alone took nine minutes because it dutifully re-ran a step that hadn't changed in weeks, then did three independent things one after another when they could have happened at once. Multiply that by every push from every engineer and you've quietly burned a workday of waiting each week. The build engine — not your Dockerfile — is often the thing standing between you and a fast pipeline, and this section is about swapping in a smarter one.

Here's the mental model. The old builder is like a cook who insists on following a recipe strictly line by line: boil the pasta, then chop the garlic, then heat the oil — even though the garlic and the oil have nothing to do with the pasta and could have been done while the water boiled. It never looks ahead, never notices what's independent, and never reuses last night's prep. BuildKit is the cook who reads the whole recipe first, sees which tasks depend on which, and runs everything that can happen in parallel at the same time — while skipping any step whose ingredients are already sitting ready from before.

Concretely, BuildKit parses your entire Dockerfile up front into a DAG (a directed acyclic graph — just a dependency map of "this step needs that step's output"). From that map it does three things the legacy builder can't: it builds stages that don't depend on each other concurrently, it prunes stages nothing asks for so they never run at all, and it unlocks the mount features — cache mounts, secret mounts (CD6), and SSH mounts — that the rest of this chapter leans on.

The common mistake is assuming this is automatic and forgetting the one line that turns the advanced features on. BuildKit is the default engine in any recent Docker, so parallelism and pruning you get for free. But the mount syntax (--mount=type=cache and friends) needs the Dockerfile to declare which frontend to use with a # syntax=docker/dockerfile:1 directive on the very first line — and if it's missing, or not literally first, you get a baffling parse error instead of a helpful one. So the rule to burn in: modern Docker means BuildKit is already running, but the mount features stay locked until that syntax line is present. The legacy builder runs instructions strictly top-to-bottom on one path. BuildKit parses the whole Dockerfile into a dependency graph, so independent stages build in parallel, unused stages are skipped, and it unlocks cache mounts, secret mounts (CD6), and SSH mounts. It is the default in modern Docker; older setups enable it with an env var.

enable-buildkit.sh# Modern Docker uses BuildKit by default. If yours doesn't:
export DOCKER_BUILDKIT=1
docker build -t myapp .

# To use cache/secret mounts, the FIRST line of the Dockerfile must be a syntax directive:
#   # syntax=docker/dockerfile:1
# That line opts into the modern frontend that understands --mount=... .

2 · Cache mounts — stop re-downloading the world advanced

You bump one library — say requests from 2.31 to 2.32 — push, and watch your build re-download two hundred wheels it already had a minute ago. Nothing about NumPy or pandas changed, yet there they go over the wire again. On a slow runner that's the difference between a build you glance at and a build you go make coffee for. It feels like the tool is punishing you for a one-line change, and in a sense it is — because of exactly how layer caching works.

Here's the why, and it's the crux of this whole section. Docker's layer cache is all-or-nothing per step: the instant an input to a RUN pip install layer changes, that entire layer is declared stale and rebuilt from scratch — and everything the layer produced, including pip's private download cache of wheels, is thrown away with it. So the wheels don't survive because they lived inside the layer that just got invalidated. Think of it like a workshop where the cleaning crew empties every bin the moment you move a single tool: your finished shelf stays if you didn't touch it, but the pile of reusable offcuts you were saving is gone every time you rebuild anything.

A cache mount is a bin the cleaners aren't allowed to touch. When you write --mount=type=cache,target=/root/.cache/pip, BuildKit attaches a persistent directory to that one RUN for the duration of the step — a scratch space that lives outside every layer and survives from build to build. The layer itself still rebuilds when a dependency changes; but pip, finding its wheel cache still full, only fetches the one package that actually changed and pulls the rest straight off local disk. The same trick works for apt's .deb cache, npm's store, Go's module and build caches — anything that keeps a reusable download or compile cache benefits.

The mistake that quietly cancels the win is a habit carried over from CD2: adding --no-cache-dir to pip install to keep images small. That flag tells pip to delete its cache — the exact thing the mount exists to preserve — so with a cache mount you must drop it. And don't confuse this with the layer cache (the next callout hammers this): they're different tools solving different halves of the problem, and a fast pipeline wants both. Layer caching helps only when a layer is unchanged. But a package manager's download cache is thrown away with the layer, so a single changed dependency re-downloads everything. A cache mount (--mount=type=cache) is a persistent directory shared across builds that is not part of any layer — pip, npm, apt, and Go all get dramatically faster incremental builds.

Dockerfile.cachemount# syntax=docker/dockerfile:1
FROM python:3.12-slim
WORKDIR /app

# apt cache mount: downloaded .debs persist across builds (not baked into a layer).
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
    --mount=type=cache,target=/var/lib/apt,sharing=locked \
    apt-get update && apt-get install -y --no-install-recommends build-essential

COPY requirements.txt .
# pip cache mount: wheels persist, so changing ONE dep doesn't re-download all of them.
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

COPY . .
CMD ["python", "app.py"]
# Note: with a cache mount you can DROP --no-cache-dir — you WANT pip's cache kept.

cache_mount_model.pydef install_time(deps, changed, cache_mount):
    """Each dep costs download + build. A cache mount keeps downloads across builds."""
    dl_per, build_per = 3.0, 0.5   # seconds
    total = 0.0
    for d in deps:
        # without a cache mount, ANY change re-downloads every dep this layer
        must_download = (d in changed) or (not cache_mount and changed)
        total += (dl_per if must_download else 0.0) + build_per
    return round(total, 1)

deps = [f"pkg{i}" for i in range(20)]
changed = {"pkg7"}                       # one dependency bumped
print("no cache mount:", install_time(deps, changed, cache_mount=False), "s")
print("with cache mount:", install_time(deps, changed, cache_mount=True), "s")

no cache mount: 70.0 s
with cache mount: 13.0 s

3 · Multi-stage refinements — targets, deps stages, cache-only stages advanced

Two Dockerfiles drift apart the way twins raised in different houses do. You start with Dockerfile for prod and Dockerfile.test for CI. A month later the test image is on Python 3.11 and prod is on 3.12, the dependency versions have quietly diverged, and a bug that reproduces in CI vanishes in prod because the two environments are no longer the same environment. The whole point of containers — "it runs the same everywhere" — has leaked out through the gap between two files that were supposed to stay in sync by hand.

In CD2 you met multi-stage builds as a way to throw away build tools before shipping. The refinement here is a different, sharper use of the same feature: one Dockerfile that produces several images, all growing from a single shared trunk. Picture a tree. The deps stage is the trunk — it installs your dependencies exactly once. From that trunk grow two branches: a test image that adds pytest and your test code, and a prod image that copies only the installed deps plus your application, with none of the test tooling. Same roots, same dependency versions, no drift — because there's only ever one place that installs them.

The mechanism has two moving parts worth naming. First, a stage can build FROM another stage (FROM deps AS test), so downstream stages inherit the expensive install rather than repeating it — and because both branches share that one deps layer, BuildKit's layer cache computes it once and hands it to both. Second, --target lets you tell a single build which branch you want: docker build --target test stops at the test image, --target prod at the lean one. CI's rhythm falls out naturally — build the test target, run the suite, then build the prod target, which reuses the deps layer the test build already warmed.

The trap to avoid is reaching for a second file the moment your test and prod needs differ. That instinct feels tidy and is a slow-motion mistake: every duplicated base image, every copied dependency line is a fact you now have to remember to change in two places, and the day you forget is the day CI and prod stop agreeing. Prefer one file with named stages and targets; let the shared deps stage be the single source of truth. CD2 introduced multi-stage. Advanced use goes further: a dedicated deps stage so the dependency layer is shared by both test and runtime; named --target builds so CI can build a test image and a prod image from one file; and ordering stages so BuildKit parallelizes them.

Dockerfile.targets# syntax=docker/dockerfile:1
FROM python:3.12-slim AS deps
WORKDIR /app
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip pip install --target=/deps -r requirements.txt

FROM deps AS test                 # test image = deps + dev tools + tests
RUN --mount=type=cache,target=/root/.cache/pip pip install --target=/deps pytest
COPY . .
ENV PYTHONPATH=/deps
CMD ["pytest", "-q"]

FROM python:3.12-slim AS prod     # prod image = deps + code only, no test tooling
WORKDIR /app
COPY --from=deps /deps /deps
COPY . .
ENV PYTHONPATH=/deps
USER 10001
CMD ["python", "app.py"]

# Build a specific target:
#   docker build --target test -t myapp:test .
#   docker build --target prod -t myapp:1.4.2 .

4 · Multi-arch images with buildx (arm64 + amd64) professional

A teammate pulls your image, runs it, and gets exec format error — three words that have ruined a lot of afternoons. The image built cleanly, pushed cleanly, and runs cleanly on your machine. But it built on an amd64 CI runner, your teammate is on an Apple Silicon (arm64) laptop, and a compiled amd64 binary is as meaningless to an arm64 CPU as a cassette tape is to a CD player. It's not corrupt; it's the wrong shape for the hardware trying to read it.

This is the piece newcomers skip because for years it didn't matter — everything was amd64, so an image was just "an image." That assumption is now wrong. A CPU architecture is an instruction set: the literal vocabulary of machine operations the chip understands, and arm64 and amd64 speak different vocabularies. When Docker compiles your dependencies, it bakes them into one of those vocabularies. Apple Silicon laptops, AWS Graviton servers, and Raspberry Pis are arm64; most cloud CI and older servers are amd64. The instant your team spans both — which it almost certainly does now — a single-arch image is a coin flip that runs for half of them and errors for the rest.

The fix is to ship an image that speaks both. That's what buildx does: it builds your Dockerfile once per target platform and publishes a manifest list — think of it as a signpost living under a single tag (myrepo/myapp:1.4.2) that says "amd64 image over here, arm64 image over there." When any machine pulls that tag, its daemon reads the signpost and silently grabs the architecture that matches its own hardware. One tag, correct everywhere, zero thought required at pull time.

Two gotchas trip everyone the first time. One: a manifest list can't live in your local image store — it only makes sense in a registry — so a multi-arch build requires --push (or an explicit OCI export); a plain docker build can't hold it. Two: buildx can produce a foreign architecture on your machine by emulating the other CPU through QEMU, and it's tempting to leave it at that — but emulated compilation runs correctly and painfully slowly (often 5–20× slower). It's fine for a one-off; for CI you want native runners for each architecture and let buildx stitch their outputs into the manifest. Your laptop may be arm64 (Apple Silicon) while production is amd64 — or the reverse on Graviton. A single-arch image built on the wrong machine fails to run or runs under slow emulation. buildx builds for multiple platforms and publishes a manifest list: one tag that the daemon resolves to the right architecture automatically.

buildx-multiarch.sh# One-time: create a builder that can target multiple platforms.
docker buildx create --name multi --use
docker buildx inspect --bootstrap

# Build BOTH arches and push a manifest list under ONE tag.
# --push is required for multi-arch: a manifest list can't live in the local store.
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t myrepo/myapp:1.4.2 \
  --push .

# Confirm both arches are present under the single tag:
docker buildx imagetools inspect myrepo/myapp:1.4.2
# Native builders per-arch are far faster than QEMU emulation — prefer them in CI.

5 · Layer-cache strategy for fast CI professional

"But it's fast on my laptop!" — the sentence that opens a thousand confused Slack threads. You ordered your layers perfectly in CD2, and locally a code change rebuilds in three seconds because Docker reuses every unchanged layer. Then CI takes four minutes for the identical change, every single time, and you can't work out what's different. The difference is brutally simple and worth internalizing: your laptop remembers, and CI forgets.

Every CI build runs on a fresh, disposable runner — a brand-new machine with an empty disk that gets thrown away when the job ends. That's a feature (clean, reproducible, no leftover state), but it means the runner has no local layer cache to reuse. Your carefully ordered layers buy you nothing, because layer caching can only reuse layers it already has, and a cold runner has none. It's like a chef who does brilliant mise en place every night but works in a rented kitchen that's stripped bare each morning: the prep discipline is real, but there's nothing on the shelves to reuse when they walk in.

The fix is to give that cold runner a place to fetch yesterday's layers. BuildKit can export the layers it builds to a durable location — most commonly your registry — and import them at the start of the next build. You wire this with two flags: cache-to writes the cache after a build, cache-from reads it before one, both pointed at a type=registry reference. Now runner N pulls the layers runner N−1 published, and the fresh machine starts warm instead of cold. Add mode=max and it caches the intermediate stage layers too — the deps stage from §3, not just the final image — so multi-stage builds benefit fully.

The subtlety that catches people is that this is one of two independent levers, and you need both. Cache import (this section) gives the runner the previous layers; correct layer order (CD2 — dependencies before code) ensures a routine code edit invalidates only the cheap layers near the end. Fix only the import and a poorly ordered Dockerfile still re-runs the 60-second pip layer whenever any file changes; fix only the order and a cold runner still rebuilds everything for lack of a cache to reuse. The Python model below makes the two levers concrete so you can see exactly what breaks when either is missing. A clean CI runner has an empty local cache, so every build starts from zero unless you import a cache from somewhere durable. BuildKit can export/import cache to a registry or a CI cache backend, so runner N reuses layers built by runner N−1. Combined with a stable layer order (CD2), CI builds drop from minutes to seconds.

build.yml# .github/workflows/build.yml  (needs docker/build-push-action + a registry)
name: build
on: [push]
jobs:
  docker:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/build-push-action@v6
        with:
          push: true
          tags: myrepo/myapp:${{ github.sha }}
          # Import last build's layers and export this build's — durable across runners.
          cache-from: type=registry,ref=myrepo/myapp:buildcache
          cache-to: type=registry,ref=myrepo/myapp:buildcache,mode=max
# 'mode=max' caches intermediate stage layers too, not just the final image.
# Action versions/inputs change — verify against current docs.

ci_cache_model.pyLAYERS = [("base", 5), ("apt", 40), ("pip", 60), ("copy-code", 2), ("final", 3)]

def ci_time(changed, cache_from_registry):
    total, broken = 0, False
    for name, secs in LAYERS:
        if name in changed:
            broken = True
        if broken:                       # this layer must rebuild
            total += secs
        elif not cache_from_registry:    # clean runner, no imported cache -> rebuild anyway
            total += secs
    return total

edit = {"copy-code"}                     # a normal code change
print("cold runner, no cache import:", ci_time(edit, cache_from_registry=False), "s")
print("warm cache imported from registry:", ci_time(edit, cache_from_registry=True), "s")

cold runner, no cache import: 110 s
warm cache imported from registry: 5 s

6 · Reproducible builds & .dockerignore discipline tech-lead

A security advisory drops, you rebuild the exact commit you shipped six months ago to patch it — and get a different image. The base tag python:3.12-slim now points at a newer underlying build, one of your unpinned transitive dependencies quietly moved a minor version, and suddenly you're debugging differences between two images that were meant to be identical, on the worst possible day to be debugging anything. A build you can't reproduce is a build you can't trust, audit, or safely roll back — and that's a tech-lead-level problem, not a convenience.

The goal is stated simply: the same commit, built twice, should yield the same image. What breaks that promise is any input that can change without your commit changing — and those inputs are sneakier than they look. A mutable tag like :latest or even :3.12-slim is a moving pointer, not a fixed thing: the registry is free to re-point it at a rebuilt base tomorrow. Unpinned dependencies drift as their authors publish. Embedded build timestamps make two byte-identical builds differ. Each is a small leak, and together they mean "rebuild the release" becomes "build something new and hope."

The discipline that closes the leaks is pin everything that feeds the build. Pin the base image by its immutable digest (the @sha256:... content hash) rather than a tag — a digest is the image's fingerprint, so it can never be swapped under you, which also ties straight back to the supply-chain provenance work in CD6. Pin your dependencies with a lockfile (ideally hash-checked). Where the toolchain supports it, normalize timestamps. The image stops being "whatever the world looked like at build time" and becomes a deterministic function of your source.

The other half of this section is a mistake so common it's almost a rite of passage: the fat build context. Before Docker runs a single instruction, it packages up the entire build directory and ships it to the builder as the "context" — and if you haven't told it otherwise, that means your whole .git history, your multi-hundred-megabyte .venv, test fixtures, and — the one that ends careers — your .env file full of secrets. It's like emailing your entire hard drive when someone asked for one document: slow, and a data leak waiting for a careless COPY . . to bake it into a shipped layer. A .dockerignore is the fix; treat it like .gitignore's stricter sibling — ignore aggressively, copy explicitly, and never assume a broad COPY is safe. Two builds of the same commit should produce the same image. The enemies are unpinned inputs (:latest, unpinned deps), embedded timestamps, and a fat, drifting build context. Pin bases by digest, pin dependencies with a lockfile, and keep the context tiny — a large context is both slow to send and a vector for accidentally shipping secrets.

Dockerfile.pinned# Pin the base by immutable digest, not a mutable tag, so the build is reproducible
# and the base cannot be swapped under you (ties back to CD6 provenance).
FROM python:3.12-slim@sha256:<digest-of-the-exact-image>
# Pair with a locked requirements file (hashes) for fully pinned dependencies:
#   pip install --require-hashes -r requirements.lock
# SOURCE_DATE_EPOCH and reproducible flags can normalize timestamps — verify current support.

context_lint.pyimport fnmatch

def build_context(files, dockerignore):
    """Return files that WOULD be sent to the builder after applying .dockerignore."""
    kept = []
    for path, size in files.items():
        if any(fnmatch.fnmatch(path, pat) or path.startswith(pat.rstrip("/") + "/")
               for pat in dockerignore):
            continue
        kept.append((path, size))
    return kept

files = {".git/objects/pack": 90, ".venv/lib": 300, "tests/big_fixture.bin": 40,
         ".env": 1, "app.py": 1, "requirements.txt": 1}
ignore = [".git/", ".venv/", "tests/", ".env", "*.pyc"]
kept = build_context(files, ignore)
sent = sum(s for _, s in kept)
print("sent to builder:", sorted(p for p, _ in kept), "=", sent, "MB")
assert ".env" not in {p for p, _ in kept}, "secret leaked into context!"
print("ok: no secret, context tiny")

sent to builder: ['app.py', 'requirements.txt'] = 2 MB
ok: no secret, context tiny

Knowledge check check yourself

export DOCKER_BUILDKIT=1     # default in modern Docker

# first line of the Dockerfile:
# syntax=docker/dockerfile:1

# syntax=docker/dockerfile:1
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip \
    pip install -r requirements.txt

def install_time(deps, changed, cache_mount):
    dl_per, build_per = 3.0, 0.5
    total = 0.0
    for d in deps:
        must_download = (d in changed) or (not cache_mount and changed)
        total += (dl_per if must_download else 0.0) + build_per
    return round(total, 1)

deps = [f"pkg{i}" for i in range(20)]
print(install_time(deps, {"pkg7"}, False), install_time(deps, {"pkg7"}, True))

70.0 13.0

# syntax=docker/dockerfile:1
FROM python:3.12-slim AS deps
WORKDIR /app
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip pip install --target=/deps -r requirements.txt

FROM deps AS test
RUN --mount=type=cache,target=/root/.cache/pip pip install --target=/deps pytest
COPY . .
ENV PYTHONPATH=/deps
CMD ["pytest", "-q"]

FROM python:3.12-slim AS prod
WORKDIR /app
COPY --from=deps /deps /deps
COPY . .
ENV PYTHONPATH=/deps
USER 10001
CMD ["python", "app.py"]

docker build --target test -t myapp:test .
docker build --target prod -t myapp:1.4.2 .

docker buildx create --name multi --use
docker buildx inspect --bootstrap

docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t myrepo/myapp:1.4.2 \
  --push .

docker buildx imagetools inspect myrepo/myapp:1.4.2

# build.yml (excerpt)
- uses: docker/build-push-action@v6
  with:
    push: true
    tags: myrepo/myapp:${{ github.sha }}
    cache-from: type=registry,ref=myrepo/myapp:buildcache
    cache-to: type=registry,ref=myrepo/myapp:buildcache,mode=max

LAYERS = [("base",5),("apt",40),("pip",60),("copy-code",2),("final",3)]
def ci_time(changed, warm):
    total, broken = 0, False
    for name, secs in LAYERS:
        if name in changed: broken = True
        if broken: total += secs
        elif not warm: total += secs
    return total
print(ci_time({"copy-code"}, False), ci_time({"copy-code"}, True))

110 5