Containerize, ship & operate an app

1 · The app + production Dockerfile essential

Here's the failure this whole project exists to prevent. A team deploys a shiny new service. The container starts, the process is running, the orchestrator sees a live PID and starts sending it real user traffic — while behind the scenes the database connection never came up. Every request 500s. The dashboard is green because "the process is alive", but customers are getting errors. The root cause is almost always the same: the app told the platform it was fine when it wasn't.

So we start where every reliable deploy starts: an app that can honestly answer one question — "am I actually able to serve traffic right now?" Think of it like a pilot's pre-flight checklist versus just "the engine turned on". The engine running is necessary but nowhere near sufficient; you check fuel, controls, and instruments before you're allowed to taxi. A container's health check is that checklist, and the precise point is this: health must reflect real dependency reachability, not process liveness. If the DB or cache is down, /healthz should return 503 ("I'm degraded, don't route to me"), not 200. That single distinction is what lets every later gate — Compose ordering, canary, rollback — do its job, because they all read that signal to decide whether traffic is safe.

The production Dockerfile is the other half of this foundation, and the common mistake is treating it as an afterthought: a bloated single-stage image running as root, with secrets baked in and no HEALTHCHECK. A production image is small (multi-stage, slim base), runs as a non-root user, pins its base and dependency versions so the build is reproducible, and declares a HEALTHCHECK that hits /healthz so the runtime can detect a sick container on its own. Get these two things right — an honest health endpoint and a disciplined image — and everything downstream becomes possible; get them wrong and no amount of fancy pipeline machinery will save you.

app.pydef handle(path, db_ok=True, cache_ok=True):
    if path == "/":        return 200, {"service": "orders", "status": "ok"}
    if path == "/healthz":
        healthy = db_ok and cache_ok
        return (200 if healthy else 503,
                {"status": "healthy" if healthy else "degraded",
                 "checks": {"db": db_ok, "cache": cache_ok}})
    return 404, {"error": "not found"}

for p in ["/", "/healthz", "/missing"]:
    print(p, "->", handle(p))
print("db down ->", handle("/healthz", db_ok=False))

/ -> (200, {'service': 'orders', 'status': 'ok'})
/healthz -> (200, {'status': 'healthy', 'checks': {'db': True, 'cache': True}})
/missing -> (404, {'error': 'not found'})
db down -> (503, {'status': 'degraded', 'checks': {'db': False, 'cache': True}})

2 · Local stack with Compose intermediate

You've almost certainly hit this exact flake. You run the app locally, it crash-loops for ten seconds, then works. Or worse: it works on your machine and crash-loops in CI. The logs show a database connection error at startup, but by the time you look, the database is up and everything's fine. It's maddening precisely because it's a timing bug — the app booted faster than Postgres finished initializing, fired its first query at a database that wasn't accepting connections yet, and died.

The real app isn't one process — it's an API and a database (and usually a cache), and those pieces have to come up in the right order. Docker Compose is how you describe that whole cast in one file and bring it up with a single command, so "clone the repo and run it" actually works for the next person. The analogy is a restaurant opening for the day: the kitchen has to be genuinely ready — ovens hot, prep done — before the front of house starts seating customers. "The lights are on" isn't the signal to open; "the kitchen reports ready" is.

The precise mechanism that fixes the flake is depends_on with condition: service_healthy. The subtle-but-critical point most people miss: waiting for a dependency to have started (service_started) is not the same as waiting for it to be healthy. Compose can only know the difference if the dependency defines a healthcheck — for Postgres, pg_isready — that flips it to "healthy" once it truly accepts connections. Gate on health, not existence, and the race disappears. As a bonus, giving the database a named volume means its data survives restarts, so you're not re-seeding an empty database every time you bounce the stack.

docker-compose.ymlservices:
  api:
    build: .
    ports: ["8000:8000"]
    depends_on: { db: { condition: service_healthy } }
  db:
    image: postgres:16
    environment: { POSTGRES_USER: app, POSTGRES_PASSWORD: secret, POSTGRES_DB: app }
    volumes: ["dbdata:/var/lib/postgresql/data"]
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app"]
      interval: 5s
      retries: 5
volumes: { dbdata: {} }

3 · The full CI/CD pipeline advanced

Every team that ships by hand eventually ships the outage. Someone builds on their laptop, skips the tests "just this once", pushes the image, and a bug that a test would have caught goes straight to production on a Friday afternoon. The problem isn't that engineers are careless — it's that manual, order-dependent, easy-to-skip steps will get skipped under pressure. The fix is to take the careful sequence a good engineer follows and make it automatic and unskippable.

That sequence is a gate chain: test → scan → build → deploy, run on every push, where any failure halts everything downstream. Think of it as a factory assembly line with inspection stations — a part that fails inspection at station one never reaches station two. The ordering isn't arbitrary either; it follows a "cheap checks first" principle. Running the fast unit tests before the slow container build means you fail in seconds on a typo instead of wasting minutes building an image you're about to throw away.

The precise wiring that enforces this in GitHub Actions is the needs: dependency between jobs — ship declares needs: test, so it physically cannot run until tests are green. That's what turns "we should test before deploying" from a good intention into a rule the machine enforces. Two more habits make the pipeline trustworthy: a scan gate (fail the build on HIGH/CRITICAL vulnerabilities, e.g. with Trivy) so known-vulnerable images can't ship, and tagging every image with the commit SHA rather than a floating latest. The SHA tag is what makes a deploy traceable to an exact line of code — and, crucially, makes rollback nothing more than re-pointing to an older, already-built tag.

deploy.ymlname: CI/CD
on: { push: { branches: [main] } }
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r requirements.txt pytest
      - run: pytest -q
  ship:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: |
          IMG=ghcr.io/${{ github.repository }}:${{ github.sha }}
          docker build -t "$IMG" .
          # trivy image --exit-code 1 --severity HIGH,CRITICAL "$IMG"   # scan gate
          docker push "$IMG"
      - run: echo "canary deploy the SHA-tagged image with health + rollback"

4 · Professional — readiness + canary before prod professional

A green pipeline is not a promise that the release is good. Tests can pass and the build can be clean, and the release can still be broken — a config mismatch, a dependency that behaves differently under real load, a subtle regression no test covered. The painful version of this is the release that looks perfect in CI, ships to 100% of users at once, and takes the whole service down before anyone can react. The blast radius is the entire user base, instantly.

Professionals add two more checkpoints that a normal pipeline doesn't have, and they sit on either side of the traffic boundary. The first is a readiness gate: the last automated check before any real user sees the release — are tests green, is the image scan score high enough, does a healthcheck exist, and are there no secrets baked into the image? If any of those fail, the build is blocked with a specific reason. The second is a canary: the first check with real traffic, but deliberately limited. Instead of flipping everyone over at once, you route a small slice — say 5% — to the new version, watch the error rate, and only ramp to 50% then 100% if it stays healthy.

The analogy is a restaurant testing a new dish on a few tables before putting it on every plate: if those tables send it back, you pull it — you haven't ruined the whole night's service. The load-bearing detail is auto-abort: the canary compares each stage's error rate against a threshold and rolls back automatically the instant it's exceeded, rather than waiting for a human to notice a dashboard. The common mistake is a "canary" that ramps traffic but has no automated abort — that's just a slow-motion full deploy. Together, the readiness gate blocks known-bad artifacts and the canary catches the ones that only misbehave under real traffic, turning a scary all-or-nothing launch into a routine, low-risk event.

gate.pydef ready(tests, image_score, health, secrets_clean):
    problems=[]
    if not tests: problems.append("tests red")
    if image_score<70: problems.append("image score low")
    if not health: problems.append("no healthcheck")
    if not secrets_clean: problems.append("secrets in image")
    return (not problems), problems

def canary(err_by_stage, threshold=0.02):
    for pct, err in err_by_stage:
        if err>threshold: return f"aborted at {pct}% (err {err:.0%})"
    return "promoted to 100%"

ok, why = ready(True, 92, True, True)
print("ready:", ok, why)
print("healthy release:", canary([(5,0.004),(50,0.006),(100,0.005)]))
print("bad release:    ", canary([(5,0.004),(50,0.09)]))

ready: True []
healthy release: promoted to 100%
bad release:     aborted at 50% (err 9%)

5 · Tech-lead — go live & own delivery tech-lead

The gap between "we deployed it" and "we own it" is where most teams quietly leak reliability. A junior engineer's job ends when the green checkmark appears. A tech lead's job is the opposite: it starts there. The failure mode this section guards against is the team that can ship but can't recover — no rehearsed rollback, no agreed definition of "too many errors", so every incident becomes an improvised, high-adrenaline scramble at 2am while everyone argues about whether to roll back.

Owning delivery means two things become objective instead of debated. The first is that rollback is a rehearsed, one-command operation — because you tagged images by SHA (§3), reverting is just re-pointing to the last-good tag, measured in seconds, not a panicked rebuild. A rollback you've never practiced is a rollback you don't actually have. The second is the SLO error budget: you agree up front on a reliability target (say 99.9%), which implies a budget of allowable errors over a window. When that budget is being burned too fast, "should we ship this risky change today?" stops being a matter of opinion and becomes a number — and when the budget is exhausted, deploys freeze until reliability recovers.

Above the individual release sits the scoreboard a tech lead is accountable for: the four DORA metrics — deploy frequency, lead time for changes, change-failure rate, and time to restore (MTTR). These turn the vague question "are we shipping well?" into four numbers you can track and deliberately move. The mindset to internalize is that these metrics reinforce each other rather than trade off: smaller, more frequent batches raise deploy frequency and lower change-failure rate (less risk per deploy), while rehearsed rollback cuts MTTR. The tech-lead skill isn't just reading the scoreboard — it's picking the weakest metric, making a deliberate change, and proving the improvement with the numbers.

Ship it, own it

1. Deploy the SHA-tagged image with /healthz wired (CD4).
2. Confirm the public URL responds and health is green.
3. Roll out by canary; abort automatically on error spike.
4. Watch the four signals + SLO error budget; alert on burn.
5. Rehearse rollback; document deploy + rollback in the README so anyone can operate it.
6. Track DORA (deploy freq, lead time, change-fail, restore) and improve the weakest.

deliver.pydef deliver(commit):
    stages = ["lint","test","scan","build"]
    for s in stages:
        if not commit.get(s, True):
            return f"blocked at {s}"
    return "live (canary passed)" if commit.get("canary_ok", True) else "rolled back"

print(deliver({"test": True, "scan": True, "canary_ok": True}))
print(deliver({"test": False}))                 # tests fail -> never deploys
print(deliver({"test": True, "canary_ok": False}))   # canary fails -> rollback

live (canary passed)
blocked at test
rolled back

def handle(path, db_ok=True, cache_ok=True):
    if path == "/":
        return 200, {"service": "app", "status": "ok"}
    if path == "/healthz":
        if db_ok and cache_ok:
            return 200, {"status": "healthy"}
        return 503, {"status": "unhealthy",
                     "db": db_ok, "cache": cache_ok}
    return 404, {"error": "not found"}

print(handle("/healthz"))                       # (200, {'status': 'healthy'})
print(handle("/healthz", db_ok=False))          # (503, ... db False)

# Dockerfile --- needs Docker to build ---
# FROM python:3.12-slim
# WORKDIR /app
# COPY requirements.txt .
# RUN pip install --no-cache-dir -r requirements.txt
# COPY . .
# HEALTHCHECK CMD curl -f http://localhost:8000/healthz || exit 1
# CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

# docker-compose.yml --- needs Docker Compose ---
# services:
#   db:
#     image: postgres:16
#     healthcheck:
#       test: ["CMD", "pg_isready", "-U", "app"]
#       interval: 5s
#       retries: 5
#   app:
#     build: .
#     depends_on:
#       db:
#         condition: service_healthy      # wait for DB health, not just start

# offline model of the ordering guarantee:
def can_start(app, deps_health):
    unhealthy = [d for d, ok in deps_health.items() if not ok]
    if unhealthy:
        return False, f"waiting on {unhealthy}"
    return True, "starting"

print(can_start("app", {"db": False}))   # (False, "waiting on ['db']")
print(can_start("app", {"db": True}))    # (True, 'starting')

def pipeline(stages):
    # stages: ordered list of (name, passed_bool)
    for name, passed in stages:
        if not passed:
            return {"deployed": False, "failed_at": name}
    return {"deployed": True, "failed_at": None}

print(pipeline([("test",True),("scan",True),("build",True),("deploy",True)]))
print(pipeline([("test",True),("scan",False),("build",True)]))  # stops at scan

# .github/workflows/deploy.yml --- needs GitHub Actions ---
# jobs:
#   ship:
#     steps:
#       - run: pytest -q                              # test
#       - run: trivy image --exit-code 1 $IMAGE       # scan (fail on vuln)
#       - run: docker build -t $IMAGE .               # build
#       # image tag pins the exact commit:
#       # IMAGE=ghcr.io/${{ github.repository }}:${{ github.sha }}
#       - run: ./deploy.sh $IMAGE                      # deploy

def ready(tests, image_score, health, secrets_clean, min_score=80):
    problems = []
    if not tests:            problems.append("tests failing")
    if image_score < min_score: problems.append(f"image score {image_score}")
    if not health:           problems.append("health check red")
    if not secrets_clean:    problems.append("secrets detected")
    return (len(problems) == 0, problems)

def canary(err_by_stage, threshold=0.02):
    # err_by_stage: dict {5:err,50:err,100:err} error rate per traffic %
    for pct in (5, 50, 100):
        if err_by_stage[pct] > threshold:
            return {"promoted": False, "aborted_at": f"{pct}%"}
    return {"promoted": True, "aborted_at": None}

print(ready(tests=True, image_score=92, health=True, secrets_clean=True))
print(canary({5:0.005, 50:0.03, 100:0.0}))   # aborts at 50% -> error spike

DEPLOYS = ["sha-aaa", "sha-bbb", "sha-ccc"]     # append-only history

def rollback():
    if len(DEPLOYS) < 2:
        raise RuntimeError("no previous version to roll back to")
    return DEPLOYS[-2]                          # repoint to prior good SHA

def budget_status(slo=0.999, actual_success=0.9985, window_budget=0.001):
    burned = (1 - actual_success)
    remaining = window_budget - burned
    return {"remaining_budget": round(remaining, 5),
            "freeze_deploys": remaining <= 0}

print("rollback to:", rollback())              # sha-bbb
print(budget_status(actual_success=0.9985))    # budget left, no freeze
print(budget_status(actual_success=0.997))     # budget blown -> freeze

def dora(deploys):
    # deploys: list of {lead_hours, failed(bool), restore_hours}
    n = len(deploys)
    freq_per_week = n            # over a 1-week window, say
    lead = sum(d["lead_hours"] for d in deploys)/n
    cfr = sum(d["failed"] for d in deploys)/n
    fails = [d["restore_hours"] for d in deploys if d["failed"]]
    mttr = sum(fails)/len(fails) if fails else 0.0
    tier = ("elite" if freq_per_week >= 7 and lead < 24 and cfr <= 0.15
            else "high" if cfr <= 0.30 else "needs work")
    return {"deploys/wk": freq_per_week, "lead_h": round(lead,1),
            "cfr": round(cfr,2), "mttr_h": round(mttr,1), "tier": tier}

log = [{"lead_hours":4,"failed":False,"restore_hours":0},
       {"lead_hours":6,"failed":True, "restore_hours":0.5},
       {"lead_hours":3,"failed":False,"restore_hours":0}]
print(dora(log))

Knowledge check check yourself