Containerize, ship & operate an app
The capstone: a FastAPI + Postgres app from code to a live, observable, reversible service — Dockerized, Compose-run, tested/scanned/built/deployed by CI/CD with readiness gates, canary, health, SLOs, rollback, and DORA ownership.
Learning objectives
- Containerize a multi-service app to the production checklist.
- Run the whole stack locally with one command.
- Automate test→scan→build→deploy with readiness + canary.
- Operate live: health, SLOs, rollback, and delivery metrics.
code/proj-cd-ship/. Python runs offline; configs are ready to use.1 · The app + production Dockerfile essential
app.pydef handle(path, db_ok=True, cache_ok=True):
if path == "/": return 200, {"service": "orders", "status": "ok"}
if path == "/healthz":
healthy = db_ok and cache_ok
return (200 if healthy else 503,
{"status": "healthy" if healthy else "degraded",
"checks": {"db": db_ok, "cache": cache_ok}})
return 404, {"error": "not found"}
for p in ["/", "/healthz", "/missing"]:
print(p, "->", handle(p))
print("db down ->", handle("/healthz", db_ok=False))
/ -> (200, {'service': 'orders', 'status': 'ok'})
/healthz -> (200, {'status': 'healthy', 'checks': {'db': True, 'cache': True}})
/missing -> (404, {'error': 'not found'})
db down -> (503, {'status': 'degraded', 'checks': {'db': False, 'cache': True}})
Before you can containerize or deploy anything, you need an app that answers requests. This tiny model of a web service takes a URL path (like / or /healthz) and returns an HTTP status code plus a little JSON reply. The star of the show is /healthz — a health check the platform will ping to decide if your app is safe to send traffic to.
def handle(path, db_ok=True, cache_ok=True):is the function.pathis which URL was requested;db_okandcache_oksay whether the database and cache are reachable (they default to healthy).if path == "/":the home route returns status200(HTTP for "OK") and a small status object.200is the code every healthy response uses.- The
/healthzroute computeshealthy = db_ok and cache_ok— it's only healthy if both dependencies are up. If healthy it returns200; if not, it returns503("Service Unavailable") so the platform stops routing to it. - Anything unrecognized falls through to
return 404— HTTP for "not found". The loop at the bottom callshandleon three paths, then calls it once more withdb_ok=Falseto show the degraded case.
What the output means: Each line prints the path and the (status, body) it returned. / and /healthz give 200; /missing gives 404; and the last line — with the DB down — flips to 503 and "degraded", exactly what a deploy system needs to see to hold back traffic.
Try this: Change the last call to handle("/healthz", cache_ok=False) and predict the output: it should also be 503, because health needs both checks to pass. This /healthz endpoint is what the Compose healthcheck and canary later rely on.
2 · Local stack with Compose intermediate
docker-compose.ymlservices:
api:
build: .
ports: ["8000:8000"]
depends_on: { db: { condition: service_healthy } }
db:
image: postgres:16
environment: { POSTGRES_USER: app, POSTGRES_PASSWORD: secret, POSTGRES_DB: app }
volumes: ["dbdata:/var/lib/postgresql/data"]
healthcheck:
test: ["CMD-SHELL", "pg_isready -U app"]
interval: 5s
retries: 5
volumes: { dbdata: {} }
A real app is more than one process — here it's an API plus a Postgres database. Docker Compose lets you describe both in one YAML file and start the whole stack with a single command (docker compose up). Indentation matters in YAML: nested lines belong to the key above them.
services:lists the containers.apiis your app:build: .means "build the image from the Dockerfile in this folder", andports: ["8000:8000"]maps port 8000 on your machine to 8000 in the container so you can reach it in a browser.depends_on: { db: { condition: service_healthy } }is the key line: the API won't start until the database reports healthy. This prevents the classic bug where the app boots faster than the DB and crashes on the first query.- The
dbservice uses the officialpostgres:16image and sets the user, password, and database name throughenvironment:variables that Postgres reads on first boot. volumes: ["dbdata:/var/lib/postgresql/data"]plus the bottomvolumes: { dbdata: {} }give the database a named volume — disk that survives restarts, so your data isn't wiped every time the container stops.- The
healthcheck:runspg_isready -U appevery 5 seconds (up to 5 retries). That command is what flips the DB to "healthy" and unblocks the API'sdepends_on.
What the output means: There's no console output here — this is configuration. Running docker compose up would start Postgres, wait until pg_isready passes, then start the API on http://localhost:8000.
Try this: Imagine deleting the depends_on block. The API could start before the DB is ready and error out. Keeping it is why one command reliably brings up the entire stack.
3 · The full CI/CD pipeline advanced
deploy.ymlname: CI/CD
on: { push: { branches: [main] } }
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: "3.12" }
- run: pip install -r requirements.txt pytest
- run: pytest -q
ship:
needs: test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: |
IMG=ghcr.io/${{ github.repository }}:${{ github.sha }}
docker build -t "$IMG" .
# trivy image --exit-code 1 --severity HIGH,CRITICAL "$IMG" # scan gate
docker push "$IMG"
- run: echo "canary deploy the SHA-tagged image with health + rollback"
This is a GitHub Actions pipeline — the automation that runs every time you push code. It does what a careful engineer would do by hand, but automatically and in order: test → build → scan → push. If any step fails, the pipeline stops and nothing ships.
on: { push: { branches: [main] } }is the trigger: this whole file runs whenever someone pushes to themainbranch.- The
testjob checks out your code, sets up Python 3.12, installs dependencies, and runspytest -q. If tests fail, the job fails — the gate that stops broken code from going further. - The
shipjob hasneeds: test, meaning it only runs after tests pass. That dependency is what enforces "never deploy red code". - Inside
ship:IMG=ghcr.io/${{ github.repository }}:${{ github.sha }}builds an image name tagged with the exact commit (github.sha). Tagging by commit means every deploy is traceable and rollback is just re-deploying an older tag. docker buildcreates the image; the commentedtrivy imageline is a security scan gate (fail on HIGH/CRITICAL vulnerabilities);docker pushuploads it to the registry. The finalechostands in for the canary deploy step.
What the output means: No local output — this runs on GitHub's servers. Green checkmarks appear on your commit when test then ship succeed; a red X means the pipeline stopped at the failing step and nothing was deployed.
Try this: Uncomment the trivy image line in your head: now a build with a critical CVE would --exit-code 1 and fail the job, blocking the docker push. That's how a scan becomes a hard gate rather than a suggestion.
4 · Professional — readiness + canary before prod professional
gate.pydef ready(tests, image_score, health, secrets_clean):
problems=[]
if not tests: problems.append("tests red")
if image_score<70: problems.append("image score low")
if not health: problems.append("no healthcheck")
if not secrets_clean: problems.append("secrets in image")
return (not problems), problems
def canary(err_by_stage, threshold=0.02):
for pct, err in err_by_stage:
if err>threshold: return f"aborted at {pct}% (err {err:.0%})"
return "promoted to 100%"
ok, why = ready(True, 92, True, True)
print("ready:", ok, why)
print("healthy release:", canary([(5,0.004),(50,0.006),(100,0.005)]))
print("bad release: ", canary([(5,0.004),(50,0.09)]))
ready: True []
healthy release: promoted to 100%
bad release: aborted at 50% (err 9%)
Before promoting a release to all users, professionals run two checks: a readiness gate (is this build allowed to deploy at all?) and a canary (roll out to a few users first, watch for errors, abort if it spikes). This code models both decisions in plain Python.
ready(tests, image_score, health, secrets_clean)collects problems into a list: tests must be green, the image scan score must be ≥70, a healthcheck must exist, and no secrets may be baked into the image. It returns(not problems), problems—Trueonly when the problem list is empty.canary(err_by_stage, threshold=0.02)walks through rollout stages like(5%, 50%, 100%). At each stage it checks the error rateerr; if it exceeds the 2% threshold it returns "aborted" immediately instead of continuing.- If every stage stays under the threshold, the loop finishes and returns
"promoted to 100%"— the release is safe for everyone. - The three calls at the bottom exercise it: a clean readiness check, a healthy release that passes every canary stage, and a bad release whose error rate jumps to 9% at the 50% stage.
What the output means: ready: True [] means no problems, so deploy is allowed. The healthy release prints promoted to 100%. The bad release prints aborted at 50% (err 9%) — the canary caught the spike and stopped the rollout before it hit everyone.
Try this: Lower image_score in the first call to 60 and re-read: ready now returns False with ["image score low"]. This is exactly how a pipeline blocks a risky build automatically.
5 · Tech-lead — go live & own delivery tech-lead
Ship it, own it
- Deploy the SHA-tagged image with
/healthzwired (CD4). - Confirm the public URL responds and health is green.
- Roll out by canary; abort automatically on error spike.
- Watch the four signals + SLO error budget; alert on burn.
- Rehearse rollback; document deploy + rollback in the README so anyone can operate it.
- Track DORA (deploy freq, lead time, change-fail, restore) and improve the weakest.
deliver.pydef deliver(commit):
stages = ["lint","test","scan","build"]
for s in stages:
if not commit.get(s, True):
return f"blocked at {s}"
return "live (canary passed)" if commit.get("canary_ok", True) else "rolled back"
print(deliver({"test": True, "scan": True, "canary_ok": True}))
print(deliver({"test": False})) # tests fail -> never deploys
print(deliver({"test": True, "canary_ok": False})) # canary fails -> rollback
live (canary passed)
blocked at test
rolled back
This ties the entire lesson together into one function: a commit flows through the delivery pipeline stage by stage, and the outcome is either live, blocked, or rolled back. It's the mental model for everything you built above.
stages = ["lint","test","scan","build"]lists the ordered gates every commit must clear before it can deploy.- The loop uses
commit.get(s, True)— it looks up each stage's result, defaulting toTrue(passing) if not mentioned. The first stage that isFalsecausesreturn f"blocked at {s}", stopping the pipeline right there. - If all gates pass, the last line checks
canary_ok: a passing canary returns"live (canary passed)"; a failing one returns"rolled back"— the safety net that reverses a bad release automatically. - The three print calls show each path: a fully passing commit, a commit whose
testisFalse, and a commit that builds fine but fails the canary.
What the output means: Three lines: live (canary passed) (everything green), blocked at test (failing tests never reach deploy), and rolled back (deployed but the canary tripped the automatic reversal). Code → live, observable, reversible.
Try this: Add "scan": False to the first commit and predict the result — it should now print blocked at scan, because scan comes before build in the stage list and the loop stops at the first failure.
docker compose up to develop → PR (DF3) runs tests+scan (TQ/CD) in CI → merge auto-builds and canary-deploys (CD) with health + rollback → measured by DORA. Delivering and owning that end-to-end loop is what a production-ready tech lead does.🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: Every safe-delivery gate later in this project stands on one thing: an app that can honestly report whether it's healthy. A health check that ignores its dependencies is worse than none — it hides outages.
Your task: Implement handle(path, db_ok, cache_ok) serving / and /healthz, plus a production Dockerfile.
Requirements:
/returns a 200 service response/healthzreturns 200 only when every dependency is up/healthzreturns 503 when a dependency is down- An unknown path returns 404
- A production Dockerfile with a
HEALTHCHECKhitting/healthz(labelled as needing Docker to build)
💡 Hint: Let the health branch reflect the db_ok/cache_ok inputs directly so an orchestrator can stop routing to a broken pod.
Show solution
A real /healthz that reflects dependencies is what makes every later gate possible. Handler is runnable; Dockerfile labeled:
def handle(path, db_ok=True, cache_ok=True):
if path == "/":
return 200, {"service": "app", "status": "ok"}
if path == "/healthz":
if db_ok and cache_ok:
return 200, {"status": "healthy"}
return 503, {"status": "unhealthy",
"db": db_ok, "cache": cache_ok}
return 404, {"error": "not found"}
print(handle("/healthz")) # (200, {'status': 'healthy'})
print(handle("/healthz", db_ok=False)) # (503, ... db False)
# Dockerfile --- needs Docker to build ---
# FROM python:3.12-slim
# WORKDIR /app
# COPY requirements.txt .
# RUN pip install --no-cache-dir -r requirements.txt
# COPY . .
# HEALTHCHECK CMD curl -f http://localhost:8000/healthz || exit 1
# CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]
A health check that returns 503 when the DB is down lets orchestrators stop routing traffic to a broken pod. A 200 that ignores dependencies is worse than none — it hides outages.
Context: The classic startup flake is the app crash-looping because the database isn't ready yet. Gating the app on a dependency's health, not just its existence, prevents it.
Your task: Write a docker-compose that gates the app on a Postgres healthcheck, and model the startup-ordering guarantee offline.
Requirements:
- Compose gates the app with
depends_on: condition: service_healthy - The Postgres service defines a healthcheck (e.g.
pg_isready) - An offline model of the rule: the app can't start while a dependency is unhealthy
- The model reports which dependency it's waiting on
- Label the compose file as needing Docker Compose
💡 Hint: Waiting on service_healthy (not just service_started) is the whole point — model that distinction in the offline check.
Show solution
Ordering startup on health, not just existence, prevents the classic "app crashes because DB isn't ready yet" flake:
# docker-compose.yml --- needs Docker Compose ---
# services:
# db:
# image: postgres:16
# healthcheck:
# test: ["CMD", "pg_isready", "-U", "app"]
# interval: 5s
# retries: 5
# app:
# build: .
# depends_on:
# db:
# condition: service_healthy # wait for DB health, not just start
# offline model of the ordering guarantee:
def can_start(app, deps_health):
unhealthy = [d for d, ok in deps_health.items() if not ok]
if unhealthy:
return False, f"waiting on {unhealthy}"
return True, "starting"
print(can_start("app", {"db": False})) # (False, "waiting on ['db']")
print(can_start("app", {"db": True})) # (True, 'starting')
condition: service_healthy makes Compose wait for pg_isready to pass before launching the app, so you never see the app crash-loop against a DB that's still initializing.
Context: The spine of safe delivery is an ordered gate chain: test → scan → build → deploy, where any failure stops everything downstream and you never ship code that failed a cheaper earlier check.
Your task: Model the pipeline stages and their pass/fail gating, and write the GitHub Actions skeleton with SHA-tagged images.
Requirements:
- An ordered stage list where the first failure halts the pipeline
- The result reports whether it deployed and, if not, which stage failed
- The Actions skeleton runs test, then vulnerability scan, then build, then deploy
- Images are tagged with the commit SHA for traceability and rollback
- Cheaper checks run before expensive ones (test before build)
💡 Hint: Iterate the stages in order and short-circuit on the first that didn't pass; the SHA tag is what makes every deploy traceable to an exact commit.
Show solution
An ordered gate chain — no stage runs unless the prior passed — is the spine of safe delivery:
def pipeline(stages):
# stages: ordered list of (name, passed_bool)
for name, passed in stages:
if not passed:
return {"deployed": False, "failed_at": name}
return {"deployed": True, "failed_at": None}
print(pipeline([("test",True),("scan",True),("build",True),("deploy",True)]))
print(pipeline([("test",True),("scan",False),("build",True)])) # stops at scan
# .github/workflows/deploy.yml --- needs GitHub Actions ---
# jobs:
# ship:
# steps:
# - run: pytest -q # test
# - run: trivy image --exit-code 1 $IMAGE # scan (fail on vuln)
# - run: docker build -t $IMAGE . # build
# # image tag pins the exact commit:
# # IMAGE=ghcr.io/${{ github.repository }}:${{ github.sha }}
# - run: ./deploy.sh $IMAGE # deploy
SHA-tagged images make every deploy traceable to an exact commit and trivially rollback-able. The gate order (test before scan before build before deploy) means you never ship code that failed a cheaper earlier check.
Context: Even a green pipeline can ship a bad build. Two more checks catch it: a readiness gate that blocks known-bad artifacts, and a canary that limits blast radius by ramping traffic and aborting on an error spike.
Your task: Add a ready() gate (tests + image score + health + secrets clean) and a canary() that ramps 5%→50%→100% and auto-aborts.
Requirements:
ready()returns pass/fail plus the specific problems found- It checks tests passing, an image score bar, health, and no secrets detected
canary()steps through 5%, 50%, 100% traffic- It aborts at the first stage whose error rate exceeds the threshold
- Report which stage aborted so the failure is diagnosable
💡 Hint: The readiness gate is the last check before any traffic; the canary is the first check with real traffic — abort before full exposure.
Show solution
The readiness gate is the last check before traffic; the canary is the check with real traffic — abort before full exposure:
def ready(tests, image_score, health, secrets_clean, min_score=80):
problems = []
if not tests: problems.append("tests failing")
if image_score < min_score: problems.append(f"image score {image_score}")
if not health: problems.append("health check red")
if not secrets_clean: problems.append("secrets detected")
return (len(problems) == 0, problems)
def canary(err_by_stage, threshold=0.02):
# err_by_stage: dict {5:err,50:err,100:err} error rate per traffic %
for pct in (5, 50, 100):
if err_by_stage[pct] > threshold:
return {"promoted": False, "aborted_at": f"{pct}%"}
return {"promoted": True, "aborted_at": None}
print(ready(tests=True, image_score=92, health=True, secrets_clean=True))
print(canary({5:0.005, 50:0.03, 100:0.0})) # aborts at 50% -> error spike
The readiness gate blocks known-bad builds; the canary catches the ones that only fail under real traffic, limiting blast radius to 5-50% before auto-aborting. Together they turn deploys from scary to routine.
Context: A deploy isn't done until you can undo it fast. Rollback to the previous SHA image is seconds, not a rebuild; an error budget makes 'should we deploy?' an objective call instead of a debate.
Your task: Model a rollback to the previous SHA-tagged image and an SLO error-budget tracker that decides when to freeze deploys.
Requirements:
- Rollback repoints to the last good SHA image (not a rebuild)
- Rollback errors cleanly when there's no previous version
- The budget tracker computes remaining error budget from actual success rate
- It signals a deploy freeze once the budget is exhausted
- Both are runnable offline over a deploy history / success numbers
💡 Hint: Keep deploys as an append-only history and roll back to the second-to-last entry; freeze when burned error exceeds the window's budget.
Show solution
Fast rollback + an error budget turn incidents into non-events and gate risky deploys automatically:
DEPLOYS = ["sha-aaa", "sha-bbb", "sha-ccc"] # append-only history
def rollback():
if len(DEPLOYS) < 2:
raise RuntimeError("no previous version to roll back to")
return DEPLOYS[-2] # repoint to prior good SHA
def budget_status(slo=0.999, actual_success=0.9985, window_budget=0.001):
burned = (1 - actual_success)
remaining = window_budget - burned
return {"remaining_budget": round(remaining, 5),
"freeze_deploys": remaining <= 0}
print("rollback to:", rollback()) # sha-bbb
print(budget_status(actual_success=0.9985)) # budget left, no freeze
print(budget_status(actual_success=0.997)) # budget blown -> freeze
Rollback is repointing to the last good SHA image — seconds, not a rebuild. The error budget makes "should we deploy?" objective: burn it down and deploys freeze until reliability recovers, protecting the SLO from a risky-change spree.
Context: A tech lead owns delivery health, and 'are we shipping well?' becomes four measurable numbers. Owning the DORA metrics means driving deliberate changes and proving the improvement with the numbers.
Your task: Compute the four DORA metrics from a deploy log and classify the team's performance tier.
Requirements:
- Deploy frequency, lead time, change-failure rate, and restore time (MTTR)
- Computed from a deploy log of
{lead_hours, failed, restore_hours}entries - MTTR averages only the failed deploys' restore times
- Classify into a tier (e.g. elite / high / needs-work) from the numbers
- Runs offline over a sample log
💡 Hint: Change-failure rate is failures over total deploys; restore time only counts the deploys that actually failed.
Show solution
DORA turns "are we shipping well?" into four measurable numbers a tech lead is accountable for:
def dora(deploys):
# deploys: list of {lead_hours, failed(bool), restore_hours}
n = len(deploys)
freq_per_week = n # over a 1-week window, say
lead = sum(d["lead_hours"] for d in deploys)/n
cfr = sum(d["failed"] for d in deploys)/n
fails = [d["restore_hours"] for d in deploys if d["failed"]]
mttr = sum(fails)/len(fails) if fails else 0.0
tier = ("elite" if freq_per_week >= 7 and lead < 24 and cfr <= 0.15
else "high" if cfr <= 0.30 else "needs work")
return {"deploys/wk": freq_per_week, "lead_h": round(lead,1),
"cfr": round(cfr,2), "mttr_h": round(mttr,1), "tier": tier}
log = [{"lead_hours":4,"failed":False,"restore_hours":0},
{"lead_hours":6,"failed":True, "restore_hours":0.5},
{"lead_hours":3,"failed":False,"restore_hours":0}]
print(dora(log))
Elite teams deploy often, with short lead time, low change-failure rate, and fast restore. Owning these metrics means driving deliberate changes — smaller batches lift frequency and cut CFR; rehearsed rollback cuts MTTR — and proving the improvement with the numbers.
✓ Checkpoint — you can move on when you can…
- Containerize a multi-service app to the checklist.
- Run the full stack locally with Compose.
- Automate test→scan→build→deploy with readiness + canary.
- Operate live: health, SLOs, rollback, DORA ownership.
| Dimension | Meets the bar | Above the bar (staff) |
|---|---|---|
| Build reproducibility | A pinned, multi-stage Dockerfile builds the same image from a clean checkout; base image and deps are version-locked. | Image is tagged by commit SHA, builds are byte-reproducible, and a fresh clone → docker compose up works with zero manual steps. |
| CI/CD correctness | Pipeline runs test → scan → build → deploy in order; a red test blocks the deploy (needs: test). | Every gate is a hard gate (scan fails on HIGH/CRITICAL), deploys are idempotent, and the pipeline is green from a clean fork. |
| Deploy safety & rollback | Releases roll out by canary with an error-rate threshold; you can roll back to a prior SHA-tagged image. | Rollback is rehearsed and one-command; canary aborts automatically on error/latency spike and the abort path is tested. |
| Health & readiness | /healthz reports dependency health (DB/cache) and returns 503 when degraded; Compose gates on service_healthy. | Readiness vs liveness are distinguished, health drives traffic routing, and a downed dependency is proven to hold back traffic. |
| Secrets & config | No secrets baked into the image or committed; config comes from env/secret store; the readiness gate checks for leaked secrets. | Secrets are scanned in CI and rotated out of history; least-privilege credentials are used and the image is verified clean. |
| Observability & SLOs | You name the signals to watch (latency, error rate, saturation, traffic) and track the DORA four. | Dashboards + alerts are wired to an SLO error budget with burn-rate alerts, and the weakest DORA metric has an improvement plan. |
Score each dimension 0 (missing), 1 (meets bar), or 2 (above bar). 0–4: keep building. 5–8: a solid, defensible submission. 9–12: staff-level — you could hand this to a reviewer and defend every call. Any dimension at 0 blocks shipping regardless of the total.
Knowledge check check yourself
Why does the /healthz endpoint return 503 when a dependency (db or cache) is down instead of a plain 200?
Show answer
What bug does depends_on: { db: { condition: service_healthy } } in Compose prevent?
Show answer
pg_isready healthcheck), preventing the classic race where the app boots faster than the DB and crashes on its first query.