CI/CD pipelines

1 · What CI/CD is (and why) essential

Picture the Friday-afternoon deploy that goes wrong. Someone SSHes into the server, pulls the latest code, forgets to run the migrations, restarts the wrong process, and goes home. Monday morning the app is down and nobody remembers exactly which of the eleven manual steps was skipped. Every team that ships by hand has lived some version of this — and CI/CD is the discipline that makes it structurally impossible.

Here's the whole idea in one sentence: let a machine do the boring, dangerous parts the exact same way every single time. A human deploying is like a chef cooking from memory — brilliant on a good day, disastrous when tired or interrupted. A pipeline is the written recipe that a robot follows to the letter: same ingredients, same order, same result, whether it's the first deploy of the day or the fiftieth at midnight. It never forgets a step because forgetting isn't something it can do.

The two halves have names worth pinning down, because people blur them. CI — Continuous Integration is the testing half: every time you push code, a fresh machine checks it out and runs your test suite, so you find out within minutes whether your change broke anything. CD — Continuous Delivery/Deployment is the shipping half: when the tests pass, the same automation builds the release and pushes it out. Together they form a chain where each link only fires if the one before it succeeded.

The mistake beginners make is treating automation as a convenience rather than a safety mechanism. "It just saves me typing" misses the point entirely. The value isn't speed — it's that a pipeline turns your quality bar into a wall. If you decide tests must pass before code ships, and you encode that in the pipeline, then it is no longer a rule people can quietly skip under deadline pressure; it's a gate that physically will not open. That shift — from good intentions to enforced invariants — is why this chapter exists.

The formal version, now that you have the feel for it: CI runs tests on every push; CD builds and ships automatically when they pass. Manual deploys are slow and error-prone — someone forgets a step, deploys the wrong tag, skips tests. A pipeline does the same steps identically every time, gated on quality.

2 · A first pipeline essential

Enough theory — let's look at the actual recipe. The abstract diagram above becomes a real, running pipeline through a small text file that lives inside your repository. This is the moment the idea gets concrete: you're about to see the exact YAML that GitHub reads and obeys, and it's shorter than most people expect.

The tool here is GitHub Actions, and the mental model is simple: you drop a file into a magic folder — .github/workflows/ — and GitHub treats it as standing orders. "Whenever this happens, do that." The file isn't a program you execute; it's a declaration you hand to GitHub's servers, which spin up fresh throwaway Linux machines to carry it out. Think of it like leaving a note for a very literal assistant: it will do precisely what the note says, nothing more, nothing less.

Three questions unlock any workflow file, and it pays to read every one you meet in that order. When does it run? (the on: trigger — here, a push to main). What separate units of work are there? (the jobs:, each on its own fresh machine). And what steps does each job take, top to bottom? Once you can answer those three, the scariest-looking pipeline config becomes readable. Like Python, YAML uses indentation to show what belongs to what, so the shape on the page mirrors the structure of the work.

The common trap is skimming past the one line that matters most. In the file below there are two jobs, and the second carries needs: test. That single line is the entire safety story — it wires the deploy job to depend on the test job, so a red test suite stops everything cold. Beginners tend to admire the docker build commands and miss that needs: is where the real engineering lives. Watch for it.

deploy.ymlname: CI/CD
on: { push: { branches: [main] } }
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r requirements.txt pytest
      - run: pytest -q
  build-deploy:
    needs: test                    # ONLY runs if tests pass
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: |
          IMG=ghcr.io/${{ github.repository }}:${{ github.sha }}
          echo "${{ secrets.GHCR_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
          docker build -t "$IMG" .
          docker push "$IMG"
      - run: echo "deploy the tagged image (gcloud run deploy / aws ecs update-service)"

3 · Model a pipeline in Python intermediate

A pipeline can feel like magic until you build a toy one yourself — then the mystery evaporates. GitHub Actions, GitLab CI, Jenkins, and every custom deploy script you'll ever meet are, at heart, the same twenty lines of Python. Writing those lines strips away the YAML and the cloud and leaves you with the one idea underneath: run a list of steps in order, and bail out the instant one fails.

The behaviour that makes a pipeline a pipeline (rather than just a list of tasks) is short-circuiting. Think of a bouncer checking IDs at a door with several checkpoints: fail the first check and you never reach the second. The pipeline doesn't heroically try to deploy after the tests failed and hope for the best — it stops, tells you which checkpoint rejected you, and refuses to run anything downstream. That "stop at the first failure" rule is the difference between a gate and a suggestion.

Why bother modeling it in plain Python at all, when GitHub does this for you? Two reasons. First, it makes the control flow you can't see in the cloud completely visible on your screen — you'll never again wonder what "the pipeline stopped at test" actually means. Second, it's genuinely how real tooling works: a deploy script, a Makefile target, an orchestration library — all of them are ultimately a loop over stages with an early exit. Once you've written the loop, you own the concept, not just the vocabulary.

The classic beginner mistake here is writing a pipeline that reports failures but keeps going anyway — collecting every error and running deploy at the end regardless. That's not a pipeline; it's a pipeline-shaped bug that ships broken code with a warning attached. The single return on first failure in the code below is what keeps it honest.

A pipeline is stages that short-circuit on failure. Modeling it makes the control flow explicit — and it's how a custom deploy tool works under the hood.

pipeline.pydef run_pipeline(stages):
    """stages: [(name, fn)]. Stop at the first failure; report where."""
    for name, fn in stages:
        ok = fn()
        print(f"{'PASS' if ok else 'FAIL'}  {name}")
        if not ok:
            return False, name          # nothing after a failure runs
    return True, None

ok = [("lint",lambda:True),("test",lambda:True),("build",lambda:True),("deploy",lambda:True)]
print("result:", run_pipeline(ok))
broken = [("lint",lambda:True),("test",lambda:False),("deploy",lambda:True)]
print("result:", run_pipeline(broken))   # stops at test; deploy never runs

PASS  lint
PASS  test
PASS  build
PASS  deploy
result: (True, None)
PASS  lint
FAIL  test
result: (False, 'test')

4 · Security scanning in the pipeline advanced

Your tests can be all green and your image can still be a liability. Passing tests prove your code does what you meant — they say nothing about the pile of third-party libraries and base-OS packages baked into the container alongside it. A known vulnerability in an old version of some logging library you've never heard of is exactly the kind of thing that ends up on the news, and no unit test will ever catch it. That gap is what security scanning fills.

The analogy is airport security. Your bags are packed and you personally trust everything in them — but you still go through the scanner, because the point isn't to catch you, it's to catch the things you didn't know were dangerous. A scanner like Trivy or Grype reads your image, looks up every package inside it against public databases of known flaws (CVEs — Common Vulnerabilities and Exposures), and hands back a list graded by severity: LOW, MEDIUM, HIGH, CRITICAL. Separate scanners do the same for your dependency manifest and hunt through your code for accidentally-committed secrets like API keys.

A scan on its own is just a report, though — and a report nobody acts on is worthless. The engineering move is to turn the report into a gate: a rule that says "if there are any HIGH or CRITICAL findings, fail the build and do not deploy." This is where scanning stops being a nice dashboard and becomes a wall in your pipeline, exactly like the test gate from §2. The lab below models that gate as a tiny function, because underneath the tooling that's all it is: count the serious findings, compare against a threshold, return pass or fail.

The mistake teams make is running scans but never wiring them to block anything — the findings pile up in a tab someone glances at quarterly, and vulnerable images ship anyway. Equally common is the opposite over-correction: setting zero tolerance, drowning in noise from unfixable base-image CVEs, and then disabling the scan out of frustration. The honest middle ground is configurable thresholds you set deliberately, which is precisely why the gate below takes max_high and max_critical as arguments rather than hardcoding them.

Professional pipelines scan before deploy: the image for known CVEs (Trivy/Grype), dependencies for vulnerabilities, and code for leaked secrets. Fail the build on high-severity findings.

scan_gate.pydef scan_gate(findings, max_high=0, max_critical=0):
    highs = sum(1 for f in findings if f["severity"] == "HIGH")
    crits = sum(1 for f in findings if f["severity"] == "CRITICAL")
    passed = highs <= max_high and crits <= max_critical
    return passed, {"high": highs, "critical": crits}

clean = scan_gate([{"severity":"LOW"},{"severity":"MEDIUM"}])
print("clean image:", clean)
vuln = scan_gate([{"severity":"CRITICAL"},{"severity":"HIGH"},{"severity":"LOW"}])
print("vulnerable image:", vuln, "-> block deploy")

clean image: (True, {'high': 0, 'critical': 0})
vulnerable image: (False, {'high': 1, 'critical': 1}) -> block deploy

5 · Advanced — environments & approvals advanced

Fully automatic deploy-straight-to-production sounds like the dream — right up until it ships a bug to a million users at 2am with nobody watching. Speed is only virtue when you can trust what you're shipping, and some risks are worth a human pause. The grown-up version of a pipeline doesn't remove the humans; it puts them at exactly the one point where judgment matters and automates everything else.

The key concept is the environment: a named target you deploy to, each with its own settings, secrets, and rules. The pattern nearly every serious team converges on is a two-stage rocket. First, staging — a production-like clone with fake or copied data, where the freshly built release lands automatically the moment tests go green, and smoke tests confirm it actually boots and serves traffic. Only after staging is happy does production come into play. Think of staging as the dress rehearsal in full costume on the real stage, and production as opening night: you never open cold.

Between the two sits the human gate — an approval. In GitHub Actions you mark an environment (say, production) as protected, and the job targeting it simply waits until an authorized person clicks "approve." Everything up to that click is machine-fast and machine-reliable; the click itself is the deliberate, reversible-decision moment where someone with context can say "not during the sale" or "let's wait for the on-call to be awake." Automation handles the toil; the human owns the risk. The jobs are chained in strict order with the same needs: keyword you already met — prod literally cannot start until staging has succeeded.

The trap is treating staging as a rubber stamp — deploying to it but never actually testing there, so it gives false confidence, or conversely gating everything behind manual approval until the friction pushes people to bypass the pipeline entirely. The sweet spot the config below encodes is deliberate: automatic all the way to staging, one approval to cross into prod. That's enough control to catch disasters and little enough friction that people actually use it.

staged.yml  deploy-staging:
    needs: test
    environment: staging           # auto-deploy to staging on green
    steps: [ ... deploy to staging + run smoke tests ... ]
  deploy-prod:
    needs: deploy-staging
    environment:
      name: production             # GitHub requires a human approval here
    steps: [ ... deploy to prod (canary via CD4) ... ]

6 · Professional — secrets & supply chain professional

Stop and reframe what your pipeline actually is: it's a robot that holds the keys to production and runs code from the internet. It has the credentials to deploy, the token to push images, and it happily executes third-party actions written by strangers. If an attacker wanted one thing in your whole organization, this is it — the pipeline is a skeleton key. Most teams guard production like a vault and leave the machine that can rewrite production wide open.

Start with secrets — the passwords, tokens, and keys your pipeline needs. The cardinal sin is hardcoding them in the workflow file, because that file lives in git, and git history is forever; a secret committed once is a secret leaked permanently, even after you "delete" it. The fix is the platform's secret store: an encrypted vault where you register secrets by name, and the pipeline references them (${{ secrets.TOKEN }}) so the real value is injected at run time and never written down. Pair that with least privilege — a token that can only push to one registry can't also delete your database, so a leak of it is a scratch rather than a fatality.

Then the subtler danger: the supply chain. Your pipeline is built from parts you didn't write — actions/checkout, some community action that posts to Slack — and each is code that runs with your secrets in scope. If the author of a popular action is compromised (or turns malicious) and you've told your workflow to always grab the "latest" version, you've handed your credentials to whoever controls that action, automatically, on your next run. This is not hypothetical; it's how several real breaches happened. The defence is to pin every third-party action to an exact, reviewed version (ideally a commit hash), so nobody can swap the code under you.

The mistake even experienced engineers make is thinking about supply-chain risk only for their application and forgetting the pipeline is downstream of the same threat. You scanned your image (§4) — good — but did you verify the action that built it? The professional endgame closes the loop: pin and verify what runs, scope what it can touch, and sign or attest your images so that later you can cryptographically prove this exact artifact came from this exact pipeline and wasn't tampered with in between. Provenance turns "we think we shipped the right thing" into "we can prove it."

CI has powerful credentials — a prime target. Use the platform's secret store (never hardcode), scope tokens to least privilege, pin action versions (a compromised third-party action can steal secrets), and sign/attest images so you can prove what you shipped.

7 · Tech-lead — measure delivery with DORA tech-lead

Ask two engineers whether their team is "good at shipping" and you'll get two vibes, not two answers. One says "we're fast," the other says "we're careful," and neither can be checked. The moment you're leading delivery rather than just doing it, gut feel stops being enough — you need numbers that let you compare this quarter to last, your team to the industry, and say honestly whether you're improving. That's the problem the DORA metrics solve.

DORA (DevOps Research and Assessment — a multi-year research program that studied thousands of teams) found that elite software delivery boils down to just four measurable things, and they split cleanly into two tensions. Speed: how often you deploy (deploy frequency) and how long a change takes to go from commit to live (lead time). Stability: what fraction of deploys break something (change-failure rate) and how fast you recover when they do (time-to-restore). Think of it as a car dashboard: speed and safety, on one panel, at a glance.

The counter-intuitive finding — and the reason DORA matters more than most metrics — is that speed and stability are not a trade-off. The naive assumption is that shipping faster must mean breaking more, so you should slow down to be safe. The data says the opposite: elite teams are both the fastest and the most reliable, because the same practices that let you ship often (small changes, strong automation, easy rollback) are exactly what make each ship safe. A team that deploys once a quarter isn't being careful; it's building up a terrifying pile of untested change for one big scary release.

The trap for a new lead is optimizing one number and congratulating yourself. "We deploy ten times a day!" means little if a third of those deploys break and take a day to fix. The four metrics are read together — and, crucially, your weakest one is your ceiling. A team that's fast but recovers slowly is still fragile; the honest move is to find the metric dragging you down and fix that first, not to polish the one you're already winning at. The classifier below encodes exactly this logic: it demands all four conditions clear the bar to earn a tier, so a single weak metric caps the whole score — pointing you straight at what to improve.

A delivery lead manages by the DORA metrics: deploy frequency, lead time for changes, change-failure rate, and time-to-restore. They tell you whether delivery is elite or struggling, and where to invest.

dora.pydef dora_tier(deploys_per_week, lead_time_h, change_fail_pct, restore_h):
    elite = deploys_per_week>=7 and lead_time_h<=24 and change_fail_pct<=15 and restore_h<=1
    high  = deploys_per_week>=1 and lead_time_h<=168 and change_fail_pct<=30 and restore_h<=24
    return "Elite" if elite else "High" if high else "Medium/Low"

print("team A:", dora_tier(14, 4, 10, 0.5))     # ships daily, recovers in minutes
print("team B:", dora_tier(0.5, 336, 40, 72))   # rare big releases, slow recovery
print("improve B: raise deploy freq (smaller changes) + cut restore time (rollback)")

team A: Elite
team B: Medium/Low
improve B: raise deploy freq (smaller changes) + cut restore time (rollback)

Push/merge (trigger)
   -> Test (CI)              # the gate: if tests fail, nothing after runs
      -> Build + scan + push  # package image, scan for CVEs, push to registry
         -> Deploy (CD)       # put the built image live

name: CI/CD
on: { push: { branches: [main] } }
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - run: pip install -r requirements.txt pytest
      - run: pytest -q
  build-deploy:
    needs: test                    # ONLY runs if tests pass
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: |
          IMG=ghcr.io/${{ github.repository }}:${{ github.sha }}
          echo "${{ secrets.GHCR_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
          docker build -t "$IMG" .
          docker push "$IMG"
      - run: echo "deploy the tagged image (gcloud run deploy / aws ecs update-service)"

def run_pipeline(stages):
    """stages: [(name, fn)]. Stop at the first failure; report where."""
    for name, fn in stages:
        ok = fn()
        print(f"{'PASS' if ok else 'FAIL'}  {name}")
        if not ok:
            return False, name          # nothing after a failure runs
    return True, None

ok = [("lint", lambda: True), ("test", lambda: True),
      ("build", lambda: True), ("deploy", lambda: True)]
print("result:", run_pipeline(ok))
broken = [("lint", lambda: True), ("test", lambda: False), ("deploy", lambda: True)]
print("result:", run_pipeline(broken))   # stops at test; deploy never runs

def scan_gate(findings, max_high=0, max_critical=0):
    highs = sum(1 for f in findings if f["severity"] == "HIGH")
    crits = sum(1 for f in findings if f["severity"] == "CRITICAL")
    passed = highs <= max_high and crits <= max_critical
    return passed, {"high": highs, "critical": crits}

clean = scan_gate([{"severity":"LOW"}, {"severity":"MEDIUM"}])
print("clean image:", clean)
vuln = scan_gate([{"severity":"CRITICAL"}, {"severity":"HIGH"}, {"severity":"LOW"}])
print("vulnerable image:", vuln, "-> block deploy")
# clean: (True, {'high': 0, 'critical': 0})
# vuln:  (False, {'high': 1, 'critical': 1}) -> block deploy

  deploy-staging:
    needs: test
    environment: staging           # auto-deploy to staging on green
    steps: [ ... deploy to staging + run smoke tests ... ]
  deploy-prod:
    needs: deploy-staging
    environment:
      name: production             # GitHub requires a human approval here
    steps: [ ... deploy to prod (canary via CD4) ... ]

def dora_tier(deploys_per_week, lead_time_h, change_fail_pct, restore_h):
    elite = deploys_per_week >= 7   and lead_time_h <= 24  and change_fail_pct <= 15 and restore_h <= 1
    high  = deploys_per_week >= 1   and lead_time_h <= 168 and change_fail_pct <= 30 and restore_h <= 24
    return "Elite" if elite else "High" if high else "Medium/Low"

print("team A:", dora_tier(14, 4, 10, 0.5))     # ships daily, recovers in minutes
print("team B:", dora_tier(0.5, 336, 40, 72))   # rare big releases, slow recovery
print("improve B: raise deploy freq (smaller changes) + cut restore time (rollback)")
# team A: Elite   /   team B: Medium/Low

Knowledge check check yourself