Progressive delivery & service mesh

1 · Rolling vs canary vs blue-green

Picture the Friday-afternoon deploy that goes wrong. You merge a change, the pipeline runs a rolling update, every Pod turns green, and you go home. Two hours later the on-call phone lights up: a config typo means one in seven answers is nonsense, and by now every user is on the broken version. Nothing was technically "down" — the process was healthy the whole time — so no alert fired until humans complained. That gap, between "the deploy succeeded" and "the deploy is good", is the entire reason this lesson exists.

Here's the mental model that unlocks all three strategies: a deploy isn't a light switch, it's a dial. The only question that matters is how many real users see the new version before you're sure it's safe. A rolling update turns that dial for you automatically and irreversibly — by the time you notice trouble, the dial is already at 100%. Progressive delivery hands you the dial and, crucially, a pair of eyes on the metrics so you can stop turning it the instant something looks off.

Think of it like tasting soup while you cook. A rolling update is dumping the whole spice jar in and serving the pot to a hundred guests at once — if it's too salty, everyone's dinner is ruined. A canary is tasting one spoonful first: you salt a small bowl, taste it, and only season the whole pot if the spoonful was good. A blue-green deploy is cooking a second pot from scratch alongside the first, tasting it fully, and then swapping which pot goes to the table — instant, clean, but you paid to run two kitchens.

The common mistake is treating "it passed CI and health checks" as "it's safe to send everyone." Health checks (which you met in K3) answer "is the process alive and accepting requests?" — a genuinely useful but shallow question. They cannot tell you the answers got worse, the model loaded the wrong weights, or latency crept up under real load. Progressive delivery exists precisely to gate on the things health checks are blind to: the outcomes users actually feel.

The default Deployment strategy is a rolling update: gradually replace old Pods with new ones. It's simple but blunt — every user hits the new version as it rolls, and a bad version is discovered only after it's widely live. Progressive delivery adds a controlled exposure step.

2 · Automating it: Argo Rollouts

A canary you drive by hand is a canary that will eventually kill you. The manual version goes like this: bump the canary to 10%, open Grafana in one tab and logs in another, stare for five minutes, decide it looks fine, bump to 50%, stare again, promote. It works — right up until the release that matters happens at 3am, or on the afternoon you're in a meeting, or when the person who knows which dashboard to watch is on holiday. Human attention is the least reliable component in your whole delivery pipeline, and canary decisions are exactly the kind of repetitive, high-stakes judgment that humans are worst at making consistently.

So we do the obvious engineering move: write the procedure down as data and let a controller execute it. That's all a Rollout is — the ramp schedule (10% → 50% → 100%), the pauses, and the pass/fail rule, expressed once in YAML so it runs identically every single time, day or night, whether or not anyone is watching. Think of it as autopilot for deployments: you still set the flight plan, but a machine holds the controls and, more importantly, reacts in milliseconds when an instrument goes red.

The mechanic worth internalizing is that Argo Rollouts replaces your Deployment object rather than sitting beside it. You hand it the same Pod template you'd give a Deployment, and it takes over the job of managing ReplicaSets — but now with a rich strategy block. This is why K6 (GitOps) sits before K10: the Rollout is just another declarative manifest, so it lives in Git and reconciles like everything else. Flagger is the equivalent tool from the Flux/mesh world; it watches an ordinary Deployment and drives a mesh's traffic weights instead of owning the Deployment itself — same idea, different plumbing.

The common trap here is writing a beautiful multi-step ramp with no analysis gate in it. A ramp full of pause steps and nothing else is just a slower rolling update — it exposes users gradually but never actually checks whether the new version is any good, so a bad release still reaches 100%, just later. The analysis step is the part that turns "wait and hope" into "measure and decide"; the next two sections are entirely about making that gate trustworthy. A Rollout's steps describe the canary ramp and where to pause for an analysis gate:

rollout.yamlapiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: llm-api }
spec:
  replicas: 6
  selector: { matchLabels: { app: llm-api } }
  template: { }            # same pod template as a Deployment
  strategy:
    canary:
      steps:
        - setWeight: 10        # 10% of traffic to the new version
        - pause: { duration: 5m }
        - analysis:            # gate: run the AnalysisTemplate below
            templates: [ { templateName: success-rate } ]
        - setWeight: 50
        - pause: { duration: 5m }
        - setWeight: 100       # full promotion if analysis kept passing

analysis.yamlapiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: success-rate }
spec:
  metrics:
    - name: success-rate
      interval: 1m
      successCondition: result[0] >= 0.99     # abort the rollout if it dips
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{app="llm-api",status!~"5.."}[2m]))
              / sum(rate(http_requests_total{app="llm-api"}[2m]))

3 · Analysis-driven promotion, modeled offline

The word "analysis" makes this sound like magic. It isn't — and that's the point. When people hear that Argo Rollouts "analyzes the canary and decides," they imagine some clever machine-learning brain judging the deploy. Strip away the CRD and the Prometheus wiring and you're left with something you could write on an index card: read two numbers, compare them to a threshold, and return the string "promote" or "abort". Demystifying that is worth a whole section, because once you see the decision is just an if-statement, you stop trusting it blindly and start asking the right question: are the numbers I'm comparing actually meaningful?

Here's the analogy: the analysis gate is a bouncer at a door with a very short list of rules. "Success rate below 99%? Not coming in. Latency over 800ms? Not coming in. More than 20% slower than the version already inside? Not coming in. Otherwise, welcome." The bouncer isn't smart — it's consistent, and consistency at 3am is exactly the value. The intelligence lives in the rules you gave it, not in the checking.

Modeling this in plain Python before touching a cluster pays off twice. First, you can test your promotion logic exhaustively on made-up numbers — feed it a healthy canary, a slow one, a flaky one — and watch it decide, without deploying anything or paying for infrastructure. Second, it forces you to be honest about which comparisons you're making. Notice the version below uses three guards: an absolute floor (a hard SLO you never cross), an absolute ceiling, and a relative guard against the stable version. That third one is the subtle, important one — a canary can be technically within SLO yet clearly a regression, and only comparing it to what's already running catches that.

The mistake to avoid is gating on the wrong metric entirely. A rollout that gates on "are we returning HTTP 200?" will happily promote a model server that returns 200 while emitting garbage answers — the very failure the knowledge-check below describes. The analysis is only as trustworthy as the SLI behind it, so choose a signal that reflects the outcome users feel (answer quality, a downstream success rate, an eval score), not just process liveness. Strip away the CRDs and the promotion decision is simple arithmetic: after each step, compare the canary's SLIs to the stable's (or to an absolute threshold); promote if healthy, abort if not, with a tolerance so you don't abort on noise. Here it is as offline-runnable Python — the same logic the controller runs:

promotion.pydef evaluate_canary(canary, stable, min_success=0.99, max_latency_ms=800, latency_tol=1.2):
    """Return ('promote'|'abort', reason). canary/stable are dicts of measured SLIs."""
    # 1) Absolute success-rate floor (an SLO-style gate).
    if canary["success_rate"] < min_success:
        return "abort", f"success {canary['success_rate']:.3f} < {min_success}"
    # 2) Absolute latency ceiling.
    if canary["p95_ms"] > max_latency_ms:
        return "abort", f"p95 {canary['p95_ms']}ms > {max_latency_ms}ms"
    # 3) Relative guard: canary shouldn't be much worse than stable (catches regressions).
    if canary["p95_ms"] > stable["p95_ms"] * latency_tol:
        return "abort", f"p95 {canary['p95_ms']}ms >> stable {stable['p95_ms']}ms"
    return "promote", "canary healthy"

stable = {"success_rate": 0.999, "p95_ms": 500}
good   = {"success_rate": 0.998, "p95_ms": 540}
bad    = {"success_rate": 0.995, "p95_ms": 900}
print(evaluate_canary(good, stable))   # promote
print(evaluate_canary(bad, stable))    # abort (latency)

('promote', 'canary healthy')
('abort', 'p95 900ms > 800ms')

4 · How traffic actually shifts

"Send 10% to the canary" sounds like one instruction. It's actually a question about who's holding the steering wheel — and getting the answer wrong is how a canary quietly lies to you. Somebody, somewhere, has to look at each incoming request and decide "stable or canary?". The whole character of your canary — how precise it is, whether you can target specific users, whether 10% really means 10% — comes down to which component makes that call and what information it has when it makes it.

The crude-but-free way leans on a fact you already know from K2: a Kubernetes Service load-balances across whatever Pods match its selector. So if you run 1 canary Pod and 9 stable Pods behind one Service, roughly a tenth of connections land on the canary — a canary "for free," no new components. The catch is that your traffic dial now has only as many notches as you have Pods. Want 5%? You'd need 1 canary against 19 stable — twenty Pods to express one number. And because it splits at the connection level, a client that opens one long-lived connection and reuses it (very common with gRPC and HTTP keep-alive) sticks to whichever Pod it first hit, so your "10%" can drift to 0% or 40% in practice.

The analogy: replica weighting is splitting a dinner bill by counting heads at each table — good enough when tables are similar, hopeless when you need to charge one specific person a precise amount. Request-level splitting is an itemized bill: a dedicated component (an ingress controller or a service mesh) inspects every request and routes exactly 5%, or routes only requests carrying a certain header, or duplicates traffic to the canary without the user seeing it. It costs you that extra component, but it buys a dial with infinite notches and the ability to route on who the request is from, not just count.

The common mistake is claiming a precise percentage while using the coarse mechanism — telling stakeholders "we're at 10%" when replica ratios and connection reuse mean the real figure is anyone's guess, then drawing confident conclusions from a canary that never actually got a representative slice of traffic. Match the mechanism to the precision you're claiming. The basic way is replica weighting: run 1 canary Pod and 9 stable Pods behind one Service, so roughly 10% of connections hit the canary — crude (tied to replica counts, connection-level not request-level). The precise way needs a component that can split at the request level: an ingress controller or a service mesh that routes an exact percentage.

5 · Service mesh: mTLS, traffic policy, observability

Every microservice team eventually rediscovers the same painful truth: the interesting problems aren't in the services, they're in the space between them. Once you have a dozen services calling each other, you find yourself re-implementing the same plumbing over and over — retries when a call fails, timeouts so a slow dependency doesn't hang you, TLS so traffic isn't sniffable, metrics so you can see which hop is slow. Every team writes its own version, in its own language, slightly wrong, and none of them agree. A service mesh is the response to that mess: pull all the between-services concerns out of the application and into a shared layer the platform owns.

The trick it uses is almost sneaky. Instead of asking every app to add a networking library, the mesh slips a tiny proxy — a sidecar (Envoy, in Istio's classic mode) — into each Pod, right next to your container. Your app still thinks it's calling http://service-b directly, but the request quietly goes to the local proxy first, which handles the encryption, the retry, the timeout, the metric, and then forwards it to service B's proxy, which does the same on the way in. It's like giving every employee a personal translator-and-bodyguard who rides along to every meeting: the employee just talks normally, while the escort handles the encryption, checks the other party's ID, retries the introduction if the line drops, and files a report on how the meeting went — none of which the employee had to learn.

Because that escort sits on the wire, it can offer three things uniformly and without a single line of app-code change: mTLS (mutual TLS — both ends prove their identity with certificates the mesh issues and rotates, giving you encrypted, authenticated service-to-service traffic, i.e. zero-trust); traffic policy (the exact request-level splitting, retries, timeouts, circuit-breaking and mirroring from §4, applied consistently everywhere); and observability (the golden metrics and traces of K8, emitted for every hop automatically). One layer, three capabilities, all languages.

Say the honest part out loud, though: that escort is a real container consuming CPU and memory on every Pod, it adds a network hop to every call, and the control plane that programs all the proxies is one more distributed system you must run, secure, and upgrade. The mesh isn't free wizardry — it's a trade, and §6 is entirely about deciding whether that trade is worth it for your problem. A service mesh puts a programmable network layer between your services. Classically it injects a sidecar proxy (Envoy in Istio) next to every Pod; all pod-to-pod traffic flows through the proxies, which the mesh control plane configures. That gives you three things without changing app code:

Sidecar vs sidecarless. The sidecar-per-Pod model is powerful but costs a proxy container's CPU/memory on every Pod and adds a network hop. Newer designs reduce this: Linkerd uses a deliberately tiny Rust 'micro-proxy'; Istio ambient mode moves L4 to a per-node component ('ztunnel') and makes L7 proxies optional. The trade-off is always feature richness vs overhead and complexity.

virtualservice.yamlapiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata: { name: llm-api }
spec:
  hosts: [ llm-api ]
  http:
    - route:
        - destination: { host: llm-api, subset: stable }
          weight: 90
        - destination: { host: llm-api, subset: canary }
          weight: 10          # exact 10% at the request level

6 · When a mesh is — and isn't — worth it

This is the section that will save you the most money and grief, and it's the one most tutorials skip. A service mesh is genuinely impressive technology, and impressive technology has a gravitational pull — the demo is slick, the diagrams are clean, and "we run a mesh" sounds like the mark of a serious platform team. That pull is exactly why so many clusters end up carrying a full mesh they don't need, paying its tax on every Pod for a capability they could have gotten for a fraction of the cost. Resisting that pull is a senior-engineer skill, and it comes down to one unglamorous habit: matching the cost of the solution to the size of the problem.

Here's the analogy to keep in your head. A mesh is a delivery van. If you're moving a house every week, the van is obviously worth its insurance, fuel, and parking. If you just need to carry your groceries home, buying a van — and paying to run it forever — because it happens to have room for groceries is a bad trade; a backpack does the job. The mesh "van" makes sense when you have a lot of between-services cargo (mTLS across dozens of services, uniform retries in five languages, fine-grained canaries every day). It's a poor trade when you have one small errand.

The decision hinges on how many of the mesh's capabilities you truly need and across how many services. Any single capability — canary splitting, retries, edge encryption — usually has a cheaper, more targeted tool: the Gateway API or your ingress splits traffic, a client library adds retries, edge TLS covers the untrusted hop. A mesh starts to win only when you need several of these at once, across many teams and languages, because at that point the per-service cost of doing it N different ways finally exceeds the mesh's fixed overhead. mTLS at scale is the capability that most often tips the balance, because implementing certificate issuance and rotation correctly in every service is genuinely miserable.

The classic mistake — worth naming so you recognize it in a design review — is adopting the whole mesh to get one feature you could buy cheaper elsewhere, most often canary traffic splitting. The right default is to start without a mesh and adopt it only when the pain is real and broad, ideally in its lower-overhead sidecarless/ambient form. A mesh is powerful and not free: sidecars add resource overhead and a latency hop, the control plane is another critical system to run and upgrade, and debugging gains a layer. The honest engineering question is whether your problems actually need it.

kind: Rollout
spec:
  replicas: 6
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 5m }
        - setWeight: 100

kind: AnalysisTemplate
metadata: { name: success-rate }
spec:
  metrics:
    - name: success-rate
      successCondition: result[0] >= 0.99
      provider:
        prometheus:
          query: |
            sum(rate(http_requests_total{status!~"5.."}[2m]))
              / sum(rate(http_requests_total[2m]))

def evaluate_canary(canary, stable, min_success=0.99, max_latency_ms=800, latency_tol=1.2):
    if canary['success_rate'] < min_success:
        return 'abort', 'low success'
    if canary['p95_ms'] > max_latency_ms:
        return 'abort', 'high latency (absolute)'
    if canary['p95_ms'] > stable['p95_ms'] * latency_tol:
        return 'abort', 'regression vs stable'
    return 'promote', 'healthy'

http:
  - match: [ { headers: { x-dogfood: { exact: "true" } } } ]
    route: [ { destination: { host: llm-api, subset: canary } } ]   # dogfooders -> canary
  - route:
      - { destination: { host: llm-api, subset: stable }, weight: 95 }
      - { destination: { host: llm-api, subset: canary }, weight: 5 }