Progressive delivery & service mesh

1 · Rolling vs canary vs blue-green

The default Deployment strategy is a rolling update: gradually replace old Pods with new ones. It's simple but blunt — every user hits the new version as it rolls, and a bad version is discovered only after it's widely live. Progressive delivery adds a controlled exposure step.

2 · Automating it: Argo Rollouts

Doing canary by hand — edit replicas, watch Grafana, decide — doesn't scale and isn't reliable at 3am. Argo Rollouts replaces the Deployment with a Rollout CRD that encodes the steps and the analysis, and drives them automatically. Flagger is the analogous tool (often paired with a mesh). A Rollout's steps describe the canary ramp and where to pause for an analysis gate:

rollout.yamlapiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: llm-api }
spec:
  replicas: 6
  selector: { matchLabels: { app: llm-api } }
  template: { }            # same pod template as a Deployment
  strategy:
    canary:
      steps:
        - setWeight: 10        # 10% of traffic to the new version
        - pause: { duration: 5m }
        - analysis:            # gate: run the AnalysisTemplate below
            templates: [ { templateName: success-rate } ]
        - setWeight: 50
        - pause: { duration: 5m }
        - setWeight: 100       # full promotion if analysis kept passing

analysis.yamlapiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: success-rate }
spec:
  metrics:
    - name: success-rate
      interval: 1m
      successCondition: result[0] >= 0.99     # abort the rollout if it dips
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{app="llm-api",status!~"5.."}[2m]))
              / sum(rate(http_requests_total{app="llm-api"}[2m]))

3 · Analysis-driven promotion, modeled offline

Strip away the CRDs and the promotion decision is simple arithmetic: after each step, compare the canary's SLIs to the stable's (or to an absolute threshold); promote if healthy, abort if not, with a tolerance so you don't abort on noise. Here it is as offline-runnable Python — the same logic the controller runs:

promotion.pydef evaluate_canary(canary, stable, min_success=0.99, max_latency_ms=800, latency_tol=1.2):
    """Return ('promote'|'abort', reason). canary/stable are dicts of measured SLIs."""
    # 1) Absolute success-rate floor (an SLO-style gate).
    if canary["success_rate"] < min_success:
        return "abort", f"success {canary['success_rate']:.3f} < {min_success}"
    # 2) Absolute latency ceiling.
    if canary["p95_ms"] > max_latency_ms:
        return "abort", f"p95 {canary['p95_ms']}ms > {max_latency_ms}ms"
    # 3) Relative guard: canary shouldn't be much worse than stable (catches regressions).
    if canary["p95_ms"] > stable["p95_ms"] * latency_tol:
        return "abort", f"p95 {canary['p95_ms']}ms >> stable {stable['p95_ms']}ms"
    return "promote", "canary healthy"

stable = {"success_rate": 0.999, "p95_ms": 500}
good   = {"success_rate": 0.998, "p95_ms": 540}
bad    = {"success_rate": 0.995, "p95_ms": 900}
print(evaluate_canary(good, stable))   # promote
print(evaluate_canary(bad, stable))    # abort (latency)

('promote', 'canary healthy')
('abort', 'p95 900ms > 800ms')

4 · How traffic actually shifts

'Send 10% to the canary' can mean two very different mechanisms, and it matters. The basic way is replica weighting: run 1 canary Pod and 9 stable Pods behind one Service, so roughly 10% of connections hit the canary — crude (tied to replica counts, connection-level not request-level). The precise way needs a component that can split at the request level: an ingress controller or a service mesh that routes an exact percentage.

5 · Service mesh: mTLS, traffic policy, observability

A service mesh puts a programmable network layer between your services. Classically it injects a sidecar proxy (Envoy in Istio) next to every Pod; all pod-to-pod traffic flows through the proxies, which the mesh control plane configures. That gives you three things without changing app code:

Sidecar vs sidecarless. The sidecar-per-Pod model is powerful but costs a proxy container's CPU/memory on every Pod and adds a network hop. Newer designs reduce this: Linkerd uses a deliberately tiny Rust 'micro-proxy'; Istio ambient mode moves L4 to a per-node component ('ztunnel') and makes L7 proxies optional. The trade-off is always feature richness vs overhead and complexity.

virtualservice.yamlapiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata: { name: llm-api }
spec:
  hosts: [ llm-api ]
  http:
    - route:
        - destination: { host: llm-api, subset: stable }
          weight: 90
        - destination: { host: llm-api, subset: canary }
          weight: 10          # exact 10% at the request level

6 · When a mesh is — and isn't — worth it

A mesh is powerful and not free: sidecars add resource overhead and a latency hop, the control plane is another critical system to run and upgrade, and debugging gains a layer. The honest engineering question is whether your problems actually need it.

kind: Rollout
spec:
  replicas: 6
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 5m }
        - setWeight: 100

kind: AnalysisTemplate
metadata: { name: success-rate }
spec:
  metrics:
    - name: success-rate
      successCondition: result[0] >= 0.99
      provider:
        prometheus:
          query: |
            sum(rate(http_requests_total{status!~"5.."}[2m]))
              / sum(rate(http_requests_total[2m]))

def evaluate_canary(canary, stable, min_success=0.99, max_latency_ms=800, latency_tol=1.2):
    if canary['success_rate'] < min_success:
        return 'abort', 'low success'
    if canary['p95_ms'] > max_latency_ms:
        return 'abort', 'high latency (absolute)'
    if canary['p95_ms'] > stable['p95_ms'] * latency_tol:
        return 'abort', 'regression vs stable'
    return 'promote', 'healthy'

http:
  - match: [ { headers: { x-dogfood: { exact: "true" } } } ]
    route: [ { destination: { host: llm-api, subset: canary } } ]   # dogfooders -> canary
  - route:
      - { destination: { host: llm-api, subset: stable }, weight: 95 }
      - { destination: { host: llm-api, subset: canary }, weight: 5 }