Kubernetes Orchestration · Capstone

Project · Ship an LLM app to K8s

Q: On launch day the app is live and serving, but every deploy still happens by an engineer running kubectl apply from their laptop, and the running state matches nobody's record. Which go-live item is failing and why does it matter under load?

The delivery item — deploys aren't GitOps-driven, so there is no single source of truth for what's running, no audit trail, and rollback is manual and error-prone. Under an incident that's exactly when you can least afford ambiguity about the deployed state or a slow, hand-typed rollback. Move delivery to Argo CD syncing from a git repo (K6): the cluster becomes a projection of git, every change is a reviewed commit, and rollback is git revert — fast and unambiguous when it matters most.

Q: Your RAG app passes every functional test but, in the go-live review, the security lead blocks launch. The Secret manifest with a real provider API key is committed in the deploy repo. Why is this a hard blocker even though 'it's just base64', and what's the fix?

A Kubernetes Secret is only base64-encoded, not encrypted — committing the manifest puts the real API key in git history in effectively plaintext, readable by anyone with repo access and impossible to fully scrub from history. That's a credential leak, not a formatting detail, and it's rightly a launch blocker. The fix: remove the value from git (and rotate the key, since it's compromised), pull the secret at runtime from a managed store (AWS Secrets Manager/Vault via a CSI driver), enable etcd encryption at rest, and keep only a non-sensitive placeholder in the repo.

The capstone that ties the whole track together. You'll take a containerized LLM/RAG app from a Dockerfile all the way to production on Kubernetes: manifests → config/secrets → Service → Ingress → probes → HPA → EKS → Helm/GitOps → observability. It's a build brief, not a lecture — you assemble everything from K1–K7. Ends with a self-assessment rubric and a go-live checklist you can actually run before flipping traffic on.

⏱️ ~3 hours🏗️ End-to-end build🎯 Intermediate→Professional

🌱 What you're shippingA real, if small, service: a RAG API (retrieval + an LLM call) with a health endpoint, externalized config and secrets, exposed over HTTPS, autoscaled, and delivered by GitOps to a managed cluster. Local steps run on kind/minikube; the EKS and GPU steps need a cloud cluster and cost money — tear down when done. Verify every AWS/Helm/Argo/GPU specific against its current docs; those drift.

Learning objectives

Assemble a full manifest set for an LLM/RAG app: Deployment, Service, ConfigMap, Secret, Ingress.
Make it production-ready: probes, resources, rolling strategy, HPA, PDB.
Take it to EKS with ECR images and IRSA for AWS access.
Package it as a Helm chart and deliver it via Argo CD (GitOps).
Add observability, then pass a go-live checklist before serving traffic.

1 · The target architecture

Before writing YAML, know the shape. A public Ingress fronts the app; the app is a Deployment behind a ClusterIP Service; it reads config from a ConfigMap and secrets from a secret store; it calls an LLM (a managed API like Bedrock, or an in-cluster vLLM server from K7) and a vector store. State (the vector DB) is a managed service or a StatefulSet.

Reuse, don't reinventEvery piece here is something you built in K1–K7. This capstone is assembly: the Deployment+Service (K2), probes/resources/HPA/PDB (K3), Ingress/storage (K4), EKS/ECR/IRSA (K5), Helm/Argo CD (K6), and — if self-hosting the model — GPU scheduling and safe ops (K7).

2 · Phase A — the app and its manifests

Containerize the RAG app (CD track), then write the core objects. Externalize everything environment-specific: the LLM endpoint and model name in a ConfigMap, the API key in a Secret.

Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.

YAML · the core Deployment + Service (needs a cluster)

rag.yamlapiVersion: apps/v1
kind: Deployment
metadata: { name: rag, labels: { app: rag } }
spec:
  replicas: 3
  selector: { matchLabels: { app: rag } }
  strategy:
    rollingUpdate: { maxUnavailable: 0, maxSurge: 1 }   # zero-downtime
  template:
    metadata: { labels: { app: rag } }
    spec:
      serviceAccountName: rag            # IRSA-backed on EKS (Phase C)
      containers:
        - name: app
          image: <acct>.dkr.ecr.us-east-1.amazonaws.com/rag:1.0.0
          ports: [ { containerPort: 8000 } ]
          envFrom:
            - configMapRef: { name: rag-config }
            - secretRef:    { name: rag-secret }
          readinessProbe: { httpGet: { path: /healthz/ready, port: 8000 }, periodSeconds: 5 }
          livenessProbe:  { httpGet: { path: /healthz/live,  port: 8000 }, periodSeconds: 10 }
          resources:
            requests: { cpu: "250m", memory: "512Mi" }
            limits:   { cpu: "1",    memory: "1Gi" }
---
apiVersion: v1
kind: Service
metadata: { name: rag }
spec:
  selector: { app: rag }
  ports: [ { port: 80, targetPort: 8000 } ]   # ClusterIP

3 · Phase B — expose, scale, protect

Add the reliability layer: an Ingress with TLS for the front door, an HPA to follow load, and a PDB so maintenance can't breach quorum.

YAML · Ingress + HPA + PDB (needs a cluster + ingress controller + metrics)

expose.yamlapiVersion: networking.k8s.io/v1
kind: Ingress
metadata: { name: rag, annotations: { kubernetes.io/ingress.class: alb } }
spec:
  tls: [ { hosts: [rag.example.com], secretName: rag-tls } ]
  rules:
    - host: rag.example.com
      http:
        paths:
          - { path: /, pathType: Prefix, backend: { service: { name: rag, port: { number: 80 } } } }
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: rag }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: rag }
  minReplicas: 3
  maxReplicas: 12
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 65 } }
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: rag }
spec: { minAvailable: 2, selector: { matchLabels: { app: rag } } }

Pick the autoscaling signal for YOUR loadCPU is fine if the app is CPU-bound. If the RAG app is latency-sensitive or calls a slow LLM, scale on queue depth / in-flight requests / p95 latency instead (K3/K7) — CPU will react too late. And the HPA needs a CPU request set to work at all. Verify the autoscaling API version against current docs.

4 · Phase C — EKS, ECR, IRSA

Move from local to a managed cluster. Push the image to ECR, create the cluster with eksctl, and give the app's ServiceAccount exactly the AWS permissions it needs via IRSA — never node-role permissions (K5).

shell · to EKS with ECR + IRSA (needs an AWS account — verify vs EKS docs)

eks.sheksctl create cluster -f cluster.yaml            # OIDC enabled (for IRSA)
aws eks update-kubeconfig --name ai-platform --region us-east-1

aws ecr create-repository --repository-name rag
# ...docker login to ECR, build, push rag:1.0.0 (see K5)...

eksctl create iamserviceaccount \
  --cluster ai-platform --namespace default --name rag \
  --attach-policy-arn arn:aws:iam::aws:policy/AmazonBedrockReadOnly --approve
# the 'rag' ServiceAccount now carries scoped AWS creds; the Deployment uses it.
# Real secrets: pull from AWS Secrets Manager via a CSI driver, not plain manifests.

5 · Phase D — Helm + GitOps delivery

Stop applying by hand. Package the manifests as a Helm chart with per-environment values, put them in a deploy repo, and let Argo CD sync the cluster to git (K6). Now deploys are commits and rollback is git revert.

YAML · the Argo CD Application that delivers the chart (verify CRD vs Argo docs)

rag-application.yamlapiVersion: argoproj.io/v1alpha1
kind: Application
metadata: { name: rag, namespace: argocd }
spec:
  project: default
  source:
    repoURL: https://github.com/example/deploy.git
    targetRevision: main
    path: rag-chart
    helm: { valueFiles: [values-prod.yaml] }
  destination: { server: https://kubernetes.default.svc, namespace: default }
  syncPolicy: { automated: { prune: true, selfHeal: true } }

6 · Phase E — observability

You can't operate what you can't see. Ship logs (CloudWatch/Fluent Bit), metrics (Container Insights or Prometheus+Grafana), and — because this is an LLM app — traces and token/cost metrics so you can see latency, error rate, and spend per request. Wire the ch08 / K7 diagnosis agent to read this signal (read-only).

LLM apps need LLM-specific signalsBeyond CPU and error rate, track tokens per request, cost per request, p95 latency, and refusal/failure rate. These are what tell you the RAG app is healthy in the way users care about — and what an AIOps agent (K7) reasons over. Keep any agent that can act behind RBAC + dry-run + approval.

7 · Self-assessment rubric

Grade your build honestly against a production bar. "Meets" is shippable; "Above" is what a platform team would be proud of.

📋 Grade your Kubernetes LLM-app deployment

Dimension	Meets the bar	Above the bar
Manifests	Deployment+Service work; labels/selectors match	Templated Helm chart, per-env values, no drift
Config & secrets	Externalized to ConfigMap/Secret	Secrets from a managed store (Secrets Manager/Vault), none in git
Health	Readiness + liveness probes correct	Startup probe for slow model load; probes never touch heavy deps
Resources	Requests + limits set	Sized from a profile; memory limit above peak; sensible QoS
Exposure	Reachable over HTTPS via Ingress	TLS automated; only the frontend is public; internal stays ClusterIP
Scaling	HPA present and functional	Scales on an inference-appropriate metric; PDB protects quorum
AWS integration	Runs on EKS from ECR	IRSA least-privilege; no node-role creds; cost controls in place
Delivery	Repeatable deploy	GitOps: git is truth, self-heal on, rollback = git revert
Observability	Logs + basic metrics	LLM-specific signals (tokens, cost, p95); traces; alerting
Safety	Secrets protected, RBAC sane	Any ops agent is diagnosis-first, RBAC-bounded, approval-gated

Score each row Meets / Above / Not yet. Any 'Not yet' on Health, Secrets, Exposure, or Safety is a launch blocker — fix before go-live.

8 · Go-live checklist

Run this the day you flip traffic on. Every item is something earlier in the track taught you to produce.

✓ Checkpoint — you can move on when you can…

Images are pinned to immutable version tags (never :latest) and pushed to ECR.
Config is in ConfigMaps; secrets come from a managed store, and no secret is committed to git.
Readiness gates traffic and liveness is cheap/dependency-free; slow model load has a startup probe.
Requests/limits are set from a profile; the memory limit sits above observed peak.
Rolling strategy is zero-downtime (maxUnavailable: 0) and rollback is tested.
The HPA scales on a signal that matches the load, and a PDB protects quorum during maintenance.
Only the intended service is public via HTTPS; internal services are ClusterIP with NetworkPolicies.
AWS access is via least-privilege IRSA; no broad permissions on the node role.
Delivery is GitOps: the cluster matches git, self-heal is on, and rollback = git revert.
Logs, metrics, and LLM-specific signals (tokens, cost, p95, error rate) are flowing with alerts set.
Any ops/AIOps agent is diagnosis-first and, if it can act, bounded by RBAC + dry-run + human approval.
Non-prod clusters have a teardown/cost alarm so a forgotten cluster can't run up a bill.

✓ Knowledge check

On launch day the app is live and serving, but every deploy still happens by an engineer running kubectl apply from their laptop, and the running state matches nobody's record. Which go-live item is failing and why does it matter under load?

Show answer

The delivery item — deploys aren't GitOps-driven, so there is no single source of truth for what's running, no audit trail, and rollback is manual and error-prone. Under an incident that's exactly when you can least afford ambiguity about the deployed state or a slow, hand-typed rollback. Move delivery to Argo CD syncing from a git repo (K6): the cluster becomes a projection of git, every change is a reviewed commit, and rollback is git revert — fast and unambiguous when it matters most.

✓ Knowledge check

Your RAG app passes every functional test but, in the go-live review, the security lead blocks launch. The Secret manifest with a real provider API key is committed in the deploy repo. Why is this a hard blocker even though 'it's just base64', and what's the fix?

Show answer

A Kubernetes Secret is only base64-encoded, not encrypted — committing the manifest puts the real API key in git history in effectively plaintext, readable by anyone with repo access and impossible to fully scrub from history. That's a credential leak, not a formatting detail, and it's rightly a launch blocker. The fix: remove the value from git (and rotate the key, since it's compromised), pull the secret at runtime from a managed store (AWS Secrets Manager/Vault via a CSI driver), enable etcd encryption at rest, and keep only a non-sensitive placeholder in the repo.

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · Assemble the core manifest setBeginner

Context: The first milestone is a working Deployment+Service+config for the RAG app on a local cluster.

Your task: Write the Deployment, Service, ConfigMap, and Secret for the RAG app and deploy it to a local kind/minikube cluster.

Requirements:

Deployment of the RAG image with matching labels/selectors
ClusterIP Service mapping 80 → 8000
ConfigMap for LLM endpoint + model name; Secret for the API key
Consume both via envFrom; pin the image version

💡 Hint: Get it Running and reachable in-cluster before adding any production concerns.

Show solution

Wire config in via envFrom and tie everything with the app: rag label:

apiVersion: apps/v1
kind: Deployment
metadata: { name: rag, labels: { app: rag } }
spec:
  replicas: 2
  selector: { matchLabels: { app: rag } }
  template:
    metadata: { labels: { app: rag } }
    spec:
      containers:
        - name: app
          image: ghcr.io/example/rag:1.0.0
          ports: [ { containerPort: 8000 } ]
          envFrom:
            - configMapRef: { name: rag-config }
            - secretRef:    { name: rag-secret }
---
apiVersion: v1
kind: Service
metadata: { name: rag }
spec: { selector: { app: rag }, ports: [ { port: 80, targetPort: 8000 } ] }

Apply, then confirm with kubectl get deploy,svc,pods -l app=rag and a port-forward curl to /healthz/ready. Keep the real API key out of git — use a placeholder locally.

Exercise 2 · Make it production-readyIntermediate

Context: A running app isn't a shippable one; the reliability layer is what a review demands.

Your task: Add probes, resources, a zero-downtime rolling strategy, an HPA, and a PDB to the RAG Deployment, each justified in a line.

Requirements:

Readiness + liveness (+ startup if model load is slow)
Requests and limits sized sensibly
Rolling strategy with maxUnavailable: 0
HPA and PDB with appropriate bounds

💡 Hint: Each control should map to a specific failure it prevents.

Show solution

Layer the K3 controls onto the container and Deployment:

          readinessProbe: { httpGet: { path: /healthz/ready, port: 8000 }, periodSeconds: 5 }
          livenessProbe:  { httpGet: { path: /healthz/live,  port: 8000 }, periodSeconds: 10 }
          resources:
            requests: { cpu: "250m", memory: "512Mi" }
            limits:   { cpu: "1",    memory: "1Gi" }
  strategy: { rollingUpdate: { maxUnavailable: 0, maxSurge: 1 } }

Plus an HPA (min 3, max 12) and a PDB (minAvailable: 2).

Justifications: readiness → no traffic to a cold Pod; liveness → auto-recover a wedged one; requests → the scheduler reserves capacity; memory limit above peak → no surprise OOM; maxUnavailable: 0 → zero-downtime deploys; HPA → capacity follows load; PDB → maintenance can't breach quorum. If the model warms slowly on boot, add a startup probe so liveness doesn't crash-loop it.

Exercise 3 · Take it to EKS with ECR + IRSAAdvanced

Context: Moving to a managed cluster with correct AWS integration is the step that makes it real production.

Your task: Give the plan and commands to run the RAG app on EKS: image in ECR, cluster via eksctl, and AWS access via IRSA — and state what changes from the local manifests.

Requirements:

Push the versioned image to ECR and update the image URI
Create the cluster with OIDC enabled
Create an IRSA ServiceAccount scoped to only the AWS actions needed
State the secret-handling change for prod

💡 Hint: The workload YAML barely changes; registry, IAM, and secrets do.

Show solution

eksctl create cluster -f cluster.yaml            # OIDC on
aws eks update-kubeconfig --name ai-platform --region us-east-1
aws ecr create-repository --repository-name rag
# docker login to ECR, build, push rag:1.0.0
eksctl create iamserviceaccount --cluster ai-platform \
  --namespace default --name rag \
  --attach-policy-arn arn:aws:iam::aws:policy/AmazonBedrockReadOnly --approve

What changes from local: the image URI points at ECR; the Deployment sets serviceAccountName: rag so it inherits the IRSA-scoped credentials (never node-role permissions); and real secrets move from a plain manifest into AWS Secrets Manager pulled via a CSI driver, with only placeholders in git. The Deployment/Service/probes/HPA/PDB YAML is otherwise identical — the payoff of standard Kubernetes. Verify every AWS command against current EKS docs.

Exercise 4 · Deliver it via Helm + Argo CDExpert

Context: GitOps delivery is what makes the deployment reproducible, auditable, and self-healing.

Your task: Package the app as a Helm chart with per-environment values and define an Argo CD Application that delivers it, then describe the new deploy and rollback workflow.

Requirements:

Chart parameterizes image tag, replicas, resources, host
values-staging / values-prod overrides
Argo CD Application with prune + selfHeal
State how a deploy and a rollback now happen

💡 Hint: The engineer commits; Argo CD deploys. Rollback is git revert.

Show solution

Chart the manifests (parameterize image tag, replicas, resources, host) and point an Argo CD Application at the deploy repo:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: { name: rag, namespace: argocd }
spec:
  source:
    repoURL: https://github.com/example/deploy.git
    targetRevision: main
    path: rag-chart
    helm: { valueFiles: [values-prod.yaml] }
  destination: { server: https://kubernetes.default.svc, namespace: default }
  syncPolicy: { automated: { prune: true, selfHeal: true } }

New deploy workflow: CI builds+tests+pushes the image and opens a PR bumping image.tag in the environment's values file; on merge, Argo CD syncs the cluster to git. No one runs kubectl apply, and self-heal reverts any manual drift.

Rollback: git revert the deploy commit — Argo CD syncs the previous state back automatically. Promotion staging → prod is a reviewed PR copying the tested tag forward. Verify Helm/Argo syntax against current docs.

Exercise 5 · Run the go-live review and pass the rubricProfessional

Context: The professional deliverable is a defensible go-live: every rubric row addressed and every blocker cleared before traffic flows.

Your task: Walk the go-live checklist against your build, identify what's below the bar, and produce the punch list that clears the launch blockers.

Requirements:

Score each rubric dimension Meets / Above / Not yet
Flag any 'Not yet' on Health, Secrets, Exposure, or Safety as a blocker
Produce the concrete fix for each blocker
State the observability signals you'll watch in the first hour

💡 Hint: Blockers first; nice-to-haves after launch.

Show solution

Score, then triage. Walk the rubric; anything 'Not yet' on Health, Secrets, Exposure, or Safety is a hard blocker and gets fixed before launch; other gaps become fast-follows.

Typical blockers and fixes: a committed real secret → remove from git, rotate the key, move to Secrets Manager (CSI), enable etcd encryption at rest; liveness hitting a heavy dependency → repoint it to a cheap in-process endpoint and add a startup probe; the app publicly exposing an internal endpoint → move internal traffic to ClusterIP + NetworkPolicy; broad node-role AWS perms → replace with least-privilege IRSA.

First-hour watch: p95 latency, error/refusal rate, HPA replica count vs load, GPU/CPU utilization, and tokens + cost per request — the LLM-specific signals that reveal a runaway spend or a degraded-quality regression the moment it starts. Have the rollback (git revert) ready and the diagnosis agent (read-only) pointed at the logs.

Exercise 6 · Ship, then operate, the LLM app for a real launchIndustry scenario

Context: Representative scenario: your RAG service launches to real users next week; you own it end-to-end from final review through the first week of operation.

Your task: Produce the full launch-and-operate plan: the deployment shape, the go-live gate, the first-week operating posture, and how an AIOps agent assists safely.

Requirements:

Summarize the production deployment (all phases) in a few lines
State the go-live gate (what must be green)
Define the first-week operating posture: dashboards, alerts, on-call, rollback
Bound the AIOps agent's role and its safety controls
Name the top three risks and how each is mitigated

💡 Hint: Tie every operating decision back to a control you built in K2–K7.

Show solution

Deployment shape: a Helm-charted RAG Deployment on EKS (images from ECR, AWS access via IRSA), fronted by an HTTPS ALB Ingress, internal pieces on ClusterIP behind NetworkPolicies, made resilient with probes / requests+limits / zero-downtime rollout / HPA / PDB, and delivered by Argo CD GitOps. Secrets come from Secrets Manager; the vector store is managed or a StatefulSet. If self-hosting the model, GPU node pool with taints+tolerations (K7).

Go-live gate: every checklist item green, with no 'Not yet' on Health, Secrets, Exposure, or Safety; a tested git revert rollback; and dashboards/alerts live.

First-week posture: dashboards for p95 latency, error/refusal rate, HPA behavior, and tokens+cost per request; alerts on latency, error rate, and cost spikes; an on-call owner; a documented rollback (git revert) and break-glass procedure; a daily review of spend and quality signals.

AIOps agent: diagnosis-first — read-only kubectl tools feeding an evidence bundle to the LLM so it explains incidents and proposes fixes (ties to ch08). It may only act behind enforced controls: least-privilege RBAC (no delete/prod-wide scope), dry-run previews, and human approval / PR-only output. Its blast radius is its RBAC, not its prompt.

Top three risks & mitigations: (1) cost runaway from bursty LLM calls → cost alarms, token/cost dashboards, HPA bounds, Spot for batch; (2) bad deploy → zero-downtime rollout gated by readiness + instant git-revert rollback; (3) credential/agent misuse → secrets in a managed store, IRSA least-privilege, and the agent bounded by RBAC+dry-run+approval. Verify all AWS/Helm/Argo/GPU specifics against current docs before launch.

Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.

Learning objectives

1 · The target architecture

2 · Phase A — the app and its manifests

3 · Phase B — expose, scale, protect

4 · Phase C — EKS, ECR, IRSA

5 · Phase D — Helm + GitOps delivery

6 · Phase E — observability

7 · Self-assessment rubric

8 · Go-live checklist

✓ Checkpoint — you can move on when you can…

🪜 Practice ladder beginner → industry

More in Projects