Kubernetes Orchestration · Capstone

Project · Ship an LLM app to K8s

Q: On launch day the app is live and serving, but every deploy still happens by an engineer running kubectl apply from their laptop, and the running state matches nobody's record. Which go-live item is failing and why does it matter under load?

The delivery item — deploys aren't GitOps-driven, so there is no single source of truth for what's running, no audit trail, and rollback is manual and error-prone. Under an incident that's exactly when you can least afford ambiguity about the deployed state or a slow, hand-typed rollback. Move delivery to Argo CD syncing from a git repo (K6): the cluster becomes a projection of git, every change is a reviewed commit, and rollback is git revert — fast and unambiguous when it matters most.

Q: Your RAG app passes every functional test but, in the go-live review, the security lead blocks launch. The Secret manifest with a real provider API key is committed in the deploy repo. Why is this a hard blocker even though 'it's just base64', and what's the fix?

A Kubernetes Secret is only base64-encoded, not encrypted — committing the manifest puts the real API key in git history in effectively plaintext, readable by anyone with repo access and impossible to fully scrub from history. That's a credential leak, not a formatting detail, and it's rightly a launch blocker. The fix: remove the value from git (and rotate the key, since it's compromised), pull the secret at runtime from a managed store (AWS Secrets Manager/Vault via a CSI driver), enable etcd encryption at rest, and keep only a non-sensitive placeholder in the repo.

The capstone that ties the whole track together. You'll take a containerized LLM/RAG app from a Dockerfile all the way to production on Kubernetes: manifests → config/secrets → Service → Ingress → probes → HPA → EKS → Helm/GitOps → observability. It's a build brief, not a lecture — you assemble everything from K1–K7. Ends with a self-assessment rubric and a go-live checklist you can actually run before flipping traffic on.

⏱️ ~3 hours🏗️ End-to-end build🎯 Intermediate→Professional

🌱 What you're shippingA real, if small, service: a RAG API (retrieval + an LLM call) with a health endpoint, externalized config and secrets, exposed over HTTPS, autoscaled, and delivered by GitOps to a managed cluster. Local steps run on kind/minikube; the EKS and GPU steps need a cloud cluster and cost money — tear down when done. Verify every AWS/Helm/Argo/GPU specific against its current docs; those drift.

Learning objectives

Assemble a full manifest set for an LLM/RAG app: Deployment, Service, ConfigMap, Secret, Ingress.
Make it production-ready: probes, resources, rolling strategy, HPA, PDB.
Take it to EKS with ECR images and IRSA for AWS access.
Package it as a Helm chart and deliver it via Argo CD (GitOps).
Add observability, then pass a go-live checklist before serving traffic.

1 · The target architecture

Here's the failure this section exists to prevent. You've done every K1–K7 chapter, so you open your editor and start typing YAML — a Deployment first, because that's the piece you remember best. Halfway through you realize you don't know whether the LLM lives in the cluster or behind an API, whether the vector store is a Pod or a managed service, or how traffic even reaches the app. So you guess, wire it up, and spend the afternoon untangling a Service that points at nothing and an Ingress with no TLS. The YAML was never the hard part; not knowing the shape first was.

Think of it like wiring a house before you frame the walls. An electrician doesn't start pulling cable and hope the rooms appear around it — they work from a floor plan that already says where the kitchen, the panel, and the outlets go. The architecture diagram is that floor plan: it fixes what talks to what and where the trust boundary sits before a single line of config exists. Every manifest you write later is just wiring one labelled edge of this picture.

The precise point: a production LLM app on Kubernetes is not one object, it's a small graph of collaborating pieces, and the edges between them are what you're really deploying. A public Ingress is the only thing exposed to the internet; behind it a ClusterIP Service load-balances to the app's Pods so nothing external ever addresses a Pod directly; the app pulls its non-secret settings from a ConfigMap and its credentials from a secret store; and it reaches out to two heavyweight dependencies — the model (a managed API like Bedrock, or an in-cluster vLLM server) and the vector store. Naming those boundaries now is what lets you reason about blast radius, cost, and failure later — the internal pieces stay private, and only the front door is public.

The common mistake this framing kills is treating the model and the vector DB as "just more containers." They're the opposite: they're the expensive, stateful, slow-to-start parts, and the whole reason for the split. The stateless RAG app scales freely on the HPA; the stateful vector store is a managed service or a StatefulSet, and the model is either someone else's problem (an API) or a GPU node pool you schedule carefully. Draw that line correctly here and Phases A–E become assembly; blur it and you'll fight it every phase.

Before writing YAML, know the shape. A public Ingress fronts the app; the app is a Deployment behind a ClusterIP Service; it reads config from a ConfigMap and secrets from a secret store; it calls an LLM (a managed API like Bedrock, or an in-cluster vLLM server from K7) and a vector store. State (the vector DB) is a managed service or a StatefulSet.

🗺️ How to read this diagram

Read it left-to-right, following the arrows: a request enters at the far left and passes through each box in turn until it reaches the dependencies on the right.

The blue "Internet (HTTPS)" box on the far left is the outside world — the only thing that ever addresses the app from outside the cluster.
The brand-coloured arrows trace the single request path: Internet → the purple Ingress (the one public door) → the amber "RAG app Deployment" — nothing external ever reaches a Pod directly.
The green "LLM + vector store" box on the right is the payoff: the two heavyweight, expensive dependencies the app calls out to — a managed model API or in-cluster vLLM, and the vector store — kept private behind the app.

In short: Only the Ingress is public; behind it the app fronts private, heavyweight dependencies — that shape is what every later manifest just wires up.

Reuse, don't reinventEvery piece here is something you built in K1–K7. This capstone is assembly: the Deployment+Service (K2), probes/resources/HPA/PDB (K3), Ingress/storage (K4), EKS/ECR/IRSA (K5), Helm/Argo CD (K6), and — if self-hosting the model — GPU scheduling and safe ops (K7).

2 · Phase A — the app and its manifests

Containerize the RAG app (CD track), then write the core objects. Externalize everything environment-specific: the LLM endpoint and model name in a ConfigMap, the API key in a Secret.

Config and secrets flow in; traffic flows through the Service. The Deployment runs the Pods; a ConfigMap and a Secret are injected via envFrom, and a ClusterIP Service load-balances port 80 to container port 8000.

🗺️ How to read this diagram

Read it left-to-right: the two config sources on the left feed the Deployment in the middle, which the Service on the right exposes.

The blue ConfigMap and red Secret boxes on the left hold the non-sensitive settings and the credential, kept separate on purpose.
The arrows labelled envFrom show both being injected as environment variables into the app container — config in git, secret never in git.
The amber Deployment runs the replica Pods; the Service on the right maps its port 80 to the container's 8000 and load-balances across the Pods using the app: rag selector.

In short: Four objects — ConfigMap + Secret feed the Deployment, the Service fronts it — and matching labels are the glue that connects them.

Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.

YAML · the core Deployment + Service (needs a cluster)

rag.yamlapiVersion: apps/v1
kind: Deployment
metadata: { name: rag, labels: { app: rag } }
spec:
  replicas: 3
  selector: { matchLabels: { app: rag } }
  strategy:
    rollingUpdate: { maxUnavailable: 0, maxSurge: 1 }   # zero-downtime
  template:
    metadata: { labels: { app: rag } }
    spec:
      serviceAccountName: rag            # IRSA-backed on EKS (Phase C)
      containers:
        - name: app
          image: <acct>.dkr.ecr.us-east-1.amazonaws.com/rag:1.0.0
          ports: [ { containerPort: 8000 } ]
          envFrom:
            - configMapRef: { name: rag-config }
            - secretRef:    { name: rag-secret }
          readinessProbe: { httpGet: { path: /healthz/ready, port: 8000 }, periodSeconds: 5 }
          livenessProbe:  { httpGet: { path: /healthz/live,  port: 8000 }, periodSeconds: 10 }
          resources:
            requests: { cpu: "250m", memory: "512Mi" }
            limits:   { cpu: "1",    memory: "1Gi" }
---
apiVersion: v1
kind: Service
metadata: { name: rag }
spec:
  selector: { app: rag }
  ports: [ { port: 80, targetPort: 8000 } ]   # ClusterIP

3 · Phase B — expose, scale, protect

Add the reliability layer: an Ingress with TLS for the front door, an HPA to follow load, and a PDB so maintenance can't breach quorum.

One app, three protections. The Ingress is the only public door (TLS-terminated); the HPA grows and shrinks replicas with load; the PDB guarantees a minimum stay up during maintenance.

🗺️ How to read this diagram

Read the flow left-to-right for traffic, then look at the two controls on the right that act on the Deployment.

The Ingress box on the left is the internet-facing front door — it terminates TLS and is the only thing exposed publicly.
The amber Deployment in the middle is the same RAG app from Phase A; the arrow into it carries the incoming HTTPS traffic.
The blue HPA above and green PDB below both point at the Deployment: the HPA changes replica count with CPU load (3→12), while the PDB keeps at least 2 Pods available so a drain or upgrade can't take the service down.

In short: Ingress opens the door safely, HPA sizes the pool to demand, and PDB keeps the pool from being drained to zero — reliability without touching the app code.

YAML · Ingress + HPA + PDB (needs a cluster + ingress controller + metrics)

expose.yamlapiVersion: networking.k8s.io/v1
kind: Ingress
metadata: { name: rag, annotations: { kubernetes.io/ingress.class: alb } }
spec:
  tls: [ { hosts: [rag.example.com], secretName: rag-tls } ]
  rules:
    - host: rag.example.com
      http:
        paths:
          - { path: /, pathType: Prefix, backend: { service: { name: rag, port: { number: 80 } } } }
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: rag }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: rag }
  minReplicas: 3
  maxReplicas: 12
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 65 } }
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: rag }
spec: { minAvailable: 2, selector: { matchLabels: { app: rag } } }

Pick the autoscaling signal for YOUR loadCPU is fine if the app is CPU-bound. If the RAG app is latency-sensitive or calls a slow LLM, scale on queue depth / in-flight requests / p95 latency instead (K3/K7) — CPU will react too late. And the HPA needs a CPU request set to work at all. Verify the autoscaling API version against current docs.

4 · Phase C — EKS, ECR, IRSA

Move from local to a managed cluster. Push the image to ECR, create the cluster with eksctl, and give the app's ServiceAccount exactly the AWS permissions it needs via IRSA — never node-role permissions (K5).

IRSA gives the Pod its own least-privilege identity. The rag ServiceAccount exchanges an OIDC token for a scoped IAM role, so the app reaches Bedrock with exactly the permissions it needs — no keys, no node-role blast radius.

🗺️ How to read this diagram

Follow the top row left-to-right — that is the identity chain — then note the image pull coming up from below.

The amber Pod on the left runs under the rag ServiceAccount; it holds no static AWS keys.
The OIDC arrow to the IAM role box is IRSA in action: the SA token is exchanged for a scoped role (here, Bedrock read-only) that the Pod assumes at runtime.
The green Bedrock box is the AWS service the app calls — reachable only because the role grants it. The ECR box below shows where the versioned container image is pulled from.

In short: The image comes from ECR and the AWS permissions come from a role scoped to the ServiceAccount — the workload YAML barely changes, but the credential model becomes least-privilege.

shell · to EKS with ECR + IRSA (needs an AWS account — verify vs EKS docs)

eks.sheksctl create cluster -f cluster.yaml            # OIDC enabled (for IRSA)
aws eks update-kubeconfig --name ai-platform --region us-east-1

aws ecr create-repository --repository-name rag
# ...docker login to ECR, build, push rag:1.0.0 (see K5)...

eksctl create iamserviceaccount \
  --cluster ai-platform --namespace default --name rag \
  --attach-policy-arn arn:aws:iam::aws:policy/AmazonBedrockReadOnly --approve
# the 'rag' ServiceAccount now carries scoped AWS creds; the Deployment uses it.
# Real secrets: pull from AWS Secrets Manager via a CSI driver, not plain manifests.

5 · Phase D — Helm + GitOps delivery

Stop applying by hand. Package the manifests as a Helm chart with per-environment values, put them in a deploy repo, and let Argo CD sync the cluster to git (K6). Now deploys are commits and rollback is git revert.

Deploys become commits. A merge to the deploy repo is what Argo CD reconciles onto the cluster; because the cluster is a projection of git, rollback is just reverting the commit and letting Argo re-sync the previous state.

🗺️ How to read this diagram

Read the top row left-to-right for a deploy; the amber arrow underneath is the reverse path for a rollback.

The git deploy repo on the left holds the Helm chart and per-environment values — the single source of truth for what should run.
The Argo CD box in the middle continuously syncs the cluster to git, pruning removed objects and self-healing manual drift.
The green EKS cluster on the right ends up matching git exactly. The amber arrow back to git shows rollback: git revert the deploy commit and Argo re-syncs the prior state — no hand-typed kubectl.

In short: Change the cluster by changing git; undo a change by reverting the commit — auditable, reviewable, and self-healing.

YAML · the Argo CD Application that delivers the chart (verify CRD vs Argo docs)

rag-application.yamlapiVersion: argoproj.io/v1alpha1
kind: Application
metadata: { name: rag, namespace: argocd }
spec:
  project: default
  source:
    repoURL: https://github.com/example/deploy.git
    targetRevision: main
    path: rag-chart
    helm: { valueFiles: [values-prod.yaml] }
  destination: { server: https://kubernetes.default.svc, namespace: default }
  syncPolicy: { automated: { prune: true, selfHeal: true } }

6 · Phase E — observability

You can't operate what you can't see. Ship logs (CloudWatch/Fluent Bit), metrics (Container Insights or Prometheus+Grafana), and — because this is an LLM app — traces and token/cost metrics so you can see latency, error rate, and spend per request. Wire the ch08 / K7 diagnosis agent to read this signal (read-only).

Signal in three flavours, two audiences. The app emits logs, metrics, and LLM-specific traces (tokens/cost); those feed human dashboards/alerts and a read-only diagnosis agent — the same evidence, consumed by people and by the AIOps assistant.

🗺️ How to read this diagram

Read left-to-right: the app on the left emits three signal types in the middle, which fan out to the two consumers on the right.

The amber RAG app is the source; the three blue boxes are the signals it ships — logs, metrics, and the LLM-specific traces (tokens and cost per request).
The arrows into the signal column show every flavour being exported; the arrows out show them landing where they're used.
The dashboards/alerts box is for humans watching p95, error rate and spend; the green diagnosis agent reads the same data but strictly read-only — it explains, it doesn't act.

In short: Emit logs + metrics + token/cost traces once, then serve both the on-call human and the read-only AIOps agent from the same signal.

LLM apps need LLM-specific signalsBeyond CPU and error rate, track tokens per request, cost per request, p95 latency, and refusal/failure rate. These are what tell you the RAG app is healthy in the way users care about — and what an AIOps agent (K7) reasons over. Keep any agent that can act behind RBAC + dry-run + approval.

7 · Self-assessment rubric

Grade your build honestly against a production bar. "Meets" is shippable; "Above" is what a platform team would be proud of.

Two lines, three tiers. "Meets" is the shippable floor and "Above" is platform-grade; anything still "Not yet" on Health, Secrets, Exposure, or Safety sits below the launch line and blocks go-live.

🗺️ How to read this diagram

Read it bottom-to-top: the arrow points from worst to best, and each band is one grade you assign per rubric row.

The red band at the bottom is "Not yet" — and for Health, Secrets, Exposure, or Safety it is a hard launch blocker, not a nice-to-have.
The blue middle band is "Meets the bar": the build is genuinely shippable at this level.
The green top band is "Above the bar" — the polish a platform team would be proud of. The upward arrow is the direction you push each dimension over time.

In short: Grade each row into one of three bands; clear every red one on the critical dimensions before you flip traffic on.

📋 Grade your Kubernetes LLM-app deployment

Dimension	Meets the bar	Above the bar
Manifests	Deployment+Service work; labels/selectors match	Templated Helm chart, per-env values, no drift
Config & secrets	Externalized to ConfigMap/Secret	Secrets from a managed store (Secrets Manager/Vault), none in git
Health	Readiness + liveness probes correct	Startup probe for slow model load; probes never touch heavy deps
Resources	Requests + limits set	Sized from a profile; memory limit above peak; sensible QoS
Exposure	Reachable over HTTPS via Ingress	TLS automated; only the frontend is public; internal stays ClusterIP
Scaling	HPA present and functional	Scales on an inference-appropriate metric; PDB protects quorum
AWS integration	Runs on EKS from ECR	IRSA least-privilege; no node-role creds; cost controls in place
Delivery	Repeatable deploy	GitOps: git is truth, self-heal on, rollback = git revert
Observability	Logs + basic metrics	LLM-specific signals (tokens, cost, p95); traces; alerting
Safety	Secrets protected, RBAC sane	Any ops agent is diagnosis-first, RBAC-bounded, approval-gated

Score each row Meets / Above / Not yet. Any 'Not yet' on Health, Secrets, Exposure, or Safety is a launch blocker — fix before go-live.

8 · Go-live checklist

Run this the day you flip traffic on. Every item is something earlier in the track taught you to produce.

The checklist is a decision gate. All items green with no "Not yet" on Health, Secrets, Exposure, or Safety → launch; any critical blocker routes to fix-and-retry, never to a hopeful go-live.

🗺️ How to read this diagram

Read left-to-right and follow the branch: the checks feed a single yes/no decision that either launches you or sends you back.

The blue box on the left is the whole checklist — every item is scored before you decide anything.
The purple decision box asks the only question that matters at the gate: is anything still "Not yet" on the four critical dimensions?
The green "no" path leads to Launch; the amber "yes" path loops to "Fix, re-run" — a blocker is never waved through, it is cleared and re-checked.

In short: Treat go-live as a gate: green everywhere on the critical items or you fix and re-run — there is no third option.

✓ Checkpoint — you can move on when you can…

Images are pinned to immutable version tags (never :latest) and pushed to ECR.
Config is in ConfigMaps; secrets come from a managed store, and no secret is committed to git.
Readiness gates traffic and liveness is cheap/dependency-free; slow model load has a startup probe.
Requests/limits are set from a profile; the memory limit sits above observed peak.
Rolling strategy is zero-downtime (maxUnavailable: 0) and rollback is tested.
The HPA scales on a signal that matches the load, and a PDB protects quorum during maintenance.
Only the intended service is public via HTTPS; internal services are ClusterIP with NetworkPolicies.
AWS access is via least-privilege IRSA; no broad permissions on the node role.
Delivery is GitOps: the cluster matches git, self-heal is on, and rollback = git revert.
Logs, metrics, and LLM-specific signals (tokens, cost, p95, error rate) are flowing with alerts set.
Any ops/AIOps agent is diagnosis-first and, if it can act, bounded by RBAC + dry-run + human approval.
Non-prod clusters have a teardown/cost alarm so a forgotten cluster can't run up a bill.

✓ Knowledge check

On launch day the app is live and serving, but every deploy still happens by an engineer running kubectl apply from their laptop, and the running state matches nobody's record. Which go-live item is failing and why does it matter under load?

Show answer

The delivery item — deploys aren't GitOps-driven, so there is no single source of truth for what's running, no audit trail, and rollback is manual and error-prone. Under an incident that's exactly when you can least afford ambiguity about the deployed state or a slow, hand-typed rollback. Move delivery to Argo CD syncing from a git repo (K6): the cluster becomes a projection of git, every change is a reviewed commit, and rollback is git revert — fast and unambiguous when it matters most.

✓ Knowledge check

Your RAG app passes every functional test but, in the go-live review, the security lead blocks launch. The Secret manifest with a real provider API key is committed in the deploy repo. Why is this a hard blocker even though 'it's just base64', and what's the fix?

Show answer

A Kubernetes Secret is only base64-encoded, not encrypted — committing the manifest puts the real API key in git history in effectively plaintext, readable by anyone with repo access and impossible to fully scrub from history. That's a credential leak, not a formatting detail, and it's rightly a launch blocker. The fix: remove the value from git (and rotate the key, since it's compromised), pull the secret at runtime from a managed store (AWS Secrets Manager/Vault via a CSI driver), enable etcd encryption at rest, and keep only a non-sensitive placeholder in the repo.

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · Assemble the core manifest setBeginner

Context: The first milestone is a working Deployment+Service+config for the RAG app on a local cluster.

Your task: Write the Deployment, Service, ConfigMap, and Secret for the RAG app and deploy it to a local kind/minikube cluster.

Requirements:

Deployment of the RAG image with matching labels/selectors
ClusterIP Service mapping 80 → 8000
ConfigMap for LLM endpoint + model name; Secret for the API key
Consume both via envFrom; pin the image version

💡 Hint: Get it Running and reachable in-cluster before adding any production concerns.

Show solution

Wire config in via envFrom and tie everything with the app: rag label:

apiVersion: apps/v1
kind: Deployment
metadata: { name: rag, labels: { app: rag } }
spec:
  replicas: 2
  selector: { matchLabels: { app: rag } }
  template:
    metadata: { labels: { app: rag } }
    spec:
      containers:
        - name: app
          image: ghcr.io/example/rag:1.0.0
          ports: [ { containerPort: 8000 } ]
          envFrom:
            - configMapRef: { name: rag-config }
            - secretRef:    { name: rag-secret }
---
apiVersion: v1
kind: Service
metadata: { name: rag }
spec: { selector: { app: rag }, ports: [ { port: 80, targetPort: 8000 } ] }

Apply, then confirm with kubectl get deploy,svc,pods -l app=rag and a port-forward curl to /healthz/ready. Keep the real API key out of git — use a placeholder locally.

Exercise 2 · Make it production-readyIntermediate

Context: A running app isn't a shippable one; the reliability layer is what a review demands.

Your task: Add probes, resources, a zero-downtime rolling strategy, an HPA, and a PDB to the RAG Deployment, each justified in a line.

Requirements:

Readiness + liveness (+ startup if model load is slow)
Requests and limits sized sensibly
Rolling strategy with maxUnavailable: 0
HPA and PDB with appropriate bounds

💡 Hint: Each control should map to a specific failure it prevents.

Show solution

Layer the K3 controls onto the container and Deployment:

          readinessProbe: { httpGet: { path: /healthz/ready, port: 8000 }, periodSeconds: 5 }
          livenessProbe:  { httpGet: { path: /healthz/live,  port: 8000 }, periodSeconds: 10 }
          resources:
            requests: { cpu: "250m", memory: "512Mi" }
            limits:   { cpu: "1",    memory: "1Gi" }
  strategy: { rollingUpdate: { maxUnavailable: 0, maxSurge: 1 } }

Plus an HPA (min 3, max 12) and a PDB (minAvailable: 2).

Justifications: readiness → no traffic to a cold Pod; liveness → auto-recover a wedged one; requests → the scheduler reserves capacity; memory limit above peak → no surprise OOM; maxUnavailable: 0 → zero-downtime deploys; HPA → capacity follows load; PDB → maintenance can't breach quorum. If the model warms slowly on boot, add a startup probe so liveness doesn't crash-loop it.

Exercise 3 · Take it to EKS with ECR + IRSAAdvanced

Context: Moving to a managed cluster with correct AWS integration is the step that makes it real production.

Your task: Give the plan and commands to run the RAG app on EKS: image in ECR, cluster via eksctl, and AWS access via IRSA — and state what changes from the local manifests.

Requirements:

Push the versioned image to ECR and update the image URI
Create the cluster with OIDC enabled
Create an IRSA ServiceAccount scoped to only the AWS actions needed
State the secret-handling change for prod

💡 Hint: The workload YAML barely changes; registry, IAM, and secrets do.

Show solution

eksctl create cluster -f cluster.yaml            # OIDC on
aws eks update-kubeconfig --name ai-platform --region us-east-1
aws ecr create-repository --repository-name rag
# docker login to ECR, build, push rag:1.0.0
eksctl create iamserviceaccount --cluster ai-platform \
  --namespace default --name rag \
  --attach-policy-arn arn:aws:iam::aws:policy/AmazonBedrockReadOnly --approve

What changes from local: the image URI points at ECR; the Deployment sets serviceAccountName: rag so it inherits the IRSA-scoped credentials (never node-role permissions); and real secrets move from a plain manifest into AWS Secrets Manager pulled via a CSI driver, with only placeholders in git. The Deployment/Service/probes/HPA/PDB YAML is otherwise identical — the payoff of standard Kubernetes. Verify every AWS command against current EKS docs.

Exercise 4 · Deliver it via Helm + Argo CDExpert

Context: GitOps delivery is what makes the deployment reproducible, auditable, and self-healing.

Your task: Package the app as a Helm chart with per-environment values and define an Argo CD Application that delivers it, then describe the new deploy and rollback workflow.

Requirements:

Chart parameterizes image tag, replicas, resources, host
values-staging / values-prod overrides
Argo CD Application with prune + selfHeal
State how a deploy and a rollback now happen

💡 Hint: The engineer commits; Argo CD deploys. Rollback is git revert.

Show solution

Chart the manifests (parameterize image tag, replicas, resources, host) and point an Argo CD Application at the deploy repo:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: { name: rag, namespace: argocd }
spec:
  source:
    repoURL: https://github.com/example/deploy.git
    targetRevision: main
    path: rag-chart
    helm: { valueFiles: [values-prod.yaml] }
  destination: { server: https://kubernetes.default.svc, namespace: default }
  syncPolicy: { automated: { prune: true, selfHeal: true } }

New deploy workflow: CI builds+tests+pushes the image and opens a PR bumping image.tag in the environment's values file; on merge, Argo CD syncs the cluster to git. No one runs kubectl apply, and self-heal reverts any manual drift.

Rollback: git revert the deploy commit — Argo CD syncs the previous state back automatically. Promotion staging → prod is a reviewed PR copying the tested tag forward. Verify Helm/Argo syntax against current docs.

Exercise 5 · Run the go-live review and pass the rubricProfessional

Context: The professional deliverable is a defensible go-live: every rubric row addressed and every blocker cleared before traffic flows.

Your task: Walk the go-live checklist against your build, identify what's below the bar, and produce the punch list that clears the launch blockers.

Requirements:

Score each rubric dimension Meets / Above / Not yet
Flag any 'Not yet' on Health, Secrets, Exposure, or Safety as a blocker
Produce the concrete fix for each blocker
State the observability signals you'll watch in the first hour

💡 Hint: Blockers first; nice-to-haves after launch.

Show solution

Score, then triage. Walk the rubric; anything 'Not yet' on Health, Secrets, Exposure, or Safety is a hard blocker and gets fixed before launch; other gaps become fast-follows.

Typical blockers and fixes: a committed real secret → remove from git, rotate the key, move to Secrets Manager (CSI), enable etcd encryption at rest; liveness hitting a heavy dependency → repoint it to a cheap in-process endpoint and add a startup probe; the app publicly exposing an internal endpoint → move internal traffic to ClusterIP + NetworkPolicy; broad node-role AWS perms → replace with least-privilege IRSA.

First-hour watch: p95 latency, error/refusal rate, HPA replica count vs load, GPU/CPU utilization, and tokens + cost per request — the LLM-specific signals that reveal a runaway spend or a degraded-quality regression the moment it starts. Have the rollback (git revert) ready and the diagnosis agent (read-only) pointed at the logs.

Exercise 6 · Ship, then operate, the LLM app for a real launchIndustry scenario

Context: Representative scenario: your RAG service launches to real users next week; you own it end-to-end from final review through the first week of operation.

Your task: Produce the full launch-and-operate plan: the deployment shape, the go-live gate, the first-week operating posture, and how an AIOps agent assists safely.

Requirements:

Summarize the production deployment (all phases) in a few lines
State the go-live gate (what must be green)
Define the first-week operating posture: dashboards, alerts, on-call, rollback
Bound the AIOps agent's role and its safety controls
Name the top three risks and how each is mitigated

💡 Hint: Tie every operating decision back to a control you built in K2–K7.

Show solution

Deployment shape: a Helm-charted RAG Deployment on EKS (images from ECR, AWS access via IRSA), fronted by an HTTPS ALB Ingress, internal pieces on ClusterIP behind NetworkPolicies, made resilient with probes / requests+limits / zero-downtime rollout / HPA / PDB, and delivered by Argo CD GitOps. Secrets come from Secrets Manager; the vector store is managed or a StatefulSet. If self-hosting the model, GPU node pool with taints+tolerations (K7).

Go-live gate: every checklist item green, with no 'Not yet' on Health, Secrets, Exposure, or Safety; a tested git revert rollback; and dashboards/alerts live.

First-week posture: dashboards for p95 latency, error/refusal rate, HPA behavior, and tokens+cost per request; alerts on latency, error rate, and cost spikes; an on-call owner; a documented rollback (git revert) and break-glass procedure; a daily review of spend and quality signals.

AIOps agent: diagnosis-first — read-only kubectl tools feeding an evidence bundle to the LLM so it explains incidents and proposes fixes (ties to ch08). It may only act behind enforced controls: least-privilege RBAC (no delete/prod-wide scope), dry-run previews, and human approval / PR-only output. Its blast radius is its RBAC, not its prompt.

Top three risks & mitigations: (1) cost runaway from bursty LLM calls → cost alarms, token/cost dashboards, HPA bounds, Spot for batch; (2) bad deploy → zero-downtime rollout gated by readiness + instant git-revert rollback; (3) credential/agent misuse → secrets in a managed store, IRSA least-privilege, and the agent bounded by RBAC+dry-run+approval. Verify all AWS/Helm/Argo/GPU specifics against current docs before launch.

Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.

Learning objectives

1 · The target architecture

2 · Phase A — the app and its manifests

3 · Phase B — expose, scale, protect

4 · Phase C — EKS, ECR, IRSA

5 · Phase D — Helm + GitOps delivery

6 · Phase E — observability

7 · Self-assessment rubric

8 · Go-live checklist

✓ Checkpoint — you can move on when you can…

🪜 Practice ladder beginner → industry

More in Projects