AI/ML on EKS

When EKS for AI: the EKS vs SageMaker vs Bedrock decision

Start with the meeting you've sat in. Someone says "let's just run our own model on Kubernetes — we already have a cluster, and it'll be cheaper than paying Bedrock per token." It sounds obviously right. Six weeks later a GPU node has been sitting at 8% utilisation, billing $4/hour around the clock, and the "cheaper" option has quietly out-spent the managed one it replaced — while also consuming an engineer's week to keep the cluster patched. The decision that felt like a no-brainer was actually the most expensive of the three options, chosen for the wrong reason. This section exists so you never make that call by reflex.

Here's the mental model that prevents it: think of the three services as a staircase of control, not a menu of rivals. Renting a hotel room (Bedrock) means you never see the plumbing — you pay per night and walk away. Renting a serviced apartment (SageMaker) gives you your own kitchen but the building still handles maintenance. Buying the house (EKS) gives you total freedom to knock down walls — and total responsibility for the roof, the boiler, and the property tax. Each step down trades less operational burden for more control, and — this is the part people miss — every step down also adds a bill that never sleeps.

So the rule for choosing is deliberately conservative: stay as high on the staircase as your hardest requirement allows. You only descend when the layer above genuinely cannot do the job — not because the lower layer feels more "real" or more in your control. The mistake to name out loud is descending for cost alone: a managed API that scales to zero is almost always cheaper than a GPU you rent by the hour and forget to turn off. The legitimate reasons to drop to EKS are control (a custom serving stack, sidecars, a specific runtime), consolidation (AI sharing one control plane, RBAC, and GitOps with services you already run on k8s), and portability (plain Kubernetes runs anywhere; the managed layers lock you to AWS). Cost only joins that list at genuinely high, sustained utilisation — and even then you have to prove it, not assume it. Keep the staircase in your head as you read the table below.

AWS gives you three very different ways to run AI, and the mistake is treating them as competitors when they are layers. Bedrock is a managed model API — you call a foundation model (W2) and never see a server. SageMaker (W8/W9) is a managed ML platform — you own the model but AWS owns the serving/ training plumbing. EKS is self-run Kubernetes — you own everything down to the node, in exchange for maximum control and portability. You move down that list only when the layer above genuinely cannot meet a hard requirement.

GPU node autoscaling with Karpenter

Picture the failure this solves. A traffic spike hits, ten vLLM pods go Pending because they each want a GPU, and… nothing happens. Your cluster has no spare GPU capacity, and the old autoscaler is staring at a fixed "GPU node group" you sized for average load, not peak. Users wait, pods sit unscheduled, and someone scrambles to bump the group by hand. The opposite failure is just as painful: you over-provision that group "to be safe," the spike never comes, and a rack of idle GPUs bills you all night. The whole point of GPU autoscaling is to escape both traps — capacity that appears when pods need it and vanishes when they don't.

The older tool, Cluster Autoscaler, thinks in terms of node groups you defined in advance — an Auto Scaling Group of, say, "g5.xlarge nodes." It can only add or remove copies of shapes you already chose, like a car park where every space is pre-marked "compact only." That's fine until a pod needs a shape you didn't pre-mark, or until you're paying for half-empty groups. Karpenter flips the model: instead of scaling pre-cut groups, it reads the pending pods directly, works out the cheapest EC2 instance type from a set you allow that actually fits what they asked for, and launches exactly that — a valet who parks each car in the perfect-sized bay rather than forcing everyone into "compact." When load falls, it consolidates: it repacks the surviving pods onto fewer nodes and terminates the ones left under-used, so you stop paying for slack.

Why this matters so much for AI specifically: GPU instances are the most expensive things you'll ever schedule, so the cost of a wrong-sized or idle node is brutal, and GPU demand is spiky (a burst of inference, a training job that ends). Karpenter's just-in-time provisioning and aggressive consolidation are therefore not just a convenience — they are the mechanism that keeps GPU spend bounded. You describe the rules with two custom resources: a NodePool (the Kubernetes-facing constraints — which instance families are allowed, Spot vs on-demand, a hard GPU cap, how eagerly to consolidate) and an EC2NodeClass (the AWS-level facts — which AMI, which subnets, which node IAM role). Keep that split in mind reading the YAML: NodePool = what to schedule, EC2NodeClass = how to build the box.

K7 covered scheduling a pod onto a GPU node; the AWS question is where do the GPU nodes come from. The classic answer is Cluster Autoscaler, which scales pre-defined node groups (Auto Scaling Groups) up and down. Karpenter is the AWS-native alternative: instead of fixed groups, it looks at pending pods and provisions right-sized EC2 instances directly, picking instance types from a flexible set — then consolidates (repacks and terminates under-used nodes) as load falls.

For GPU AI that means: a burst of pending vLLM pods requesting nvidia.com/gpu can cause Karpenter to launch, say, a g5 instance in seconds; when the burst clears, consolidation tears it back down so you are not paying for an idle GPU. You express what is allowed with two CRDs — a NodePool (scheduling constraints/limits) and an EC2NodeClass (the AWS-level details: AMI, subnets, IAM role).

gpu-nodepool.yamlapiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu
spec:
  template:
    spec:
      # only launch GPU instance families; let Karpenter pick the cheapest that fits
      requirements:
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values: ["g5", "g6", "p4d", "p5"]     # verify families/availability in your region
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]          # allow both; Spot preferred when available
      taints:
        - key: nvidia.com/gpu                     # repel non-GPU pods (K7)
          effect: NoSchedule
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: gpu
  limits:
    nvidia.com/gpu: 16          # hard cap on total GPUs this pool may create (cost guardrail)
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 2m        # reclaim idle/under-used GPU nodes quickly
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: gpu
spec:
  amiFamily: AL2023            # verify the current GPU-capable AMI family/alias
  role: "KarpenterNodeRole-my-cluster"   # the node IAM role (placeholder)
  subnetSelectorTerms:   [{ tags: { "karpenter.sh/discovery": "my-cluster" } }]
  securityGroupSelectorTerms: [{ tags: { "karpenter.sh/discovery": "my-cluster" } }]

AWS AI accelerators: Inferentia & Trainium (Neuron)

Here's the pitch that gets a finance director excited and an engineer nervous. "We can cut our inference bill by a third by moving off NVIDIA GPUs onto Amazon's own chips." It can be true — and it can also be a two-week detour that ends in "our model doesn't compile." The tension in this section is exactly that gap between a real cost win and a real compatibility tax, and the skill is knowing which one you're walking into before you commit a platform to it.

The analogy that keeps it straight: a GPU (NVIDIA, via CUDA) is a Swiss Army knife — it runs almost any model you throw at it because the whole ML world has spent a decade building for CUDA. Amazon's custom silicon — Inferentia (the inf2 instances, tuned for inference) and Trainium (trn1 and newer, tuned for training) — is more like a specialised power tool: cheaper and faster per unit of work for the jobs it was built for, useless for the ones it wasn't. You program these chips through the Neuron SDK, and the critical difference from a GPU is that you don't just load your model and run — you must compile it with the Neuron compiler first, ahead of time, into a form the chip understands.

That compile step is where the tax lives, and it's the mistake to name loudly: not every model or operator is supported. If your architecture and its ops are on the supported list, compilation is a one-time build step and you're rewarded with better tokens-per-dollar. If it isn't — an exotic layer, a newer architecture the compiler hasn't caught up to — you can hit a wall that no amount of config fixes, and the "savings" evaporate into engineering time. On EKS the scheduling side is reassuringly familiar: just as K7 used a device plugin to advertise nvidia.com/gpu, a Neuron device plugin (a DaemonSet) advertises the chips as aws.amazon.com/neuron, and pods request that resource instead. Same Kubernetes pattern, different resource name. The honest posture: treat Neuron as a genuine lever for high-volume, steady inference on a supported model — and always prototype the compile-and-load-test step on your actual model before you build anything around it. The table and lab below show the trade-off and the scheduling glue.

GPUs are not the only silicon on AWS. Amazon builds custom AI chips: Inferentia (inf2 instances) for inference and Trainium (trn1/newer) for training. You program them through the Neuron SDK, and on EKS a Neuron device plugin advertises the chips as a schedulable resource — the same idea as the NVIDIA device plugin in K7, but the resource is aws.amazon.com/neuron instead of nvidia.com/gpu.

The pitch is cost and throughput: for a model that Neuron supports well, inf2 can serve more tokens per dollar than a comparable GPU. The tax is real, though: you must compile the model with the Neuron compiler ahead of time, and not every model/operator is supported — the ecosystem is narrower and younger than CUDA's. It is a genuine win for high-volume inference on a supported architecture, and a frustrating dead end if your model isn't supported. Prototype the compile step before committing.

neuron-pod.yaml# The Neuron device plugin (a DaemonSet, like K7's NVIDIA one) must be installed first;
# it advertises aws.amazon.com/neuron as a schedulable resource.
apiVersion: v1
kind: Pod
metadata:
  name: neuron-inference
spec:
  nodeSelector:
    node.kubernetes.io/instance-type: inf2.xlarge   # verify the instance type exists in your region
  containers:
    - name: server
      image: <account>.dkr.ecr.<region>.amazonaws.com/my-neuron-model:pinned  # compiled for Neuron
      resources:
        limits:
          aws.amazon.com/neuron: 1        # request 1 Neuron device (NOT nvidia.com/gpu)

Spot GPUs & cost governance on EKS

Two stories sit behind this section, and every EKS-AI team lives one of them. The first: an engineer discovers Spot GPUs, moves the customer-facing chat endpoint onto them to save 70%, and three days later AWS reclaims the single node with a two-minute warning — the endpoint goes dark during business hours and the "saving" becomes an incident review. The second, quieter and more common: nobody looks at utilisation for a month, and the invoice reveals a GPU node that ran at 5% the entire time, burning thousands of dollars doing almost nothing. One story is about using the discount recklessly; the other is about not watching the meter at all. This section is the antidote to both.

Start with the discount itself. Spot capacity is EC2's spare inventory sold at a steep discount off on-demand prices — the catch is that AWS can take it back whenever it needs the hardware, giving you only about a two-minute warning before the instance disappears. Think of it as flying standby: dramatically cheaper, but you might get bumped, so it's perfect when being bumped is merely annoying and catastrophic when it isn't. That single property — reclaimable on short notice — cleanly sorts every AI workload into "fine on Spot" or "not." Batch and offline inference, experimentation, and training that checkpoints are all interruptible: a reclaim just reschedules the work or resumes from the last checkpoint, costing minutes. A single-replica, latency-critical endpoint is the opposite: one reclaim is one outage. The professional move for anything in between is a base-plus-burst split — enough on-demand replicas to serve minimum traffic no matter what, with Spot layered on only for the peaks.

But cheap capacity is only half the discipline; the other half is seeing where the money goes, because the deadliest EKS-AI cost isn't a dramatic mistake — it's the idle GPU nobody noticed. You cannot govern what you cannot see, so the order of operations is: make GPU spend visible first (cost-allocation tags so spend is attributable per team and model, CloudWatch or Container Insights for node utilisation, Kubecost for per-namespace Kubernetes cost), then bound it (the Karpenter GPU cap and consolidation from section 2), and only then argue about which service is cheapest. This is why Karpenter's consolidation is a cost feature, not just a scheduling one — it directly attacks idle time, the single biggest silent bill. Keep that hierarchy — survive interruptions, then watch the meter, then compare — as you read the workload table and the offline cost math below.

The single biggest EKS-AI cost lever is Spot capacity: EC2 spare capacity at a large discount off on-demand, in exchange for the fact that AWS can reclaim the instance with a short (~2-minute) warning. That reclaim risk is exactly why Spot fits interruptible AI work — batch/offline inference, experimentation, and fault-tolerant training with checkpointing — but is a poor fit for a single-replica, latency-critical endpoint that can't absorb a sudden node loss.

Handle interruptions rather than hope: run enough replicas across instances so losing one is survivable, set a PodDisruptionBudget, and let a node-interruption handler (Karpenter consumes the interruption/rebalance signals natively) drain the node gracefully on the 2-minute warning. For training, checkpoint frequently so a reclaim costs minutes, not the run.

How does GPU-on-EKS cost compare to a SageMaker endpoint or Bedrock tokens? Qualitatively: Bedrock bills per token and scales to zero — cheapest when traffic is spiky or low, no idle cost at all. SageMaker endpoints bill per instance-hour (like a reserved GPU) but AWS runs the serving. EKS is also per instance-hour but you own utilisation — it only beats the others when you keep the GPUs genuinely busy and amortise the ops effort. Idle capacity flips EKS from cheapest to most expensive. The offline calculator in the exercises below makes this concrete; verify current prices before trusting any absolute number.

Serving LLMs on EKS the AWS way

You already know how to serve a model on Kubernetes — K7 taught the mechanics. So why a whole section on doing it "the AWS way"? Because the moment you're on EKS, three mundane-sounding questions turn into security and cost decisions that sink teams: where do the multi-gigabyte weights come from, where does the container image come from, and how does the pod get permission to fetch them? Get the last one wrong and you end up with an AWS access key hard-coded into a Dockerfile — the single most common way cloud credentials leak, and a finding that fails every security audit. This section is the AWS glue around the generic serving you already understand.

The picture to hold is a supply chain with a security checkpoint. The weights — often tens of gigabytes — live in S3, AWS's object store, because baking them into a container image would make it enormous and slow to pull. The serving image (your vLLM or KServe runtime) lives in ECR, AWS's private container registry. At start-up an init container syncs the weights from S3 onto a local volume, then the real server container boots and reads them. Finally the endpoint is exposed to the world through the AWS Load Balancer Controller — an ALB for ordinary HTTP traffic, or an NLB when you need raw TCP or gRPC. That's the flow the diagram traces: S3 → init container → pod → load balancer.

The checkpoint in that chain — and the reason this section earns its place — is IRSA, IAM Roles for Service Accounts. Here's the analogy: instead of handing the pod a permanent master key it carries in its pocket (a static access key), IRSA is a valet system. You annotate the pod's Kubernetes ServiceAccount with an IAM role's ARN; when the pod starts, it presents its identity to the cluster's OIDC provider and is handed a temporary, automatically-rotated credential scoped to exactly that one role — nothing more. If the pod is compromised, the attacker inherits only what that narrow role allows (say, read-only on one S3 prefix), and the credential expires on its own. This is the exact AWS mirror of the least-privilege principle K7 enforced with RBAC: RBAC bounds what a pod can do to the cluster; IRSA bounds what it can do to AWS. The same mechanism is what lets an EKS pod call Bedrock in the hybrid pattern from section 1 without ever storing a key. The labs below wire it up: a ServiceAccount, a pod that pulls from S3, and the tight IAM policy that makes the role nearly powerless.

K7 showed the generic serving mechanics (weights on a PVC, vLLM, KServe, autoscale on queue depth). The AWS-specific glue is where the bytes and permissions come from: model weights live in S3, container images in ECR, the inference endpoint is exposed through the AWS Load Balancer Controller (ALB for HTTP, NLB for raw TCP/gRPC), and — critically — pods get AWS permissions via IRSA (IAM Roles for Service Accounts), not baked-in keys.

IRSA is the AWS-native way to give a pod least-privilege AWS access. You annotate the pod's ServiceAccount with an IAM role ARN; the pod then assumes that role via the cluster's OIDC provider and gets temporary credentials scoped to exactly that role — no long-lived keys in the image or env. This is the mechanism a pod uses to read model weights from S3, or to call Bedrock in the hybrid pattern from section 1.

irsa-s3-serve.yaml# 1) A ServiceAccount annotated with the IAM role it should assume (IRSA).
apiVersion: v1
kind: ServiceAccount
metadata:
  name: model-loader
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/eks-model-reader  # placeholder
---
# 2) A pod that uses that SA; an initContainer syncs weights from S3 before vLLM starts.
apiVersion: v1
kind: Pod
metadata:
  name: llm
spec:
  serviceAccountName: model-loader          # -> pod gets the role's temporary creds via IRSA
  initContainers:
    - name: fetch-weights
      image: amazon/aws-cli:latest          # pin a version in prod
      command: ["aws", "s3", "sync", "s3://my-models/my-llm/", "/models/my-llm/"]
      volumeMounts: [ { name: models, mountPath: /models } ]
  containers:
    - name: vllm
      image: <account>.dkr.ecr.<region>.amazonaws.com/vllm:pinned   # image from ECR
      args: ["--model", "/models/my-llm"]
      resources: { limits: { nvidia.com/gpu: 1 } }
      volumeMounts: [ { name: models, mountPath: /models } ]
  volumes:
    - name: models
      emptyDir: {}

model-reader-policy.json{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadOneModelPrefix",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::my-models",
        "arn:aws:s3:::my-models/my-llm/*"
      ]
    }
  ]
}

AI operating EKS on AWS

Imagine the incident you'd never live down. You give an operating agent read-only access to diagnose cluster problems, feeling safe because its Kubernetes RBAC is tightly scoped — it can't delete a Deployment, can't touch a secret. Then, during an incident, a poisoned log line or a hallucinated command nudges it, and it terminates three EC2 nodes and starts listing every S3 bucket in the account. Nothing it did violated its RBAC. How? Because on EKS the agent had a second identity you forgot to lock down. That blind spot is the entire subject of this section.

The mental shift is this: on a plain Kubernetes cluster, an agent has one boundary — its RBAC, which governs what it can do to cluster objects. On EKS, the same pod also carries an AWS identity via IRSA (the mechanism from section 5), and that identity governs what it can do to the AWS account — EC2, S3, Bedrock, even the EKS control-plane API itself. So an "in-cluster" agent has two locks on two different doors: RBAC on the cluster door, and its IAM role on the AWS door. The dangerous, invisible mistake is scoping the first door carefully and leaving the second wide open — because RBAC has nothing to say about whether the pod can terminate a node or read a bucket. Those live entirely on the AWS side, outside Kubernetes' view.

The AWS-specific surface an operating agent touches is worth naming so you can bound each part: alongside the Kubernetes API there's the EKS control-plane API (node groups, add-ons), CloudWatch and Container Insights as the evidence source it reads to diagnose, and — the sharp edge — that IAM boundary. The safety recipe is the ch08 / K7 pattern with one extra AWS lock bolted on: keep the agent diagnosis-first (read cluster state and CloudWatch, explain the incident, propose a fix rather than apply it), gate every mutation behind dry-run plus human approval or a GitOps PR, and scope both boundaries narrowly — a tight RBAC role and a tight IRSA policy. The rule to carry away: on EKS, "least privilege" is never one setting. It's two, and the AWS one is the boundary teams forget until the blast radius escapes the cluster.

K7 built the generic AIOps agent (read-only kubectl diagnosis, RBAC/dry-run/approval as the enforced safety layers) and it ties back to the ch08 DevOps-agent capstone. The AWS-specific surface an agent touches on EKS is worth naming, because it widens the blast radius beyond the cluster: alongside the Kubernetes API there is the EKS control-plane API (managing node groups, add-ons), CloudWatch / Container Insights as the evidence source, and — the sharp edge — the pod's IAM boundary.

gpu-nodepool.yamlapiVersion: karpenter.sh/v1
kind: NodePool
metadata: { name: gpu }
spec:
  template:
    spec:
      requirements:
        - { key: karpenter.k8s.aws/instance-family, operator: In, values: ["g5","g6","p4d","p5"] }
        - { key: karpenter.sh/capacity-type, operator: In, values: ["spot","on-demand"] }
      taints:
        - { key: nvidia.com/gpu, effect: NoSchedule }
      nodeClassRef: { group: karpenter.k8s.aws, kind: EC2NodeClass, name: gpu }
  limits: { nvidia.com/gpu: 16 }                 # hard cap = cost guardrail
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 2m

break_even.pydef eks_vs_managed(node_hourly, hours_run, utilisation, managed_total):
    """utilisation in (0,1]; managed_total = cost of the same useful work on SageMaker/Bedrock."""
    eks_total = node_hourly * hours_run                 # you pay for idle time too
    useful_hours = hours_run * utilisation
    eks_per_useful = eks_total / useful_hours           # true unit cost of useful work
    winner = "EKS" if eks_total < managed_total else "managed"
    return {
        "eks_total": round(eks_total, 2),
        "eks_per_useful_hour": round(eks_per_useful, 2),
        "winner": winner,
    }

# Low utilisation: an idle-heavy EKS GPU node loses to the managed option
print(eks_vs_managed(node_hourly=4.0, hours_run=730, utilisation=0.10, managed_total=1200))
# -> {'eks_total': 2920.0, 'eks_per_useful_hour': 40.0, 'winner': 'managed'}

# High utilisation: a well-packed EKS node wins
print(eks_vs_managed(node_hourly=4.0, hours_run=730, utilisation=0.85, managed_total=3500))
# -> {'eks_total': 2920.0, 'eks_per_useful_hour': 4.71, 'winner': 'EKS'}

irsa-sa.yaml# ServiceAccount -> role (IRSA)
apiVersion: v1
kind: ServiceAccount
metadata:
  name: model-loader
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/eks-model-reader

# IAM policy attached to that role — read-only, one prefix:
# {
#   "Version": "2012-10-17",
#   "Statement": [{
#     "Effect": "Allow",
#     "Action": ["s3:GetObject", "s3:ListBucket"],
#     "Resource": ["arn:aws:s3:::my-models",
#                  "arn:aws:s3:::my-models/my-llm/*"]
#   }]
# }

Knowledge check