AI/ML on EKS

When EKS for AI: the EKS vs SageMaker vs Bedrock decision

AWS gives you three very different ways to run AI, and the mistake is treating them as competitors when they are layers. Bedrock is a managed model API — you call a foundation model (W2) and never see a server. SageMaker (W8/W9) is a managed ML platform — you own the model but AWS owns the serving/ training plumbing. EKS is self-run Kubernetes — you own everything down to the node, in exchange for maximum control and portability. You move down that list only when the layer above genuinely cannot meet a hard requirement.

GPU node autoscaling with Karpenter

K7 covered scheduling a pod onto a GPU node; the AWS question is where do the GPU nodes come from. The classic answer is Cluster Autoscaler, which scales pre-defined node groups (Auto Scaling Groups) up and down. Karpenter is the AWS-native alternative: instead of fixed groups, it looks at pending pods and provisions right-sized EC2 instances directly, picking instance types from a flexible set — then consolidates (repacks and terminates under-used nodes) as load falls.

For GPU AI that means: a burst of pending vLLM pods requesting nvidia.com/gpu can cause Karpenter to launch, say, a g5 instance in seconds; when the burst clears, consolidation tears it back down so you are not paying for an idle GPU. You express what is allowed with two CRDs — a NodePool (scheduling constraints/limits) and an EC2NodeClass (the AWS-level details: AMI, subnets, IAM role).

gpu-nodepool.yamlapiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu
spec:
  template:
    spec:
      # only launch GPU instance families; let Karpenter pick the cheapest that fits
      requirements:
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values: ["g5", "g6", "p4d", "p5"]     # verify families/availability in your region
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]          # allow both; Spot preferred when available
      taints:
        - key: nvidia.com/gpu                     # repel non-GPU pods (K7)
          effect: NoSchedule
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: gpu
  limits:
    nvidia.com/gpu: 16          # hard cap on total GPUs this pool may create (cost guardrail)
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 2m        # reclaim idle/under-used GPU nodes quickly
---
apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: gpu
spec:
  amiFamily: AL2023            # verify the current GPU-capable AMI family/alias
  role: "KarpenterNodeRole-my-cluster"   # the node IAM role (placeholder)
  subnetSelectorTerms:   [{ tags: { "karpenter.sh/discovery": "my-cluster" } }]
  securityGroupSelectorTerms: [{ tags: { "karpenter.sh/discovery": "my-cluster" } }]

AWS AI accelerators: Inferentia & Trainium (Neuron)

GPUs are not the only silicon on AWS. Amazon builds custom AI chips: Inferentia (inf2 instances) for inference and Trainium (trn1/newer) for training. You program them through the Neuron SDK, and on EKS a Neuron device plugin advertises the chips as a schedulable resource — the same idea as the NVIDIA device plugin in K7, but the resource is aws.amazon.com/neuron instead of nvidia.com/gpu.

The pitch is cost and throughput: for a model that Neuron supports well, inf2 can serve more tokens per dollar than a comparable GPU. The tax is real, though: you must compile the model with the Neuron compiler ahead of time, and not every model/operator is supported — the ecosystem is narrower and younger than CUDA's. It is a genuine win for high-volume inference on a supported architecture, and a frustrating dead end if your model isn't supported. Prototype the compile step before committing.

neuron-pod.yaml# The Neuron device plugin (a DaemonSet, like K7's NVIDIA one) must be installed first;
# it advertises aws.amazon.com/neuron as a schedulable resource.
apiVersion: v1
kind: Pod
metadata:
  name: neuron-inference
spec:
  nodeSelector:
    node.kubernetes.io/instance-type: inf2.xlarge   # verify the instance type exists in your region
  containers:
    - name: server
      image: <account>.dkr.ecr.<region>.amazonaws.com/my-neuron-model:pinned  # compiled for Neuron
      resources:
        limits:
          aws.amazon.com/neuron: 1        # request 1 Neuron device (NOT nvidia.com/gpu)

Spot GPUs & cost governance on EKS

The single biggest EKS-AI cost lever is Spot capacity: EC2 spare capacity at a large discount off on-demand, in exchange for the fact that AWS can reclaim the instance with a short (~2-minute) warning. That reclaim risk is exactly why Spot fits interruptible AI work — batch/offline inference, experimentation, and fault-tolerant training with checkpointing — but is a poor fit for a single-replica, latency-critical endpoint that can't absorb a sudden node loss.

Handle interruptions rather than hope: run enough replicas across instances so losing one is survivable, set a PodDisruptionBudget, and let a node-interruption handler (Karpenter consumes the interruption/rebalance signals natively) drain the node gracefully on the 2-minute warning. For training, checkpoint frequently so a reclaim costs minutes, not the run.

How does GPU-on-EKS cost compare to a SageMaker endpoint or Bedrock tokens? Qualitatively: Bedrock bills per token and scales to zero — cheapest when traffic is spiky or low, no idle cost at all. SageMaker endpoints bill per instance-hour (like a reserved GPU) but AWS runs the serving. EKS is also per instance-hour but you own utilisation — it only beats the others when you keep the GPUs genuinely busy and amortise the ops effort. Idle capacity flips EKS from cheapest to most expensive. The offline calculator in the exercises below makes this concrete; verify current prices before trusting any absolute number.

Serving LLMs on EKS the AWS way

K7 showed the generic serving mechanics (weights on a PVC, vLLM, KServe, autoscale on queue depth). The AWS-specific glue is where the bytes and permissions come from: model weights live in S3, container images in ECR, the inference endpoint is exposed through the AWS Load Balancer Controller (ALB for HTTP, NLB for raw TCP/gRPC), and — critically — pods get AWS permissions via IRSA (IAM Roles for Service Accounts), not baked-in keys.

IRSA is the AWS-native way to give a pod least-privilege AWS access. You annotate the pod's ServiceAccount with an IAM role ARN; the pod then assumes that role via the cluster's OIDC provider and gets temporary credentials scoped to exactly that role — no long-lived keys in the image or env. This is the mechanism a pod uses to read model weights from S3, or to call Bedrock in the hybrid pattern from section 1.

irsa-s3-serve.yaml# 1) A ServiceAccount annotated with the IAM role it should assume (IRSA).
apiVersion: v1
kind: ServiceAccount
metadata:
  name: model-loader
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/eks-model-reader  # placeholder
---
# 2) A pod that uses that SA; an initContainer syncs weights from S3 before vLLM starts.
apiVersion: v1
kind: Pod
metadata:
  name: llm
spec:
  serviceAccountName: model-loader          # -> pod gets the role's temporary creds via IRSA
  initContainers:
    - name: fetch-weights
      image: amazon/aws-cli:latest          # pin a version in prod
      command: ["aws", "s3", "sync", "s3://my-models/my-llm/", "/models/my-llm/"]
      volumeMounts: [ { name: models, mountPath: /models } ]
  containers:
    - name: vllm
      image: <account>.dkr.ecr.<region>.amazonaws.com/vllm:pinned   # image from ECR
      args: ["--model", "/models/my-llm"]
      resources: { limits: { nvidia.com/gpu: 1 } }
      volumeMounts: [ { name: models, mountPath: /models } ]
  volumes:
    - name: models
      emptyDir: {}

model-reader-policy.json{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadOneModelPrefix",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::my-models",
        "arn:aws:s3:::my-models/my-llm/*"
      ]
    }
  ]
}

AI operating EKS on AWS

K7 built the generic AIOps agent (read-only kubectl diagnosis, RBAC/dry-run/approval as the enforced safety layers) and it ties back to the ch08 DevOps-agent capstone. The AWS-specific surface an agent touches on EKS is worth naming, because it widens the blast radius beyond the cluster: alongside the Kubernetes API there is the EKS control-plane API (managing node groups, add-ons), CloudWatch / Container Insights as the evidence source, and — the sharp edge — the pod's IAM boundary.

gpu-nodepool.yamlapiVersion: karpenter.sh/v1
kind: NodePool
metadata: { name: gpu }
spec:
  template:
    spec:
      requirements:
        - { key: karpenter.k8s.aws/instance-family, operator: In, values: ["g5","g6","p4d","p5"] }
        - { key: karpenter.sh/capacity-type, operator: In, values: ["spot","on-demand"] }
      taints:
        - { key: nvidia.com/gpu, effect: NoSchedule }
      nodeClassRef: { group: karpenter.k8s.aws, kind: EC2NodeClass, name: gpu }
  limits: { nvidia.com/gpu: 16 }                 # hard cap = cost guardrail
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 2m

break_even.pydef eks_vs_managed(node_hourly, hours_run, utilisation, managed_total):
    """utilisation in (0,1]; managed_total = cost of the same useful work on SageMaker/Bedrock."""
    eks_total = node_hourly * hours_run                 # you pay for idle time too
    useful_hours = hours_run * utilisation
    eks_per_useful = eks_total / useful_hours           # true unit cost of useful work
    winner = "EKS" if eks_total < managed_total else "managed"
    return {
        "eks_total": round(eks_total, 2),
        "eks_per_useful_hour": round(eks_per_useful, 2),
        "winner": winner,
    }

# Low utilisation: an idle-heavy EKS GPU node loses to the managed option
print(eks_vs_managed(node_hourly=4.0, hours_run=730, utilisation=0.10, managed_total=1200))
# -> {'eks_total': 2920.0, 'eks_per_useful_hour': 40.0, 'winner': 'managed'}

# High utilisation: a well-packed EKS node wins
print(eks_vs_managed(node_hourly=4.0, hours_run=730, utilisation=0.85, managed_total=3500))
# -> {'eks_total': 2920.0, 'eks_per_useful_hour': 4.71, 'winner': 'EKS'}

irsa-sa.yaml# ServiceAccount -> role (IRSA)
apiVersion: v1
kind: ServiceAccount
metadata:
  name: model-loader
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/eks-model-reader

# IAM policy attached to that role — read-only, one prefix:
# {
#   "Version": "2012-10-17",
#   "Statement": [{
#     "Effect": "Allow",
#     "Action": ["s3:GetObject", "s3:ListBucket"],
#     "Resource": ["arn:aws:s3:::my-models",
#                  "arn:aws:s3:::my-models/my-llm/*"]
#   }]
# }

Knowledge check