Security, RBAC & policy

1 · RBAC: subjects, verbs, resources — least privilege

Start with the incident you're trying not to cause. A pod running your LLM API gets popped — a dependency has a remote-code-execution bug, or a prompt injection tricks a tool into shelling out. The attacker is now inside a container. The only question that matters next is: how much of your cluster can that container reach through the Kubernetes API? If the answer is "everything", you don't have a compromised pod, you have a compromised company. RBAC is the dial that sets that answer, and it's the single most important control in this lesson.

Here's the mental model. Think of RBAC like keycards in an office building. A keycard doesn't know who you are — it just carries a set of doors it opens. You hand a keycard to a person (or, in Kubernetes, to a workload), and from then on that identity can open exactly those doors and no others. The genius of the system is that it's default-deny: a brand-new keycard opens nothing. You never write "block the server room" — the server room is already locked to everyone; you only ever add the specific doors someone needs. That's why there are no "deny" rules in RBAC: permissions are purely additive on top of a zero baseline. To lock something down you simply don't grant it.

The most common mistake is the one that feels like a fix. Something "isn't working", the logs say forbidden, and under deadline pressure someone binds cluster-admin to the workload — handing out the master key that opens every door in the building because one office was locked. It works instantly, the ticket closes, and a landmine is now buried in your cluster: the day that pod is compromised, the blast radius is the entire cluster. The discipline that prevents this is least privilege — start from zero, grant the exact verbs on the exact resources the workload provably needs, and verify with kubectl auth can-i that it can do those things and nothing more. Most application pods, it turns out, need almost no API access at all.

RBAC answers 'who can do what, where'. Four object types combine: a Role (namespaced) or ClusterRole (cluster-wide) lists allowed verbs on resources; a RoleBinding or ClusterRoleBinding grants that Role to a subject (a User, Group, or — for workloads — a ServiceAccount). There are no 'deny' rules: permissions are additive and default-deny — you can only grant.

rbac.yamlapiVersion: v1
kind: ServiceAccount
metadata: { name: llm-api, namespace: web }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: read-config, namespace: web }
rules:
  - apiGroups: [""]
    resources: [configmaps]
    verbs: [get, list, watch]        # read config only — no secrets, no write
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: llm-api-read-config, namespace: web }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: Role, name: read-config }
subjects:
  - kind: ServiceAccount
    name: llm-api
    namespace: web

audit.sh# Impersonate the SA and test specific permissions:
kubectl auth can-i list configmaps -n web \
  --as=system:serviceaccount:web:llm-api        # -> yes
kubectl auth can-i delete secrets -n web \
  --as=system:serviceaccount:web:llm-api        # -> no  (good)
kubectl auth can-i '*' '*' --all-namespaces \
  --as=system:serviceaccount:web:llm-api        # -> no  (never grant this)

yes
no
no

2 · Pod Security Standards at admission

RBAC just locked down what a pod can ask the cluster to do — but that leaves a second, sneakier door wide open. A pod can be perfectly polite to the API and still be dangerous because of what it is: running as root, mounting the host's filesystem, sharing the node's network namespace, or granting itself Linux capabilities that let it break out of the container and onto the machine underneath. Those settings live in the Pod spec's securityContext, and if nobody checks them, any manifest can request them. Pod Security is the layer that reads the spec at the door and says "no."

The analogy: RBAC is the keycard that controls which rooms you enter; Pod Security is the metal detector at the entrance that inspects what you're carrying in. It doesn't care what you plan to do inside — it refuses to admit a workload that shows up "armed" (privileged, root, host-mounted) in the first place. And crucially it runs at admission time: the check happens the instant you kubectl apply, before the pod ever schedules, so a non-compliant pod is rejected rather than quietly running in a dangerous state you discover months later.

Kubernetes ships this as three named Pod Security Standards, which are just pre-agreed strictness levels so you don't have to invent your own: privileged (anything goes — for trusted system workloads only), baseline (blocks the obviously reckless, like privileged containers and host namespaces), and restricted (the hardened target — non-root, drop all capabilities, seccomp, read-only root filesystem). The common mistake here isn't picking the wrong level; it's the rollout. Flip a busy namespace straight to enforce: restricted and you'll reject every running workload the moment it redeploys — an outage you caused with a security label. That's why each standard runs in one of three modes — enforce, audit, warn — so you can watch who would be rejected before you actually start rejecting them.

RBAC controls the API; Pod Security controls what a Pod may be. The built-in Pod Security Admission controller enforces three Pod Security Standards per namespace via labels: privileged (no restrictions), baseline (blocks the obviously dangerous — host namespaces, privileged containers), and restricted (hardened — non-root, drop all capabilities, seccomp). Each level runs in one of three modes: enforce, audit, warn.

ns-restricted.yamlapiVersion: v1
kind: Namespace
metadata:
  name: web
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/warn: restricted     # also warn on kubectl apply
# Pods that run as root or add capabilities will be REJECTED at admission.

A Pod that satisfies restricted looks like this — the same hardening cd6 taught for the image, now enforced by the cluster:

hardened-pod.yamlspec:
  securityContext:
    runAsNonRoot: true
    seccompProfile: { type: RuntimeDefault }
  containers:
    - name: app
      image: ghcr.io/acme/llm-api@sha256:...   # pin by digest (cd6)
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities: { drop: ["ALL"] }

3 · Policy engines: OPA Gatekeeper & Kyverno

Pod Security is a great metal detector — but it only knows how to look for weapons. It has exactly three profiles, and they answer one narrow question: "is this pod running dangerously?" Your organization has a dozen other rules that have nothing to do with container privilege: every image must come from your own registry, every namespace must carry a cost-center label for billing, nobody is allowed to deploy a :latest tag, no hostPath mounts, resource limits are mandatory. Pod Security can't express any of that. You need something programmable — a place to write "at admission, here is our definition of an acceptable object."

That programmable checkpoint is a policy engine, and the mechanism it plugs into is one of Kubernetes' most powerful extension points: the admission webhook. Picture the API server as a customs desk that every object must clear before it's allowed into the country. A validating webhook is the inspector who stamps "approved" or "rejected"; a mutating webhook is the clerk who can quietly stamp a missing label onto your form before it proceeds. A policy engine registers itself as that inspector, so on every create-or-update the API server phones it and asks "does this pass our rules?" — and your rules, written once, are enforced on everyone, humans and CI pipelines alike, with no way to sneak around them.

Two engines dominate, and the choice is mostly about the language you want to write rules in. OPA Gatekeeper uses Rego, a real purpose-built policy language — powerful and reusable across non-Kubernetes systems, but a genuine learning curve. Kyverno writes policies in plain YAML that looks like the Kubernetes resources you already know, so most teams get productive in an afternoon. The mistake that bites everyone regardless of engine is going straight to enforce: an Enforce policy with a subtle bug can block every deployment cluster-wide — including the deploy that fixes the policy. Always roll out in audit mode first, watch the violation reports, and only flip to enforce once it's clean; and always exempt kube-system so a bad rule can't brick the control plane itself.

Pod Security is fixed to three profiles. For your org rules — 'every image must come from our registry', 'every namespace must have a cost-center label', 'no :latest tags' — you need a policy engine running as a validating (and mutating) admission webhook. Two dominate: OPA Gatekeeper (policies in Rego, via ConstraintTemplates + Constraints) and Kyverno (policies in YAML, no new language).

kyverno-registry.yamlapiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: allowed-registries }
spec:
  validationFailureAction: Enforce      # block on violation (Audit to observe first)
  rules:
    - name: only-acme-registry
      match:
        any: [ { resources: { kinds: [Pod] } } ]
      validate:
        message: "images must come from ghcr.io/acme/"
        pattern:
          spec:
            containers:
              - image: "ghcr.io/acme/*"

registry.regopackage k8sallowedregistry
violation[{"msg": msg}] {
  container := input.review.object.spec.containers[_]
  not startswith(container.image, "ghcr.io/acme/")
  msg := sprintf("image %v not from an approved registry", [container.image])
}

4 · Secrets: base64 is not encryption

Here's the sentence that has ended careers: "it's fine, it's in a Kubernetes Secret." The word "Secret" does a lot of unearned reassurance. New engineers see the value come back as an unreadable string of characters and assume it's encrypted and safe. It is not. A Kubernetes Secret is merely base64-encoded — and base64 is not a lock, it's an envelope. Anyone who can read the Secret object, or read the etcd database behind the API, or grab a backup of it, can decode the value back to plaintext in one command. Your API keys, your database passwords, your model provider tokens sit there in a costume, not a vault.

The analogy worth internalizing: base64 is Pig Latin, not a cipher. It scrambles the letters into something you can't read at a glance, but it uses no key and hides nothing from anyone who bothers to unscramble it. So the phrase "base64 is not encryption" isn't pedantry — it's the whole point of this section. Treating a raw Secret as secure is the classic mistake, and it shows up in two flavors: assuming the encoded value protects the data, and — worse — committing that "Secret" YAML straight into a Git repo, where it's now permanently in the history for every person and CI system with repo access.

So the real work of secrets management is two separate questions. First, defense at rest: turn on encryption-at-rest for etcd (a provider setting, or KMS-backed on EKS) so a stolen etcd snapshot is ciphertext, and lock down who can get secrets via RBAC — because a token that can read secrets can usually escalate. Second, and more architecturally interesting, where does the source of truth actually live and how does the value get into the cluster without ever touching Git in plaintext? That question has three well-worn answers — External Secrets, Sealed Secrets, and Vault — each making a different trade about where the real secret lives and who is allowed to decrypt it.

A Kubernetes Secret is only base64-encoded, not encrypted — anyone who can read the object (or etcd) reads the value. Two baseline hardening steps: enable encryption at rest for etcd (a provider config, or KMS on EKS) and lock down secrets RBAC. But the real question is where the source of truth lives and how it gets in — three common patterns:

externalsecret.yamlapiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata: { name: llm-api-keys, namespace: web }
spec:
  refreshInterval: 1h
  secretStoreRef: { name: aws-sm, kind: ClusterSecretStore }
  target: { name: llm-api-keys }        # the k8s Secret it creates/keeps in sync
  data:
    - secretKey: ANTHROPIC_API_KEY
      remoteRef: { key: prod/llm-api, property: anthropic_api_key }

5 · Supply-chain admission: only run trusted images

Everything so far protects the cluster from what runs inside it — this section protects it from what you never meant to run at all. Your CI pipeline builds a clean, scanned, signed image and pushes it. Good. But the cluster will happily run any image it's told to: a public image someone typo'd into a manifest, an image from an attacker who phished a registry credential, a stale tag that got repointed at malicious content overnight. The image name in a Pod spec is a claim, and until now nobody has been checking whether that claim is trustworthy. Supply-chain admission is the check.

Think of it as passport control for containers. In cd6 you gave each image a tamper-evident passport at build time: you scanned it, generated an SBOM (its ingredient list), signed it with cosign (a cryptographic seal only your pipeline's key can produce), and attached provenance (a SLSA attestation of how and where it was built). All of that is worthless, though, if the border guard waves everyone through. The admission policy is the border guard: at the cluster door it verifies the cosign signature against your public key and rejects anything that isn't sealed by you — so a leaked registry cred or a wrong image reference simply cannot board.

The mistake to avoid is thinking any one of these controls is "the" answer, because they defend at different moments against different threats — and this is the key insight of the section: digest ≠ signature ≠ scan. Pinning by digest (@sha256:…) freezes exactly which bytes run so a tag can't be swapped under you. Signature verification at admission proves those bytes came from your pipeline. Ongoing scanning catches CVEs that are discovered after you deployed, against images that were perfectly clean the day they shipped. Skip any one and you have a gap: a signed image can still be vulnerable, a scanned image can still be swapped, a pinned image can still be malicious if it was never yours. You want all three, running at build, at admission, and on a cadence.

cd6 built trust into the image — scanning, an SBOM, a signature (cosign), and provenance (SLSA attestation). This lesson enforces that trust at the cluster door: an admission policy that rejects any image that isn't signed by your key and pinned by digest. Now a leaked registry credential or a typo'd public image can't run in prod.

verify-images.yamlapiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: verify-signatures }
spec:
  validationFailureAction: Enforce
  rules:
    - name: check-cosign
      match: { any: [ { resources: { kinds: [Pod] } } ] }
      verifyImages:
        - imageReferences: [ "ghcr.io/acme/*" ]
          attestors:
            - entries:
                - keys: { publicKeys: |
                    -----BEGIN PUBLIC KEY-----
                    ...your cosign public key...
                    -----END PUBLIC KEY----- }

6 · NetworkPolicy: the in-cluster firewall

Here's the fact that surprises almost everyone the first time they hear it: by default, every pod in your cluster can open a network connection to every other pod. Your public-facing frontend can talk straight to your database. A logging sidecar can reach the payments service. A compromised, low-value pod in one namespace can port-scan and connect to the crown-jewel workload three namespaces away. Kubernetes ships with a completely flat, open network — great for getting started, terrifying once you assume (as security demands you do) that something will eventually be breached.

The fix is a NetworkPolicy, and the mental model is exactly the office building again — this time the internal locked doors between departments, not just the front entrance. An open-plan office where anyone can walk into any room is convenient until one visitor turns out to be a thief; then you wish accounting had been behind a badge reader. NetworkPolicy lets you say, by label selector, "only pods labelled app: api may connect to pods labelled app: db, and only on port 5432." The defense-in-depth pattern is to start with a default-deny in each namespace — lock every internal door — and then reopen only the specific flows your application actually needs. This is what contains lateral movement: a compromised frontend can no longer wander to the database just because they share a network.

But there's a trap here nastier than anything else in this lesson, because it fails silently. A NetworkPolicy is only a request — it does nothing on its own. The actual enforcement is done by your cluster's CNI (the networking plugin: Calico, Cilium, or EKS's VPC CNI with policy enabled). If your CNI doesn't enforce policy, the API server still accepts your NetworkPolicy YAML without complaint, it shows up in kubectl get, and traffic flows exactly as before. You get a beautiful audit-ready policy and zero protection — a false sense of security, which is arguably worse than none. So the non-negotiable step is to confirm your CNI enforces policy and then test it: curl from a pod that should be denied and watch it actually fail before you trust the lock.

By default, every Pod can talk to every other Pod — a flat network. A compromised frontend can reach your database directly. A NetworkPolicy restricts pod-to-pod traffic by label selector; the defense-in-depth move is a default-deny in each namespace, then allow only the flows you need. (Note: NetworkPolicy needs a CNI that enforces it — Calico, Cilium; some setups ignore it silently.)

networkpolicy.yaml# 1) Deny all ingress in the namespace:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny-ingress, namespace: web }
spec:
  podSelector: {}          # all pods
  policyTypes: [Ingress]
---
# 2) Allow only the api pods to reach the db pods on 5432:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: allow-api-to-db, namespace: web }
spec:
  podSelector: { matchLabels: { app: db } }
  policyTypes: [Ingress]
  ingress:
    - from: [ { podSelector: { matchLabels: { app: api } } } ]
      ports: [ { protocol: TCP, port: 5432 } ]

kind: ServiceAccount
metadata: { name: llm-api, namespace: web }
---
kind: Role
metadata: { name: read-config, namespace: web }
rules: [ { apiGroups: [""], resources: [configmaps], verbs: [get,list,watch] } ]
---
kind: RoleBinding
metadata: { name: llm-api-rb, namespace: web }
roleRef: { kind: Role, name: read-config, apiGroup: rbac.authorization.k8s.io }
subjects: [ { kind: ServiceAccount, name: llm-api, namespace: web } ]

# Namespace
labels:
  pod-security.kubernetes.io/enforce: restricted
  pod-security.kubernetes.io/warn: restricted

# Compliant Pod
securityContext: { runAsNonRoot: true, seccompProfile: { type: RuntimeDefault } }
containers:
  - securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities: { drop: [ALL] }

kind: ClusterPolicy
metadata: { name: allowed-registries }
spec:
  validationFailureAction: Enforce      # start as Audit
  rules:
    - name: only-acme
      match: { any: [ { resources: { kinds: [Pod] } } ] }
      validate:
        message: "images must come from ghcr.io/acme/"
        pattern: { spec: { containers: [ { image: "ghcr.io/acme/*" } ] } }

verifyImages:
  - imageReferences: [ "ghcr.io/acme/*" ]
    attestors: [ { entries: [ { keys: { publicKeys: "" } } ] } ]