Security, RBAC & policy
A cluster is a shared, multi-tenant control plane — one over-broad grant or one un-scanned image is a breach. This lesson is defense-in-depth for Kubernetes: RBAC done right (Roles/ClusterRoles/Bindings, ServiceAccounts, least privilege), Pod Security Standards at admission, policy engines (OPA Gatekeeper / Kyverno) to enforce org rules, secrets management (External Secrets, Sealed Secrets, Vault), image & supply-chain admission (signatures, provenance), and NetworkPolicy as the in-cluster firewall. It builds directly on cd6 — container security & supply chain.
kubectl here needs a cluster (minikube/kind local, or EKS) and output is illustrative. This lesson extends cd6 — Security & supply chain (image scanning, SBOMs, signing) from the image into the cluster; read cd6 first if signing/SBOM is new. Policy engines and admission move fast — verify Gatekeeper, Kyverno, Pod Security Admission, External Secrets and Vault field names against their current docs before relying on any spec below.Learning objectives
- Model RBAC correctly: Role vs ClusterRole, RoleBinding vs ClusterRoleBinding, and least privilege.
- Bind permissions to a ServiceAccount a workload uses (not a human, not default).
- Apply Pod Security Standards (privileged / baseline / restricted) via Pod Security Admission.
- Enforce org policy at admission with OPA Gatekeeper or Kyverno, and write one policy.
- Choose a secrets strategy: External Secrets Operator, Sealed Secrets, or Vault — and why base64 isn't security.
- Add image/supply-chain admission (signature + provenance) and NetworkPolicy as defense-in-depth.
1 · RBAC: subjects, verbs, resources — least privilege
RBAC answers 'who can do what, where'. Four object types combine: a Role (namespaced) or ClusterRole (cluster-wide) lists allowed verbs on resources; a RoleBinding or ClusterRoleBinding grants that Role to a subject (a User, Group, or — for workloads — a ServiceAccount). There are no 'deny' rules: permissions are additive and default-deny — you can only grant.
| Object | Scope | Use it for |
|---|---|---|
| Role | One namespace | Permissions inside a single namespace (most app grants) |
| ClusterRole | Cluster-wide OR reusable | Cluster resources (nodes), or a reusable rule set bound per-namespace |
| RoleBinding | One namespace | Grant a Role (or a ClusterRole, scoped to this ns) to a subject |
| ClusterRoleBinding | Cluster-wide | Grant a ClusterRole everywhere — powerful, use sparingly |
rbac.yamlapiVersion: v1
kind: ServiceAccount
metadata: { name: llm-api, namespace: web }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: read-config, namespace: web }
rules:
- apiGroups: [""]
resources: [configmaps]
verbs: [get, list, watch] # read config only — no secrets, no write
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: llm-api-read-config, namespace: web }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: Role, name: read-config }
subjects:
- kind: ServiceAccount
name: llm-api
namespace: web
audit.sh# Impersonate the SA and test specific permissions:
kubectl auth can-i list configmaps -n web \
--as=system:serviceaccount:web:llm-api # -> yes
kubectl auth can-i delete secrets -n web \
--as=system:serviceaccount:web:llm-api # -> no (good)
kubectl auth can-i '*' '*' --all-namespaces \
--as=system:serviceaccount:web:llm-api # -> no (never grant this)
yes
no
noClusterRole/* verbs:[*]) to a workload 'to make it work' — one compromised pod owns the cluster. (2) Using the default ServiceAccount and auto-mounting its token into every pod that doesn't need the API (set automountServiceAccountToken: false). (3) Granting secrets read broadly — a token that can read secrets can often escalate. Start from zero, add the exact verbs a workload needs, and audit with kubectl auth can-i.2 · Pod Security Standards at admission
RBAC controls the API; Pod Security controls what a Pod may be. The built-in Pod Security Admission controller enforces three Pod Security Standards per namespace via labels: privileged (no restrictions), baseline (blocks the obviously dangerous — host namespaces, privileged containers), and restricted (hardened — non-root, drop all capabilities, seccomp). Each level runs in one of three modes: enforce, audit, warn.
ns-restricted.yamlapiVersion: v1
kind: Namespace
metadata:
name: web
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/enforce-version: latest
pod-security.kubernetes.io/warn: restricted # also warn on kubectl apply
# Pods that run as root or add capabilities will be REJECTED at admission.
A Pod that satisfies restricted looks like this — the same hardening cd6 taught for the image, now enforced by the cluster:
hardened-pod.yamlspec:
securityContext:
runAsNonRoot: true
seccompProfile: { type: RuntimeDefault }
containers:
- name: app
image: ghcr.io/acme/llm-api@sha256:... # pin by digest (cd6)
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
enforce: restricted will reject running workloads on their next deploy. Roll out gradually: set audit + warn first, watch which Pods violate, fix their securityContexts, then switch to enforce. Verify the label keys and the exact restricted requirements against the current Pod Security docs — they are versioned.3 · Policy engines: OPA Gatekeeper & Kyverno
Pod Security is fixed to three profiles. For your org rules — 'every image must come from our registry', 'every namespace must have a cost-center label', 'no :latest tags' — you need a policy engine running as a validating (and mutating) admission webhook. Two dominate: OPA Gatekeeper (policies in Rego, via ConstraintTemplates + Constraints) and Kyverno (policies in YAML, no new language).
| OPA Gatekeeper | Kyverno | |
|---|---|---|
| Policy language | Rego (a real language — powerful, a learning curve) | YAML (Kubernetes-native, easy to start) |
| Mutation | Limited / assign mutators | First-class mutate + generate |
| Reuse | ConstraintTemplate reused across Constraints | Per-policy rules |
| Good when | Complex logic, sharing Rego with non-k8s systems | Most k8s teams wanting fast, readable policy |
kyverno-registry.yamlapiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: allowed-registries }
spec:
validationFailureAction: Enforce # block on violation (Audit to observe first)
rules:
- name: only-acme-registry
match:
any: [ { resources: { kinds: [Pod] } } ]
validate:
message: "images must come from ghcr.io/acme/"
pattern:
spec:
containers:
- image: "ghcr.io/acme/*"
registry.regopackage k8sallowedregistry
violation[{"msg": msg}] {
container := input.review.object.spec.containers[_]
not startswith(container.image, "ghcr.io/acme/")
msg := sprintf("image %v not from an approved registry", [container.image])
}
Enforce policy with a bug can block all deploys cluster-wide, including the fix. Also decide the webhook failurePolicy: Fail (fail-closed — safer, but if the policy pod is down, admissions are blocked) vs Ignore (fail-open — availability over enforcement). Exempt kube-system. Verify current CRD/field names against the Kyverno/Gatekeeper docs.4 · Secrets: base64 is not encryption
A Kubernetes Secret is only base64-encoded, not encrypted — anyone who can read the object (or etcd) reads the value. Two baseline hardening steps: enable encryption at rest for etcd (a provider config, or KMS on EKS) and lock down secrets RBAC. But the real question is where the source of truth lives and how it gets in — three common patterns:
| Approach | How it works | Best for |
|---|---|---|
| External Secrets Operator | Syncs secrets from AWS Secrets Manager / Vault into k8s Secrets | Cloud-hosted secret store as source of truth |
| Sealed Secrets | Encrypt a Secret to a cluster public key; the ciphertext is safe to commit to Git | GitOps — secrets in the repo, decryptable only in-cluster |
| Vault (agent/CSI) | Inject secrets into pods at runtime; short-lived, dynamic, leased | Dynamic/rotating credentials, strong audit |
externalsecret.yamlapiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata: { name: llm-api-keys, namespace: web }
spec:
refreshInterval: 1h
secretStoreRef: { name: aws-sm, kind: ClusterSecretStore }
target: { name: llm-api-keys } # the k8s Secret it creates/keeps in sync
data:
- secretKey: ANTHROPIC_API_KEY
remoteRef: { key: prod/llm-api, property: anthropic_api_key }
5 · Supply-chain admission: only run trusted images
cd6 built trust into the image — scanning, an SBOM, a signature (cosign), and provenance (SLSA attestation). This lesson enforces that trust at the cluster door: an admission policy that rejects any image that isn't signed by your key and pinned by digest. Now a leaked registry credential or a typo'd public image can't run in prod.
verify-images.yamlapiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: verify-signatures }
spec:
validationFailureAction: Enforce
rules:
- name: check-cosign
match: { any: [ { resources: { kinds: [Pod] } } ] }
verifyImages:
- imageReferences: [ "ghcr.io/acme/*" ]
attestors:
- entries:
- keys: { publicKeys: |
-----BEGIN PUBLIC KEY-----
...your cosign public key...
-----END PUBLIC KEY----- }
@sha256:…) so the tag can't be moved under you; verify a signature at admission so only images your pipeline built can run; and keep scanning (cd6) on a cadence because new CVEs land against images you already deployed. Digest ≠ signature ≠ scan — you want all three. Verify cosign/Kyverno verifyImages fields against current docs.6 · NetworkPolicy: the in-cluster firewall
By default, every Pod can talk to every other Pod — a flat network. A compromised frontend can reach your database directly. A NetworkPolicy restricts pod-to-pod traffic by label selector; the defense-in-depth move is a default-deny in each namespace, then allow only the flows you need. (Note: NetworkPolicy needs a CNI that enforces it — Calico, Cilium; some setups ignore it silently.)
networkpolicy.yaml# 1) Deny all ingress in the namespace:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny-ingress, namespace: web }
spec:
podSelector: {} # all pods
policyTypes: [Ingress]
---
# 2) Allow only the api pods to reach the db pods on 5432:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: allow-api-to-db, namespace: web }
spec:
podSelector: { matchLabels: { app: db } }
policyTypes: [Ingress]
ingress:
- from: [ { podSelector: { matchLabels: { app: api } } } ]
ports: [ { protocol: TCP, port: 5432 } ]
curl from a non-allowed pod. Verify CNI policy support against your platform's current docs.✓ Checkpoint — you can move on when you can…
- Explain Role vs ClusterRole and RoleBinding vs ClusterRoleBinding, and bind a Role to a ServiceAccount.
- Audit a ServiceAccount's real permissions with
kubectl auth can-iand name the three RBAC sins. - Apply a Pod Security Standard and roll it out via audit/warn before enforce.
- Write a Gatekeeper or Kyverno policy and explain audit-first + fail-open vs fail-closed.
- Choose between External Secrets, Sealed Secrets and Vault, and say why base64 Secrets aren't secure.
- Enforce signed-image admission and a default-deny NetworkPolicy — and the CNI caveat.
A workload 'wasn't working', so a teammate ran kubectl create clusterrolebinding fix --clusterrole=cluster-admin --serviceaccount=web:llm-api and it started working. What's the risk, and how should it have been fixed?
Show answer
kubectl auth can-i. cluster-admin on a workload is almost never correct.Your team commits Kubernetes manifests to Git for GitOps, but security says 'no plaintext secrets in the repo'. You also can't manually kubectl create secret (that breaks GitOps). What are your options?
Show answer
kubeseal; the resulting ciphertext is safe to commit, and only the in-cluster controller (holding the private key) can decrypt it into a real Secret. External Secrets Operator: commit only a reference (an ExternalSecret pointing at AWS Secrets Manager / Vault); the operator fetches the value at runtime and keeps a k8s Secret in sync. Both keep the plaintext out of Git while staying declarative. Also enable etcd encryption-at-rest, since a k8s Secret is only base64.🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: Most 'permission denied' fixes are over-corrected into cluster-admin; least privilege is the habit.
Your task: Give a pod read-only access to ConfigMaps in its namespace — and nothing else — via a ServiceAccount.
Requirements:
- Create a ServiceAccount for the workload
- Write a namespaced Role for get/list/watch on configmaps
- Bind it with a RoleBinding
- Show how you'd verify it can't read secrets
💡 Hint: kubectl auth can-i --as=system:serviceaccount:
Show solution
kind: ServiceAccount
metadata: { name: llm-api, namespace: web }
---
kind: Role
metadata: { name: read-config, namespace: web }
rules: [ { apiGroups: [""], resources: [configmaps], verbs: [get,list,watch] } ]
---
kind: RoleBinding
metadata: { name: llm-api-rb, namespace: web }
roleRef: { kind: Role, name: read-config, apiGroup: rbac.authorization.k8s.io }
subjects: [ { kind: ServiceAccount, name: llm-api, namespace: web } ]Verify: kubectl auth can-i get configmaps -n web --as=system:serviceaccount:web:llm-api → yes; ... can-i get secrets ... → no. The workload references the SA via spec.serviceAccountName: llm-api. Needs a cluster; verify vs docs.
Context: Enforcing that pods run non-root and drop capabilities stops a whole class of container escapes.
Your task: Enforce the 'restricted' Pod Security Standard on a namespace and make a Pod comply, rolling it out safely.
Requirements:
- Label the namespace to enforce restricted
- Show the securityContext a compliant Pod needs
- Describe the safe rollout order
- Note it's admission-time, per-namespace
💡 Hint: audit + warn first, then enforce, so you don't break running deploys.
Show solution
# Namespace
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/warn: restricted
# Compliant Pod
securityContext: { runAsNonRoot: true, seccompProfile: { type: RuntimeDefault } }
containers:
- securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: [ALL] }Rollout: set audit+warn first, watch violations on next deploys, fix securityContexts, then flip to enforce. Pod Security Admission is built in and evaluated at admission per namespace. Needs a cluster; verify the versioned restricted requirements vs docs.
Context: Pod Security can't express 'images only from our registry' — that's a policy-engine job.
Your task: Write a Kyverno ClusterPolicy that blocks images not from your registry, and explain the safe rollout.
Requirements:
- Match Pods and validate the image prefix
- Give a clear failure message
- Explain Audit-then-Enforce
- Mention failurePolicy fail-open vs fail-closed and exempting kube-system
💡 Hint: validationFailureAction: Audit first; then Enforce once it's clean.
Show solution
kind: ClusterPolicy
metadata: { name: allowed-registries }
spec:
validationFailureAction: Enforce # start as Audit
rules:
- name: only-acme
match: { any: [ { resources: { kinds: [Pod] } } ] }
validate:
message: "images must come from ghcr.io/acme/"
pattern: { spec: { containers: [ { image: "ghcr.io/acme/*" } ] } }Rollout: deploy with Audit, review policy reports for violators, fix them, then flip to Enforce. Decide the webhook failurePolicy: Fail (safer, but blocks admission if the policy pod is down) vs Ignore; always exempt kube-system so a bad policy can't brick the control plane. Needs a cluster; verify vs Kyverno docs.
Context: Your GitOps repo must not contain plaintext secrets, but manual kubectl breaks the GitOps model.
Your task: Design a secrets flow that keeps plaintext out of Git while staying fully declarative, comparing two approaches.
Requirements:
- Explain why a k8s Secret alone isn't secure
- Present Sealed Secrets and External Secrets
- State when each is the better fit
- Add etcd encryption-at-rest as a baseline
💡 Hint: Commit ciphertext (Sealed) or a reference (External) — never the value.
Show solution
Why base64 isn't enough: a k8s Secret is base64-encoded, so anyone reading the object or etcd reads the value — enable encryption-at-rest (KMS on EKS) as a baseline and lock down secrets RBAC.
Sealed Secrets: kubeseal encrypts to the cluster's public key; commit the ciphertext, and only the in-cluster controller can decrypt it. Best when you want the secret (encrypted) to live in the repo.
External Secrets Operator: commit an ExternalSecret that references AWS Secrets Manager / Vault; the operator syncs the real value into a k8s Secret. Best when a cloud secret store is your source of truth and you want rotation. Both are declarative and GitOps-safe. Verify CRD apiVersions vs docs.
Context: A leaked registry cred or a typo'd public image running in prod is a supply-chain incident waiting to happen.
Your task: Design cluster admission that only admits images built and signed by your pipeline, tying it to cd6.
Requirements:
- Reference the cd6 build-time trust (scan, SBOM, cosign sign, provenance)
- Enforce signature verification at admission
- Require digest-pinning
- Keep ongoing scanning and state the three distinct layers
💡 Hint: Digest ≠ signature ≠ scan — you want all three, at different times.
Show solution
Build time (cd6): scan the image, generate an SBOM, sign with cosign, and attach SLSA provenance in the pipeline.
Admission time: a Kyverno verifyImages (or the sigstore policy-controller) rule rejects any ghcr.io/acme/* image not signed by your public key — so only pipeline-built images run.
verifyImages:
- imageReferences: [ "ghcr.io/acme/*" ]
attestors: [ { entries: [ { keys: { publicKeys: "" } } ] } ] Three distinct layers: (1) digest pinning so the tag can't be moved; (2) signature verification so only your builds run; (3) ongoing scanning because new CVEs hit already-deployed images. Needs a cluster; verify cosign/Kyverno fields vs current docs.
Context: Representative scenario: you run a multi-tenant LLM platform on EKS for several product teams and must pass a security review — 'assume a pod will be compromised; limit the blast radius'.
Your task: Produce a layered security design spanning identity, workload hardening, policy, secrets, supply chain and network, and name the failure mode each layer contains.
Requirements:
- Least-privilege RBAC per workload/tenant with no shared cluster-admin
- Pod Security restricted + policy engine for org rules
- GitOps-safe secrets with encryption-at-rest
- Signed-image admission tied to cd6
- Default-deny NetworkPolicy with an enforcing CNI
- Name the failure mode each layer defends against
💡 Hint: Each layer assumes the one outside it failed — that's defense-in-depth.
Show solution
Identity (RBAC): per-workload ServiceAccounts with minimal Roles, per-tenant namespaces, no cluster-admin on workloads, automountServiceAccountToken: false where unused. → contains a compromised pod to its namespace/verbs.
Workload hardening (Pod Security restricted): non-root, drop ALL caps, read-only rootfs, seccomp. → contains container escape / privilege escalation.
Policy engine (Kyverno): enforce org rules — approved registries, required labels, no :latest, no hostPath — audit-first, fail-closed with kube-system exempt. → contains misconfiguration drift.
Secrets: External/Sealed Secrets (no plaintext in Git), etcd encryption-at-rest, tight secrets RBAC. → contains secret leakage via repo/etcd.
Supply chain (cd6 + admission): scan + SBOM + cosign sign at build; verify signature + digest at admission; scan on a cadence. → contains malicious/vulnerable images.
Network: default-deny NetworkPolicy per namespace, allow only required flows, enforcing CNI (Calico/Cilium). → contains lateral movement.
The principle: every layer assumes the one outside it already failed, so no single flaw is catastrophic — the blast radius of a compromised pod is its namespace, its narrow RBAC, and its allowed network flows, nothing more. Verify every CRD/label/field against current docs; this area drifts fast.