Operators & CRDs
Every controller you have used — Deployment, HPA, Argo Rollouts, the Prometheus Operator — is the same idea: a control loop that watches desired state and works to make actual state match. This lesson teaches that pattern directly: Custom Resource Definitions (CRDs) that extend the Kubernetes API with your own objects, and operators — controllers that reconcile those objects into real infrastructure. We model the reconcile loop in plain Python (offline-runnable), walk conceptually through writing a controller, survey how real platforms extend k8s, and answer the money question: when to build an operator vs when not to.
kubectl here need a cluster (minikube/kind local, or EKS) and output is illustrative. Operator SDKs (Kubebuilder, Operator SDK, the client libraries) and CRD schema fields drift — verify against current docs. The core payoff of this lesson, the reconcile loop, is modeled in plain Python that runs offline with no cluster or dependencies, so you can internalize the pattern before touching a real controller.Learning objectives
- Explain the controller pattern: watch → diff desired vs actual → act → repeat (a level-triggered loop).
- Define a CRD to extend the Kubernetes API with a custom object, and a Custom Resource instance.
- Describe what an operator is: a controller + CRDs that encodes operational knowledge.
- Trace and implement reconcile logic (modeled offline in Python) that is idempotent and level-triggered.
- Use existing operators to run stateful software, and know how real platforms extend k8s.
- Decide when to build an operator vs not — the cost/benefit that keeps you honest.
1 · The pattern you've used all along: control loops
Kubernetes is, at heart, a pile of controllers each running the same loop: observe the desired state (from an object's spec), observe the actual state (the world), compute the difference, and take action to close the gap — forever. A Deployment controller sees 'you want 3 replicas, there are 2' and creates one Pod. This is declarative: you state the goal, the controller continuously drives toward it.
2 · CRDs: teaching the API server a new noun
A CustomResourceDefinition adds a new kind to the Kubernetes API — after you apply one, kubectl get <yourkind> works, RBAC applies, and it's stored in etcd, exactly like a built-in object. On its own a CRD is just structured storage with validation; it does nothing until a controller watches it. Suppose we want a high-level LLMService that captures 'run this model with this many replicas' so app teams don't hand-write Deployments:
crd.yamlapiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata: { name: llmservices.ai.example.com }
spec:
group: ai.example.com
names: { kind: LLMService, plural: llmservices, singular: llmservice, shortNames: [llm] }
scope: Namespaced
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
model: { type: string }
replicas: { type: integer, minimum: 1 }
required: [model, replicas]
llmservice.yamlapiVersion: ai.example.com/v1
kind: LLMService
metadata: { name: chat, namespace: web }
spec:
model: llama-3-8b
replicas: 3
# `kubectl get llmservice -n web` now lists this. Nothing happens yet —
# a controller must watch LLMService and create the real Deployment/Service.
LLMService objects and query them, but the cluster won't create a single Pod from one. The operator (next section) is the controller that watches LLMService objects and reconciles each into the Deployment, Service, HPA, etc. it implies. Split the concepts: CRD is the noun; the operator is the verb.3 · Reconcile logic, modeled offline in Python
Here is the heart of every operator, with the cluster replaced by a plain dict so it runs offline. The reconcile function takes the desired Custom Resource and the actual world, and makes actual match — creating what's missing, updating what drifted, and doing nothing when already correct (idempotence). Study this and you understand Kubebuilder's Reconcile():
reconcile.pydef desired_deployment(cr):
"""Translate a high-level LLMService CR into the low-level object it implies."""
return {"name": cr["name"], "replicas": cr["spec"]["replicas"],
"image": f"vllm:{cr['spec']['model']}"}
def reconcile(cr, cluster):
"""Make the cluster match one LLMService. Returns the action taken.
LEVEL-TRIGGERED: acts on the current gap. IDEMPOTENT: safe to re-run."""
want = desired_deployment(cr)
have = cluster.get(want["name"])
if have is None:
cluster[want["name"]] = dict(want) # create what's missing
return "created"
if have != want:
cluster[want["name"]].update(want) # update drifted fields
return "updated"
return "noop" # already correct -> do nothing
# --- drive it like the control loop would ---
cr = {"name": "chat", "spec": {"model": "llama-3-8b", "replicas": 3}}
world = {}
print(reconcile(cr, world)) # created
print(reconcile(cr, world)) # noop (idempotent: nothing changed)
cr["spec"]["replicas"] = 5
print(reconcile(cr, world)) # updated (drift closed)
del world["chat"] # something deleted our Deployment...
print(reconcile(cr, world)) # created (self-heals on next loop)
created
noop
updated
created4 · Writing a controller: the conceptual walkthrough
A real operator wires that reconcile function to the API server. Conceptually the steps are the same whatever the SDK (Kubebuilder/Operator SDK in Go, or Kopf/client libraries in Python):
| Step | What you do | Why |
|---|---|---|
| 1. Define the CRD | Schema for your kind (spec + status) | Gives the API server the new noun |
| 2. Watch | Subscribe to add/update/delete of your kind (and its children) | So the loop wakes on relevant changes |
| 3. Reconcile | Diff desired vs actual; create/update/delete children idempotently | The control-loop body (section 3) |
| 4. Status + events | Write observed state back to the CR; emit events | Users see what the operator did |
| 5. Finalizers | Clean up external resources before the CR is deleted | No orphaned infra on delete |
operator_loop.py# Pseudocode for the operator loop. Real code uses Kubebuilder (Go) or Kopf (Python).
for event in watch(kind="LLMService"): # 2. wake on add/update/delete
cr = event.object
try:
action = reconcile(cr, live_cluster) # 3. the idempotent body
set_status(cr, ready=True, note=action) # 4. write status back
except TransientError:
requeue(cr, backoff=True) # retry later, don't crash
# The loop also periodically re-reconciles everything (resync) so it's level-triggered,
# not dependent on catching every event.
Reconcile() — the exact function you modeled offline. In Python, Kopf lets you decorate a handler. Focus your effort on correct, idempotent reconcile logic and good status reporting; let the framework own the plumbing. Verify the current scaffolding commands against the tool's docs.5 · How real platforms extend Kubernetes
You have already been using operators throughout this track — that's the point. The pattern is how the ecosystem ships operational knowledge as software:
| Operator / CRDs | What it manages | Seen in |
|---|---|---|
| Prometheus Operator | Prometheus, ServiceMonitor, PrometheusRule | K8 (observability) |
| Argo Rollouts / Argo CD | Rollout, Application CRDs | K10 / K6 (GitOps) |
| cert-manager | Certificate CRDs → issued + rotated TLS certs | Ingress/mesh TLS |
| External Secrets / Sealed Secrets | ExternalSecret / SealedSecret → real Secrets | K9 (security) |
| Cloud DB / Kafka / Elastic operators | Stateful software with backups, failover, upgrades | Data platforms |
The recurring win: a database operator encodes a DBA's runbook — provision, take backups, fail over, upgrade — as a controller, so a team gets 'production Postgres' from a short Custom Resource instead of a wiki page of manual steps. That is the operator value proposition: operational knowledge as code, reconciled continuously.
6 · When to build an operator — and when not to
Operators are powerful and are also a maintained piece of software running in your cluster with broad permissions. Building one is a real, ongoing commitment. The honest test: do you have domain operational logic that must run continuously and automatically, that plain Kubernetes objects can't express?
| Build/adopt an operator when… | …don't build one when… |
|---|---|
| You repeat a multi-step operational runbook often (backup, failover, rotate) | A Deployment/Job/CronJob already expresses it |
| You want a high-level abstraction for many teams (self-service) | One team, one app — a Helm chart is enough |
| State must be continuously reconciled/self-healed | A one-time setup script would do |
| A mature operator exists for your stateful software (adopt it) | You'd be reinventing a well-supported one |
✓ Checkpoint — you can move on when you can…
- Explain the reconcile control loop and why level-triggered + idempotent makes it robust.
- Define a CRD and a Custom Resource, and say why a CRD alone does nothing.
- Trace the offline reconcile: created → noop → updated → self-heal, and why each result happens.
- List the conceptual steps to write a controller (CRD, watch, reconcile, status, finalizers).
- Name three real operators you've used in this track and what they manage.
- Decide when to build/adopt an operator vs use Helm or a CronJob instead.
Someone deletes the Deployment that your operator created from an LLMService. Minutes later it's back, and nobody re-ran anything. Explain the mechanism, and what property of the reconcile code makes this safe.
Show answer
created → noop → updated → created behavior in the offline model.A team wants to 'build an operator' to install their app's 5 manifests with a couple of configurable values, and to run a nightly cleanup job. Is an operator the right tool? What would you suggest?
Show answer
values.yaml covers it with far less to maintain than a custom controller. A nightly cleanup is a CronJob. An operator is justified only when you need continuous reconciliation of custom state and real operational logic that plain objects can't express (backup/failover/rotation, or a self-service abstraction for many teams). Building an operator here would add a permanently-running, broadly-permissioned component to maintain for no benefit — pick the simpler tool.🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: Every k8s controller — and every operator — is the same loop; naming it is the first step.
Your task: In your own words, describe the reconcile loop and why it's called level-triggered and declarative.
Requirements:
- State the four steps of the loop
- Define level-triggered vs edge-triggered
- Explain why that makes controllers robust
- Connect it to a built-in like the Deployment controller
💡 Hint: It acts on the current gap, not on a one-time event.
Show solution
The loop: (1) read desired state (spec), (2) observe actual state (the world), (3) diff them, (4) act to close the gap — then repeat forever.
Level- vs edge-triggered: edge-triggered acts on a one-off event ('a pod died'); level-triggered acts on the current gap ('desired 3, have 2'). Level-triggered is robust because even if the controller missed events (it was down), it still fixes the gap on the next pass.
Declarative: you state the goal; the controller continuously drives toward it. The Deployment controller is exactly this — you say '3 replicas', it keeps 3 alive no matter what kills them.
Context: Extending the API with your own kind is the foundation of every operator.
Your task: Define a CRD for a custom kind and write one instance, and explain what does (and doesn't) happen on apply.
Requirements:
- A CRD with a small validated spec
- A Custom Resource instance of that kind
- State what kubectl can now do
- State why no workload is created yet
💡 Hint: A CRD is validated storage; behavior needs a controller.
Show solution
kind: CustomResourceDefinition
metadata: { name: llmservices.ai.example.com }
spec:
group: ai.example.com
names: { kind: LLMService, plural: llmservices }
scope: Namespaced
versions: [ { name: v1, served: true, storage: true, schema: { openAPIV3Schema:
{ type: object, properties: { spec: { type: object, properties:
{ model: {type: string}, replicas: {type: integer} } } } } } } ]
---
kind: LLMService
apiVersion: ai.example.com/v1
metadata: { name: chat }
spec: { model: llama-3-8b, replicas: 3 }After apply, kubectl get llmservice works and the object is validated + stored in etcd. But no Pod is created — a CRD is only structured storage; behavior requires a controller that watches this kind. Needs a cluster; verify schema fields vs docs.
Context: Idempotent reconcile is the single most important property of operator code.
Your task: Write offline Python that reconciles a desired CR into an actual world and returns created/updated/noop, and demonstrate self-heal.
Requirements:
- Translate the CR into the object it implies
- Create if missing, update if drifted, noop if equal
- Show it's safe to run repeatedly
- Show self-heal after a delete
💡 Hint: Compare want vs have; only act on a real difference.
Show solution
def reconcile(cr, world):
want = {'name': cr['name'], 'replicas': cr['spec']['replicas']}
have = world.get(want['name'])
if have is None:
world[want['name']] = dict(want); return 'created'
if have != want:
world[want['name']].update(want); return 'updated'
return 'noop'
cr = {'name':'chat','spec':{'replicas':3}}; w = {}
reconcile(cr, w) # created
reconcile(cr, w) # noop
cr['spec']['replicas']=5; reconcile(cr, w) # updated
del w['chat']; reconcile(cr, w) # created (self-heal)Because it only acts on the gap, re-running is safe (noop when equal) and a deleted object is recreated on the next pass — that's idempotence + level-triggering. Runs offline; no cluster needed.
Context: A production operator is reconcile plus the lifecycle concerns around it.
Your task: Describe the pieces a real controller adds around the reconcile body and why each matters.
Requirements:
- Watch (and periodic resync)
- Status subresource + events
- Finalizers for cleanup on delete
- Owner references and requeue-with-backoff
💡 Hint: These are the difference between a toy loop and a safe operator.
Show solution
Watch + resync: subscribe to add/update/delete of your kind and its children, and periodically re-reconcile everything so the loop stays level-triggered even if events are missed.
Status + events: write observed state back to the CR's status and emit Kubernetes events so users can kubectl describe and see what the operator did/why it's waiting.
Finalizers: a finalizer blocks deletion until the operator cleans up external resources (e.g. a cloud bucket) — otherwise you orphan infra.
Owner references + backoff: set the CR as the owner of children so they're garbage-collected with it; on transient errors, requeue with backoff rather than crashing. Frameworks (Kubebuilder/Kopf) provide all this scaffolding — you fill in reconcile. Verify SDK specifics vs docs.
Context: A team wants production Postgres on Kubernetes and is debating a hand-rolled StatefulSet vs an operator.
Your task: Make a reasoned recommendation, covering what an operator encodes and the risks of rolling your own.
Requirements:
- List the operational tasks production Postgres needs
- Explain what a mature DB operator encodes
- State the risks of a hand-rolled StatefulSet + scripts
- Give a recommendation with caveats
💡 Hint: Backups, failover, and safe upgrades are the hard, ongoing parts.
Show solution
What production Postgres needs: provisioning, automated backups + restore, failover (promote a replica on primary loss), safe version upgrades, connection routing, and monitoring — continuously, not once.
What a mature operator encodes: exactly those runbooks as a control loop, so a short Postgres CR yields a self-healing, backed-up, failover-capable cluster — years of DBA knowledge as software.
Risks of rolling your own: a StatefulSet + scripts gives you storage and identity but not backup/failover/upgrade logic; you'd be reimplementing (and forever maintaining) the hard parts, and data systems are unforgiving of bugs.
Recommendation: adopt a mature, well-supported operator rather than build — after verifying its maturity, community, and version compatibility with your cluster. Build custom only if no suitable operator exists and the operational logic is genuinely yours.
Context: Representative scenario: your platform team is asked to give product teams 'one-line' LLM services and to run several stateful systems, and there's enthusiasm to 'build operators for everything'.
Your task: Produce a strategy for how to extend Kubernetes: where a custom operator is justified, where to adopt existing ones, and where Helm/CronJobs suffice — with the failure mode each choice avoids.
Requirements:
- A self-service LLMService abstraction — build or not?
- Stateful systems (DB/Kafka/search) — adopt existing operators
- Simple installs and scheduled tasks — Helm / CronJob
- The maintenance/permission cost of every operator you run
- Name the failure mode each choice avoids
💡 Hint: Operators are for continuous reconciliation + real ops logic; everything else is simpler.
Show solution
Self-service LLMService (build — carefully): if many teams need a high-level 'run this model' abstraction that reconciles Deployment+Service+HPA+autoscaling policy continuously, a custom operator is justified — it encodes your serving runbook and self-heals. Scaffold with Kubebuilder/Kopf; invest in idempotent reconcile, status, and finalizers. → avoids every team hand-writing (and drifting) low-level manifests.
Stateful systems (adopt, don't build): use mature operators for Postgres/Kafka/Elastic — they encode backup/failover/upgrade. → avoids reinventing hard, data-loss-prone operational logic.
Simple installs & tasks (Helm / CronJob): templated manifests → Helm (K6); scheduled cleanup/reporting → CronJob. → avoids paying operator complexity for non-control-loop problems.
The cost lens: every operator is a permanently-running, broadly-permissioned component you must maintain, upgrade, and secure (K9). So the strategy is: build few, adopt many, template the rest — reach for a custom control loop only when there's genuine continuous reconciliation and domain ops logic. Verify all CRD/SDK/operator specifics against current docs; this ecosystem drifts fast.