Operators & CRDs

1 · The pattern you've used all along: control loops

You have been trusting this pattern for ten chapters without naming it. Every time you ran kubectl scale deployment ... --replicas=3 and walked away, you were relying on something to keep three Pods alive — to notice when a node died at 3 a.m. and quietly bring a replacement up before you woke. Nobody re-ran your command. That silent, tireless custodian is a control loop, and once you see it clearly here you will recognize it inside every Kubernetes object you have ever touched, and you'll be ready to write one of your own.

The best analogy is the thermostat on your wall. You don't tell it "turn the heater on for 12 minutes" — you tell it the goal: "keep this room at 21°C." From then on it does one thing in an endless loop: read the actual temperature, compare it to the target, and if there's a gap, act (heat or cool); if there's no gap, do nothing. It never assumes it "already handled" the room — it just keeps checking and correcting. A Kubernetes controller is that thermostat pointed at cluster state instead of air temperature. You declare the goal in an object's spec; the controller reads actual reality, diffs it against the goal, and nudges the world toward it, forever.

Now the precise version, which will land because you have the feel for it. Kubernetes is, at heart, a pile of controllers each running the same loop: observe the desired state (from an object's spec), observe the actual state (the world), compute the difference, and take action to close the gap — forever. A Deployment controller sees 'you want 3 replicas, there are 2' and creates one Pod. This is declarative: you state the goal, the controller continuously drives toward it.

The common mistake here is thinking in terms of commands instead of goals. Newcomers picture Kubernetes as a robot that executes a script — "create Pod, wait, create Pod" — and then are baffled when a Pod they deleted reappears, or frustrated that there's no "run this once" button. That mental model is wrong and it will fight you the entire chapter. The right model is a goal that is continuously enforced: you never command an action, you register a desired end-state and something works ceaselessly to maintain it. Hold onto that reframing, because an operator — the thing we build in this lesson — is nothing more than your own control loop enforcing your own goal. Master the loop and the rest of the chapter is just applying it to custom objects.

2 · CRDs: teaching the API server a new noun

Here's the surprising thing about Kubernetes: you can add your own words to its vocabulary. Out of the box the API knows nouns like Pod, Deployment, Service, ConfigMap. But nothing about those is privileged — you can teach the API server an entirely new noun, say LLMService, and from that moment it behaves like a first-class citizen: kubectl get llmservice works, it obeys RBAC, it lives in etcd, it shows up in kubectl explain. The tool that performs this teaching is a CustomResourceDefinition, and it is the foundation everything else in this lesson stands on.

Think of the Kubernetes API as a database with a fixed set of tables. A CRD is you running "CREATE TABLE" — you hand the API server a schema (what fields an LLMService has, which are required, what types they must be) and it creates a new place to store and validate objects of that shape. Just like a fresh database table, defining it doesn't make anything happen: an empty table doesn't process orders, and a CRD doesn't launch Pods. It gives you validated, queryable storage and nothing more. That distinction is the single most important idea in this section, so say it plainly: on its own a CRD is just structured storage with validation; it does nothing until a controller watches it.

The mistake almost everyone makes on their first CRD is expecting behavior. They write a beautiful LLMService schema, apply it, create an instance with replicas: 3, run kubectl get pods — and see nothing. Then they conclude the CRD is broken. It isn't. A CRD with no controller is like defining a form with no clerk to read it: the paperwork is filed and validated, but no one has been hired to act on it. The "clerk" is the operator in the next section. Keeping the noun (the data) mentally separate from the verb (the behavior) will save you hours of confusion. With that framing set, suppose we want a high-level LLMService that captures 'run this model with this many replicas' so app teams don't hand-write Deployments:

crd.yamlapiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata: { name: llmservices.ai.example.com }
spec:
  group: ai.example.com
  names: { kind: LLMService, plural: llmservices, singular: llmservice, shortNames: [llm] }
  scope: Namespaced
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                model:    { type: string }
                replicas: { type: integer, minimum: 1 }
              required: [model, replicas]

llmservice.yamlapiVersion: ai.example.com/v1
kind: LLMService
metadata: { name: chat, namespace: web }
spec:
  model: llama-3-8b
  replicas: 3
# `kubectl get llmservice -n web` now lists this. Nothing happens yet —
# a controller must watch LLMService and create the real Deployment/Service.

3 · Reconcile logic, modeled offline in Python

This is the section that turns "operators" from a buzzword into something you could write on a napkin. People assume the reconcile function at the core of every operator — the thing Kubebuilder scaffolds, the thing the Prometheus Operator and cert-manager and every cloud database operator are built around — must be some intricate distributed-systems machine. It is not. Strip away the cluster plumbing and the whole idea fits in about a dozen lines of ordinary Python, and you can run it right now with no cluster, no YAML, no dependencies. Get this function into your fingers and you understand the beating heart of the entire ecosystem.

The intuition is a checklist you re-run rather than a script you run once. Imagine a diligent building superintendent whose job is "the lobby should have three working light bulbs." Every round they walk in, count the working bulbs, and compare to the goal: zero bulbs → install three; three bulbs → touch nothing and move on; someone swapped in the wrong wattage → replace the wrong ones. Crucially, they don't keep a diary of "I installed bulbs last Tuesday" and refuse to look again — each round they judge only by what they see right now. That "act on the present gap, not on memory of past events" discipline is called being level-triggered, and the "safe to run the exact same check a thousand times" property is called idempotence. Reconcile is that superintendent's round, expressed in code.

The classic beginner error is writing reconcile as if it fires once per event. The tempting shape is "when a new LLMService is created, create a Deployment" — event-driven, edge-triggered thinking. It looks fine and it is quietly broken: miss the event (your operator was restarting), get a duplicate event, or have someone delete the Deployment out from under you, and an edge-triggered handler either does nothing or does the wrong thing twice. The fix is to never ask "what just happened?" and always ask "what's the difference between what I want and what exists right now?" — then act only on that difference. That is exactly what the code below does, and it's why re-running it is always safe. Here is the heart of every operator, with the cluster replaced by a plain dict so it runs offline. The reconcile function takes the desired Custom Resource and the actual world, and makes actual match — creating what's missing, updating what drifted, and doing nothing when already correct (idempotence). Study this and you understand Kubebuilder's Reconcile():

reconcile.pydef desired_deployment(cr):
    """Translate a high-level LLMService CR into the low-level object it implies."""
    return {"name": cr["name"], "replicas": cr["spec"]["replicas"],
            "image": f"vllm:{cr['spec']['model']}"}

def reconcile(cr, cluster):
    """Make the cluster match one LLMService. Returns the action taken.
    LEVEL-TRIGGERED: acts on the current gap. IDEMPOTENT: safe to re-run."""
    want = desired_deployment(cr)
    have = cluster.get(want["name"])
    if have is None:
        cluster[want["name"]] = dict(want)      # create what's missing
        return "created"
    if have != want:
        cluster[want["name"]].update(want)      # update drifted fields
        return "updated"
    return "noop"                                # already correct -> do nothing

# --- drive it like the control loop would ---
cr = {"name": "chat", "spec": {"model": "llama-3-8b", "replicas": 3}}
world = {}
print(reconcile(cr, world))          # created
print(reconcile(cr, world))          # noop   (idempotent: nothing changed)
cr["spec"]["replicas"] = 5
print(reconcile(cr, world))          # updated (drift closed)
del world["chat"]                   # something deleted our Deployment...
print(reconcile(cr, world))          # created (self-heals on next loop)

created
noop
updated
created

4 · Writing a controller: the conceptual walkthrough

You now hold the hardest 20% of an operator — the reconcile logic — so what's the other 80%? It's the wiring that keeps that little function fed with reality and lets it write reality back: waking it up when something changes, telling users what it decided, and cleaning up gracefully when an object is deleted. This section walks that wiring conceptually, and the good news is you will write almost none of it by hand — but you must understand it, because getting these lifecycle pieces wrong is how real operators leak resources and lie to their users.

A helpful analogy: your reconcile function is a skilled chef, but a restaurant is more than a chef. Someone has to seat customers and pass tickets to the kitchen (the watch), the kitchen has to send dishes out and call back "table 5 is ready" (status and events), and at closing time someone has to make sure the gas is off and the fridge locked before the doors are bolted (finalizers). The chef is the interesting, domain-specific part; the rest is the standard machinery every restaurant needs. An operator framework hands you a fully-staffed restaurant and asks only that you be the chef — you fill in Reconcile(), it runs the front-of-house.

The trap to avoid is treating the lifecycle pieces as optional garnish. It's tempting to write the reconcile body, see Pods appear, and declare victory — skipping status, skipping finalizers. That produces an operator that works in the demo and betrays you in production: with no status, users run kubectl describe and see nothing about why their service is stuck; with no finalizer, deleting the Custom Resource silently orphans the cloud bucket or DNS record it created, and you find out on the invoice. So read the table below not as trivia but as the checklist that separates a toy from something you'd let run unattended. A real operator wires that reconcile function to the API server. Conceptually the steps are the same whatever the SDK (Kubebuilder/Operator SDK in Go, or Kopf/client libraries in Python):

operator_loop.py# Pseudocode for the operator loop. Real code uses Kubebuilder (Go) or Kopf (Python).
for event in watch(kind="LLMService"):        # 2. wake on add/update/delete
    cr = event.object
    try:
        action = reconcile(cr, live_cluster)    # 3. the idempotent body
        set_status(cr, ready=True, note=action) # 4. write status back
    except TransientError:
        requeue(cr, backoff=True)               # retry later, don't crash
# The loop also periodically re-reconciles everything (resync) so it's level-triggered,
# not dependent on catching every event.

5 · How real platforms extend Kubernetes

Here's the payoff that recontextualizes the whole track: you've been an operator user since chapter 8. When you installed the Prometheus Operator for observability, or Argo Rollouts for progressive delivery, or cert-manager to mint TLS certificates, you weren't using "add-ons" — you were installing custom control loops that watch CRDs and reconcile them into real infrastructure, exactly the pattern you just built in a dozen lines of Python. The reason this lesson comes near the end of the Kubernetes track is that you can finally see what those tools were all along.

The mental model to lock in is operational knowledge as software. Think about what it takes to run production Postgres: a seasoned DBA carries a runbook — how to provision, when and how to back up, how to promote a replica when the primary dies, how to upgrade a version without losing data. Historically that knowledge lived in a human's head and a wiki page, and it executed only when someone was awake to run it. A database operator takes that runbook and compiles it into a control loop: the knowledge now runs continuously, at 3 a.m., without anyone paging the DBA. That is the deep idea — an operator is a runbook that reconciles itself, turning "steps a person performs" into "state a loop maintains."

Don't mistake this for a niche, exotic corner of Kubernetes. The common misconception is that operators are something only platform wizards touch; in reality the modern cluster is mostly operators — the pattern is how the ecosystem distributes expertise. Every entry in the table below is a team packaging their hard-won domain knowledge so you get it as a short Custom Resource instead of a manual procedure, and recognizing them as instances of one pattern is what lets you reason about, debug, and choose between them. You have already been using operators throughout this track — that's the point. The pattern is how the ecosystem ships operational knowledge as software:

The recurring win: a database operator encodes a DBA's runbook — provision, take backups, fail over, upgrade — as a controller, so a team gets 'production Postgres' from a short Custom Resource instead of a wiki page of manual steps. That is the operator value proposition: operational knowledge as code, reconciled continuously.

6 · When to build an operator — and when not to

Now that you know how to build an operator, the most valuable skill is knowing when not to. There's a predictable moment on every platform team: someone learns the pattern, gets excited, and proposes "let's build an operator for that." It's the Kubernetes version of "when all you have is a hammer, everything looks like a nail" — and it produces expensive, permanently-running, over-permissioned components that solve problems a ten-line Helm chart already solved. This section is the antidote: a clear-eyed cost/benefit test so your enthusiasm for the pattern doesn't cost your team a maintenance burden they'll resent.

Frame it as a genuine cost, not a free capability. An operator isn't a script you run and forget — it's a long-lived service that sits in your cluster with broad RBAC permissions (it has to create and delete Deployments, Secrets, and more), which means it's another thing to version, patch for CVEs, monitor, and reason about when the cluster misbehaves. You are, in effect, hiring a full-time robot employee and taking on the duty to manage it forever. That's completely worth it when the robot does a job no simpler tool can — and pure overhead when it doesn't. So the question is never "could I build an operator for this?" (you almost always could) but "does this job actually need one?"

The reliable tell is whether the problem is a control-loop problem. An operator earns its keep only when there's genuine continuous reconciliation of state plus real operational logic — the backup/failover/rotate/self-heal kind of work that has to keep happening, unattended, in response to a drifting world. If instead you just need to template and install a bundle of manifests, that's Helm (K6). If you need to run a task on a schedule, that's a CronJob. If it's a one-time setup, it's a script. The honest test that cuts through the excitement: do you have domain operational logic that must run continuously and automatically, that plain Kubernetes objects can't express? Operators are powerful and are also a maintained piece of software running in your cluster with broad permissions. Building one is a real, ongoing commitment.

kind: CustomResourceDefinition
metadata: { name: llmservices.ai.example.com }
spec:
  group: ai.example.com
  names: { kind: LLMService, plural: llmservices }
  scope: Namespaced
  versions: [ { name: v1, served: true, storage: true, schema: { openAPIV3Schema:
    { type: object, properties: { spec: { type: object, properties:
      { model: {type: string}, replicas: {type: integer} } } } } } } ]
---
kind: LLMService
apiVersion: ai.example.com/v1
metadata: { name: chat }
spec: { model: llama-3-8b, replicas: 3 }

def reconcile(cr, world):
    want = {'name': cr['name'], 'replicas': cr['spec']['replicas']}
    have = world.get(want['name'])
    if have is None:
        world[want['name']] = dict(want);  return 'created'
    if have != want:
        world[want['name']].update(want);  return 'updated'
    return 'noop'

cr = {'name':'chat','spec':{'replicas':3}}; w = {}
reconcile(cr, w)          # created
reconcile(cr, w)          # noop
cr['spec']['replicas']=5; reconcile(cr, w)   # updated
del w['chat']; reconcile(cr, w)              # created (self-heal)