Networking, Ingress & storage

1 · The cluster networking model

Picture the moment your app first fails in a way that makes no sense. Your frontend Pod is running, your backend Pod is running, both are green in kubectl get pods — and yet the frontend can't reach the backend. You spend an hour suspecting your code before someone asks the real question: "how is one Pod even supposed to find another?" That question — networking — is invisible until it breaks, and then it's the only thing that matters. This section gives you the mental model that turns those mysteries into two-minute diagnoses.

Here's the analogy that makes it click. Think of your cluster as an apartment building where every tenant (Pod) gets their own phone line with a direct number, and any tenant can dial any other tenant with no operator in between. That's the founding promise of Kubernetes networking: a flat, fully-connected space. There's no "wall" between Pods by default, no address translation to untangle — Pod A can talk to Pod B as if they were on the same LAN, even when they're on physically different machines across a datacenter.

The common mistake is assuming that because you're used to Docker on one laptop — where containers share a host and you juggle port mappings — Kubernetes works the same way. It doesn't. There's no -p 8080:80 port-forwarding tax between Pods; each Pod has a real, routable IP. The catch that trips people is the opposite of what they expect: connectivity is too open by default (any Pod can reach any Pod), which is exactly why §4's NetworkPolicies exist to lock it back down.

Kubernetes networking has one founding rule: every Pod gets its own IP, and every Pod can reach every other Pod directly — no NAT between Pods. A CNI plugin (Calico, Cilium, the AWS VPC CNI in EKS) implements this. On top of it sit the abstractions from K2: a Service gives a stable virtual IP, and kube-proxy (or eBPF) makes that IP load-balance to Pod IPs.

2 · Cluster DNS

Every flat network in §1 hides a problem: IPs don't hold still. A Pod that gets rescheduled to another node comes back with a brand-new IP; scale a Service from 3 Pods to 10 and there are seven addresses that didn't exist a second ago. If your frontend had hardcoded 10.1.4.7 for the backend, it now points at nothing — or worse, at some unrelated Pod that inherited that address. Something has to translate a stable name into whatever IPs are live right now. That something is cluster DNS.

The analogy is a company phone directory. You don't memorize your colleague's desk-phone number and re-memorize it every time they change desks — you look up "Priya in Payments" and the directory returns wherever she sits today. Cluster DNS is that directory for Services: you ask for backend.payments by name, and it hands back the Service's current virtual IP, which then load-balances across whatever Pods are healthy. The name is the contract; the addresses underneath are free to churn.

The mistake to name is the reflex to debug with raw IPs. When something can't connect, it's tempting to kubectl get pod -o wide, copy a Pod IP, and curl it directly to "prove" the network works. That test lies to you: it might pass this second and fail the next when the Pod moves, and it teaches you nothing about the path real traffic takes. The professional habit is to always address Services by their DNS name and treat a name-resolution failure — not an IP ping — as the signal. The naming pattern below is worth burning into memory, because the same rules explain both "why does the short name work here but not there?" and most cross-namespace connectivity bugs.

Every Service gets a DNS name automatically, so code never hardcodes IPs. The pattern is <service>.<namespace>.svc.cluster.local. Within the same namespace you can use just the short <service> name.

dns.sh# same namespace -> short name works:
curl http://backend/healthz
# cross namespace -> fully-qualified:
curl http://backend.payments.svc.cluster.local/healthz
# see the DNS records a Service exposes:
kubectl run tmp --rm -it --image=busybox -- nslookup backend.payments.svc.cluster.local

3 · Ingress and ingress controllers

Imagine giving every service in your company its own street address and its own front-desk receptionist. Ten microservices, ten public IPs, ten cloud load balancers, ten monthly bills, ten TLS certificates to renew. That's what happens if you expose each Service as a LoadBalancer, and it gets absurd fast. What you actually want is one grand entrance to the building, with a receptionist who reads where a visitor is headed and points them down the right hallway. That single front door is an Ingress.

Concretely, an Ingress is a set of routing rules: "requests for app.example.com/api go to the backend Service; everything else on that host goes to the frontend Service; and by the way, terminate HTTPS here so the internal Services can speak plain HTTP." It's Layer 7 — it understands hostnames and URL paths, not just IPs and ports — which is why it can fan one address out to many backends by reading the request. One load balancer, one certificate, many services behind it.

Now the mistake that costs people an entire afternoon, and it's important enough that this section returns to it twice more. An Ingress object is just a document. Writing the rules and running kubectl apply creates the spec, but nothing reads it and nothing routes traffic — because the thing that does the routing is a separate program called an ingress controller (NGINX, Traefik, or the AWS Load Balancer Controller) that you install yourself and that watches for Ingress objects to fulfill. The Ingress is the receptionist's instructions; the controller is the actual receptionist. Apply an Ingress on a cluster with no controller and you get eerie silence: the object exists, kubectl get ingress shows it, and not one byte reaches your Pods. Wire the two together with a matching ingressClassName and it springs to life.

A LoadBalancer Service gives one external IP per service — expensive and unwieldy. An Ingress is a single HTTP(S) entry point that routes by host and path to many backend Services, and terminates TLS. Crucially, an Ingress object is inert without an ingress controller (NGINX, Traefik, or the AWS Load Balancer Controller) actually running to fulfill it.

ingress.yamlapiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  annotations:
    # controller-specific annotations go here; verify against your controller's docs
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx           # which controller fulfills this
  tls:
    - hosts: [app.example.com]
      secretName: web-tls           # a TLS Secret (e.g. from cert-manager)
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service: { name: backend, port: { number: 80 } }
          - path: /
            pathType: Prefix
            backend:
              service: { name: frontend, port: { number: 80 } }

4 · NetworkPolicies

Here's the uncomfortable consequence of §1's flat network. If any Pod can reach any Pod, then the day an attacker slips into your least important Pod — a public-facing frontend, a scrappy little metrics sidecar — they can open a connection straight to your database and start reading rows. Nothing in the default cluster stops them. Most breaches aren't a straight line to the crown jewels; they're a foothold in something trivial followed by a quiet walk across an open network. NetworkPolicies are how you close those hallways.

Think of it as the difference between an open-plan office where anyone can wander into any room, and a building with badge readers on every door. A NetworkPolicy is a Pod-level firewall: you select a group of Pods by their labels and declare who is allowed to connect to them and on which ports. "Only Pods labelled app: backend may reach Pods labelled app: db, and only on 5432" is a sentence you can write directly as policy.

The subtlety that catches everyone is how you get to least-privilege. The intuitive move is to add "allow" rules for the traffic you want — but on a flat network, allow-rules alone change nothing, because everything is already allowed. The correct pattern is inverted: first apply a default-deny that shuts all inbound traffic to a namespace, which flips the network from open to closed, and only then punch specific holes for the connections you actually need. Deny first, allow narrowly — the same instinct as a firewall's default-drop rule. There's also a quieter trap the callout below dwells on: a policy is only real if your CNI plugin enforces it. On a cluster whose CNI ignores NetworkPolicy, you can apply perfect-looking rules and feel secure while nothing is actually blocked — the worst kind of failure, because it's invisible.

By default, every Pod can talk to every other Pod — a flat network. In production you want least-privilege: a database should only accept connections from its app. A NetworkPolicy is a Pod-level firewall selected by labels. The standard pattern is default-deny, then explicitly allow what's needed.

netpol.yamlapiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
spec:
  podSelector: {}                 # all Pods in this namespace
  policyTypes: [Ingress]          # deny all inbound by default
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: db-allow-app
spec:
  podSelector: { matchLabels: { app: db } }   # applies to db Pods
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector: { matchLabels: { app: backend } }  # only backend may connect
      ports:
        - { protocol: TCP, port: 5432 }

5 · Persistent storage — PV, PVC, StorageClass

The first time this bites is unforgettable. You run a database in a Pod, load it with data, everything works — then the Pod restarts (a node reboot, a rollout, an OOM kill) and your data is gone. Not corrupted: gone, as if you'd never written it. That's not a bug; it's the design. A Pod's container filesystem is scratch space, wiped clean on every restart, because Kubernetes assumes Pods are cattle you can kill and recreate at will. Anything you want to keep has to live somewhere the Pod's death can't touch.

The mental model is a hotel room versus a storage unit. The Pod is a hotel room — you get a fresh, clean one each stay, and whatever you leave on the desk when you check out is thrown away. Persistent storage is the rented storage unit down the street: it stays yours across check-ins, and you mount it into today's room. When the Pod restarts and a new one takes its place, that new Pod can mount the same storage unit and pick up exactly where the last one left off.

Now the three-word vocabulary people constantly muddle — PV, PVC, StorageClass — because they sound interchangeable and aren't. The clean way to hold them apart is by who owns each. A PersistentVolume (PV) is the actual disk — a real EBS volume, a real NFS export — the physical storage unit itself. A PersistentVolumeClaim (PVC) is your app's request for storage ("I need 20Gi I can write to"); the Pod only ever names a PVC, never a raw PV, so the app stays ignorant of the underlying disk. A StorageClass is the menu and the vending machine: it says "class gp3 means an AWS gp3 SSD" and, when a claim comes in, automatically provisions a matching PV to satisfy it. The claim is the order slip, the StorageClass is the fulfillment engine, the PV is the goods. Keep that ownership split straight and the YAML below reads itself; blur it and every storage bug looks like magic.

A Pod's container filesystem is ephemeral: restart the Pod and it's wiped. For data that must survive — a database, uploaded files, a model cache — you need a PersistentVolume (PV), a piece of durable storage. Pods don't reference PVs directly; they make a PersistentVolumeClaim (PVC) — a request for storage — and a StorageClass dynamically provisions a matching PV (e.g. an EBS volume on AWS).

pvc.yamlapiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: model-cache
spec:
  accessModes: [ReadWriteOnce]     # one node mounts it read-write
  storageClassName: gp3            # a StorageClass in the cluster (e.g. EBS gp3)
  resources:
    requests:
      storage: 20Gi
---
# ...in a Pod/Deployment container spec:
#   volumeMounts: [ { name: cache, mountPath: /models } ]
# spec.volumes:
#   - name: cache
#     persistentVolumeClaim: { claimName: model-cache }

6 · StatefulSets — when identity matters

Try to run a 3-replica database as an ordinary Deployment and you'll produce a catastrophe that looks fine for a week. A Deployment says "give me three copies of this Pod, I don't care which is which" — the Pods get random names, come and go interchangeably, and can share or lose storage freely. That's perfect for stateless web servers and exactly wrong for a database, where three processes suddenly writing to one disk (or disagreeing about who's the primary) quietly corrupt each other's data. The knowledge check earlier in this page is precisely that story. StatefulSets exist because some workloads simply cannot be anonymous and interchangeable.

Think of the difference between temp workers and named employees. A Deployment's Pods are temps from an agency: any one will do, they're numbered by whoever's on shift, and you don't hand a temp the keys to a specific locked cabinet. A StatefulSet's Pods are named employees: db-0 is always db-0, it always comes back to the same desk with the same locked drawer (its own PVC), and it has a stable phone extension others can rely on. When db-0 restarts, it returns as db-0 attached to its original disk — identity and storage preserved — which is the whole point.

The mistake goes in both directions, so hold two rules at once. Reaching for a StatefulSet by default "to be safe" is over-engineering — most of your workloads are genuinely stateless and belong in a simpler, faster-scaling Deployment; a StatefulSet's ordered, one-at-a-time startup is a real cost you shouldn't pay without reason. But forcing a stateful workload into a Deployment is the dangerous error, the data-corrupting one. The honest test is a single question: does each replica need a stable name and its own durable disk? Databases, Kafka brokers, and clustered stores answer yes; almost everything else answers no. And the most senior answer of all, which the tip below and Exercise 5 both push, is often to dodge the question entirely — run the database as a managed service and keep your cluster blissfully stateless.

A Deployment's Pods are interchangeable and get random names. Some workloads — databases, message brokers, anything where each replica has a stable identity and its own persistent disk — need more. A StatefulSet gives each Pod a stable ordinal name (db-0, db-1), a stable network identity, and its own PVC, created in order.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: { name: web }
spec:
  ingressClassName: nginx
  tls: [ { hosts: [app.example.com], secretName: web-tls } ]
  rules:
    - host: app.example.com
      http:
        paths:
          - { path: /api, pathType: Prefix, backend: { service: { name: backend,  port: { number: 80 } } } }
          - { path: /,    pathType: Prefix, backend: { service: { name: frontend, port: { number: 80 } } } }

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny-ingress }
spec: { podSelector: {}, policyTypes: [Ingress] }
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: db-allow-backend }
spec:
  podSelector: { matchLabels: { app: db } }
  policyTypes: [Ingress]
  ingress:
    - from: [ { podSelector: { matchLabels: { app: backend } } } ]
      ports: [ { protocol: TCP, port: 5432 } ]

apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: model-cache }
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: gp3
  resources: { requests: { storage: 20Gi } }

        volumeMounts: [ { name: cache, mountPath: /models } ]
      volumes:
        - name: cache
          persistentVolumeClaim: { claimName: model-cache }