Scalability & CAP

1 · Two ways to scale essential

Here's the moment every successful product eventually hits. Your app was fine for months. Then a post goes viral, or a big customer onboards their whole team, and suddenly the dashboards go red: CPU pinned at 100%, requests queueing, timeouts. The 3am question is brutally simple — the one server is full, now what? There are only two answers to that question, and knowing which one to reach for (and why) is the entire foundation of scalability.

The everyday analogy is a restaurant that's slammed. You have two moves. Move one: replace your one overworked chef with a faster, stronger super-chef who can cook twice as much. That's vertical scaling — same single worker, just bigger. Move two: hire a second chef, then a third, and split the orders between them. That's horizontal scaling — more workers, work divided. The first is instant and requires zero change to how the kitchen runs; the second is more work to coordinate but has no ceiling.

And that ceiling is the whole point. The super-chef eventually can't get any faster — there's no chef alive who cooks a thousand meals a minute, just as there's no single machine with unlimited CPU and RAM. Vertical scaling is capped (the biggest cloud box is only so big) and gets expensive fast — doubling a machine's power often more than doubles its price. Horizontal scaling, by contrast, uses cheap commodity boxes and grows almost without limit: Google and Netflix don't run on one titanic computer, they run on fleets of ordinary ones. That's why modern systems scale horizontally.

The common mistake is thinking these are equal options you pick by taste. They're not — they're a sequence. You reach for vertical first because it's dead simple (bump the instance size, no code changes, done by lunch), and you switch to horizontal when vertical runs out of room. The catch — the thing this whole chapter really teaches — is that horizontal scaling only works if your app servers are stateless: any box must be able to handle any request. Hire a second chef, but if only the first chef knows which orders are half-cooked, adding staff just creates chaos. Get statelessness wrong and adding machines makes things worse, not better. The diagram traces that exact progression: load grows → go vertical → then go horizontal.

Vertical = a bigger machine (simple, but capped and pricey). Horizontal = more machines (cheap, effectively unlimited — but requires stateless app servers). Modern systems scale horizontally.

2 · Statelessness, demonstrated essential

This is the bug that ruins the launch you were so proud of. A team adds a second server behind a load balancer to handle the new traffic — textbook horizontal scaling. It deploys clean. Then users start reporting they're randomly logged out mid-session, carts empty themselves, half-filled forms vanish. Nothing is broken in any single server; each one works perfectly on its own. The problem is that requests bounce between two boxes that don't share what they remember. That invisible failure has a name, and avoiding it is what "stateless" means.

Here's the plain-language version. State is just "stuff a server remembers between requests" — who's logged in, what's in your cart, how far through a wizard you are. A server is stateful when it keeps that memory inside itself, and stateless when it keeps none and instead reads and writes everything to a shared place outside itself (a cache or database). The analogy that makes it stick: a stateful server is a barista who remembers your usual by face — great until you walk into a different branch where nobody knows you. A stateless server is a coffee chain where your order lives on an app both branches can read; walk into any store, they pull up your usual instantly. The knowledge isn't in the barista's head — it's in the shared system.

Why this earns a whole section: statelessness is the precondition for everything else in this chapter. Load balancers, auto-scaling, rolling deploys, surviving a crashed box — all of them assume any server can pick up any request. The moment a server hoards state, you're forced into "sticky sessions" (pinning each user to one box), which quietly cancels most of the benefit of having many boxes: you can't balance load evenly, you can't drain a box for maintenance without dropping its users, and one crash logs out everyone it was holding.

The common mistake is the sneaky one: storing "just a little" state in memory because it's fast and easy — a cache dictionary here, a logged-in-users set there. It works flawlessly on your laptop and in a one-server test, because there's only ever one memory. It fails the instant a second box exists and a request lands on the "wrong" one. The lab below makes the failure visible in the smallest possible code: watch the same operation return False on a stateful pair of servers and True once the state moves to a shared store.

stateless.py# STATEFUL: session lives in one server's memory -> must be sticky
class StatefulServer:
    def __init__(self): self.sessions = {}
    def login(self, user): self.sessions[user] = "token"
    def check(self, user): return user in self.sessions

s1, s2 = StatefulServer(), StatefulServer()
s1.login("ava")
print("stateful, wrong server:", s2.check("ava"))   # False! request hit s2

# STATELESS: session in a SHARED store -> any server works
shared = {}
class StatelessServer:
    def login(self, user): shared[user] = "token"
    def check(self, user): return user in shared

a, b = StatelessServer(), StatelessServer()
a.login("ava")
print("stateless, any server:", b.check("ava"))     # True

stateful, wrong server: False
stateless, any server: True

3 · Replication & eventual consistency intermediate

You've felt this one as a user, even if you didn't know its name. You change your profile photo, hit save, see the confirmation — then open the app on your phone and the old photo is still there for a few seconds before it updates. Or you post a comment, refresh, and it's briefly gone. Nothing is broken. You've just witnessed replication lag: your write landed on one machine, but the copy you happened to read from hadn't caught up yet. This section is about why that gap exists and why, most of the time, it's a feature and not a bug.

First, why keep copies at all? Two reasons, both about survival and speed. Survival: if all your data lives on one machine and that machine dies, so does your company — replication keeps standby copies so a failure is a shrug, not a catastrophe. Speed: reads usually vastly outnumber writes (think how many times a tweet is read versus written), so you let a fleet of read-only copies (replicas) answer all those reads while one machine (the primary) handles writes. One writer, many readers — that's the classic primary-replica shape.

The analogy: think of the primary as the original whiteboard in a meeting room and replicas as photos of it on people's phones. When someone updates the whiteboard, the phone photos don't change until each person re-snaps it. There's always a moment where the whiteboard says one thing and a phone still shows the old version. That moment is the lag — and the promise that every phone will eventually match the whiteboard, just not instantly, is exactly what eventual consistency means. "Eventual" is a real, honest word here: correct soon, not correct now.

The common mistake — and it's a serious one — is assuming a read right after a write always returns the fresh value. On a single machine it does; across replicas it often doesn't, and code that assumes "write then immediately read the same thing back" will flake intermittently in ways that are maddening to debug (works on your machine, fails one time in fifty in production). The engineering judgment is knowing where the lag is acceptable: a stale like-count or view-count for half a second is fine, nobody's harmed. A stale bank balance, or an inventory count that lets you oversell the last item, is not. The lab below strips this to its bones — a primary that takes a write and a replica that returns None until sync() runs — so you can see the stale-read window open and close with your own eyes.

Replication keeps copies on multiple machines: survive failures and serve reads from replicas. The catch is sync lag — a replica may briefly serve stale data (eventual consistency).

replication.pyclass Primary:
    def __init__(self): self.data = {}
    def write(self, k, v): self.data[k] = v

class Replica:
    def __init__(self, primary): self.primary = primary; self.data = {}
    def sync(self): self.data = dict(self.primary.data)   # periodic catch-up
    def read(self, k): return self.data.get(k)

primary = Primary(); replica = Replica(primary)
primary.write("x", 1)
print("before sync (stale):", replica.read("x"))   # None
replica.sync()
print("after sync:", replica.read("x"))             # 1

before sync (stale): None
after sync: 1

4 · The CAP theorem advanced

Picture the exact moment things go wrong. Two of your datacenters normally chatter constantly, keeping their data in step. Then a fiber line gets cut, or a switch fails, and they can't talk to each other — but both are still up, and users are still hitting both. A customer updates their data on the east coast. A moment later they read it from the west coast, which hasn't heard the update and can't hear it because the link is down. What should the west-coast side do: answer with the old value, or refuse to answer at all? There is no third option, and that forced dilemma is the CAP theorem.

Let's define the three letters plainly. Consistency (C) = every read sees the most recent write (no stale answers). Availability (A) = every request gets a non-error response (the system always answers). Partition-tolerance (P) = the system keeps working even when the network splits and machines can't reach each other. CAP's claim is that when a partition is happening, you can have C or A but not both at once.

Here's the piece almost everyone gets wrong, so read slowly. CAP is not "pick 2 of 3, like choosing toppings." P is not optional — networks genuinely fail, cables get cut, packets drop, so any real distributed system must tolerate partitions whether you like it or not. That collapses the "choose 2 of 3" myth into the only choice that actually exists: during a partition, do you sacrifice C or sacrifice A? A system that keeps consistency by refusing to answer is called CP (e.g. traditional SQL clusters, ZooKeeper, etcd — the sort of things that guard money and configuration). A system that keeps answering, accepting possibly-stale data, is AP (e.g. Cassandra, DynamoDB, DNS — the sort of things that must never go dark). And crucially — this is the honest, precise version — the trade only bites during a partition. When the network is healthy, a well-built system happily gives you both consistency and availability; CAP is a statement about the failure moment, not about everyday operation.

The common mistake is calling a system "a CA system" or claiming you "chose CP over AP" as a permanent, global personality trait. Reality is finer-grained: the choice is made per-operation, at the instant a partition occurs, and mature systems even let you pick per-request (a "read-your-writes" query can demand consistency while a background analytics read tolerates staleness). The tidy "CP vs AP" split is the right interview-level model — clear and correct as far as it goes — but know that PACELC refines it: even with no partition (Else), you still trade Latency against Consistency, because keeping copies perfectly in sync costs round-trips. The lab below makes the core dilemma tangible: the same partition, one node that refuses to stay correct and one that answers to stay up.

CAP: when a network partition splits your machines, you must choose Consistency (reject requests to stay correct) or Availability (answer, possibly stale). Partitions happen, so P isn't optional — the real choice is CP vs AP.

cap.pyclass Node:
    def __init__(self, mode): self.mode = mode; self.value = None; self.reachable = True
    def read(self):
        if not self.reachable and self.mode == "CP":
            return ("error", "unavailable to stay consistent")   # CP: refuse
        return ("ok", self.value)                                 # AP: answer (maybe stale)

cp = Node("CP"); cp.value = "v1"; cp.reachable = False   # partitioned off
ap = Node("AP"); ap.value = "v1"; ap.reachable = False
print("CP node:", cp.read())     # refuses
print("AP node:", ap.read())     # answers with possibly-stale v1

CP node: ('error', 'unavailable to stay consistent')
AP node: ('ok', 'v1')

5 · Expert — capacity estimation expert

Here's a scene that plays out in every system-design interview and every real design review. Someone asks, "so how many servers does this need?" and the room splits into two kinds of engineers. One kind freezes, because it feels like you'd need to build the thing and measure it. The other kind grabs the nearest whiteboard, writes down three numbers, multiplies and divides for ninety seconds, and says "about four app servers and two read replicas, call it six with headroom." That second engineer isn't smarter — they just know that capacity is arithmetic, not clairvoyance. This section teaches you to be that engineer.

The technique has a name — back-of-the-envelope estimation — and the whole idea is that being roughly right beats being precisely stuck. The analogy is planning a wedding: you don't email 200 guests to ask exactly how hungry they'll be. You estimate — "200 people, most eat one plate, some go back for seconds, so cater for ~250" — and you're fine. Servers are the same: you take a couple of known facts (how many users, how often they act), turn them into requests-per-second, and divide by what one machine can handle. The magic step most beginners skip is the peak multiplier: traffic is never flat, it clumps at lunch, at 8pm, on launch day, so you size for the busy hour, not the daily average, or you'll fall over precisely when everyone's watching.

The mental toolkit for this is a handful of latency numbers worth committing to memory — not to recite, but so your estimates land in the right order of magnitude. Knowing that a memory read is ~100 nanoseconds while a cross-continent round-trip is ~150 milliseconds — a gap of over a million times — instantly tells you why chatty cross-region calls kill performance and why caching in memory is transformative. You don't need these exact; you need to never be off by a factor of a thousand.

The common mistake is false precision — spending an hour deriving "7.3 servers" from shaky assumptions, when the honest answer is "single digits, let's say 8 with headroom." Interviewers and senior reviewers grade the method, not the digits: did you state your assumptions out loud, go from users → average QPS → peak QPS → divide by per-server capacity → add headroom? Say it as a narrative and your assumptions become correctable in real time. The lab below runs exactly that pipeline in code — a million users become a concrete, defensible fleet size — and the same shape of math (estimate the load, divide by per-unit capacity) will later size your databases, caches, and queues in SD7.

Design work needs rough numbers: 1 server or 1,000? Learn to estimate from a few facts. Latency figures every engineer should know, roughly:

capacity.pydef size_fleet(dau, req_per_user_per_day, per_server_qps, peak_factor=3):
    avg_qps = dau * req_per_user_per_day / 86_400
    peak_qps = avg_qps * peak_factor            # traffic is bursty
    servers = -(-int(peak_qps) // per_server_qps)   # ceil division
    return round(avg_qps), round(peak_qps), servers

avg, peak, servers = size_fleet(dau=1_000_000, req_per_user_per_day=20, per_server_qps=500)
print(f"avg {avg} qps, peak ~{peak} qps -> {servers} app servers (+ headroom)")

# read-heavy? size read replicas separately:
reads = peak * 0.9                               # 90% reads
replicas = -(-int(reads) // 800)                 # each replica ~800 read qps
print(f"peak reads ~{round(reads)} qps -> {replicas} read replicas")

avg 231 qps, peak ~694 qps -> 2 app servers (+ headroom)
peak reads ~625 qps -> 1 read replicas

def vertical(cores, per_core_rps=100, max_cores=32):
    return min(cores, max_cores) * per_core_rps      # capped by biggest box

def horizontal(nodes, per_node_rps=800):
    return nodes * per_node_rps                        # add boxes

print("vertical @64 cores:", vertical(64))   # 3200 (capped at 32 cores)
print("horizontal @5 nodes:", horizontal(5)) # 4000
# Beyond the biggest single box, only horizontal scaling adds capacity.

# Stateful: session lives on ONE node -> requests to another node fail
class StatefulNode:
    def __init__(self): self.sessions = {}
    def login(self, user): self.sessions[user] = "token"
    def check(self, user): return user in self.sessions

n1, n2 = StatefulNode(), StatefulNode()
n1.login("ada")
print(n1.check("ada"), n2.check("ada"))   # True False  <-- broken on n2!

# Stateless: session in a SHARED store; any node works
shared = {}
class StatelessNode:
    def login(self, user): shared[user] = "token"
    def check(self, user): return user in shared

s1, s2 = StatelessNode(), StatelessNode()
s1.login("ada")
print(s1.check("ada"), s2.check("ada"))   # True True

class Replicated:
    def __init__(self, n_replicas):
        self.primary = {}
        self.replicas = [{} for _ in range(n_replicas)]
        self.pending = []
    def write(self, k, v):
        self.primary[k] = v
        self.pending.append((k, v))          # not yet on replicas
    def replicate(self):
        for k, v in self.pending:
            for r in self.replicas: r[k] = v
        self.pending.clear()
    def read_replica(self, i, k):
        return self.replicas[i].get(k)

db = Replicated(2)
db.write("x", 1)
print(db.read_replica(0, "x"))   # None  <-- stale, replication hasn't run
db.replicate()
print(db.read_replica(0, "x"))   # 1     <-- eventually consistent

def handle_write(policy, partitioned):
    if partitioned:
        if policy == "CP":
            return "REJECT (stay consistent, sacrifice availability)"
        if policy == "AP":
            return "ACCEPT (stay available, risk divergence)"
    return "ACCEPT (no partition)"

for p in ("CP", "AP"):
    print(p, "->", handle_write(p, partitioned=True))
# CP -> REJECT (stay consistent, sacrifice availability)
# AP -> ACCEPT (stay available, risk divergence)

def estimate(daily_writes, bytes_each, retention_years):
    per_day_bytes = daily_writes * bytes_each
    total_bytes = per_day_bytes * 365 * retention_years
    avg_write_bps = per_day_bytes / 86400          # bytes per second
    GB = 1024 ** 3
    return {
        "per_day_GB": round(per_day_bytes / GB, 2),
        "total_TB": round(total_bytes / GB / 1024, 2),
        "avg_write_KBps": round(avg_write_bps / 1024, 1),
    }

print(estimate(daily_writes=10_000_000, bytes_each=500, retention_years=3))
# {'per_day_GB': 4.66, 'total_TB': 4.99, 'avg_write_KBps': 56.5}

def scaling_decision(read_rps, write_rps,
                     single_box_cap=1000, replica_read_cap=8000):
    plan = []
    if read_rps + write_rps <= single_box_cap:
        plan.append("single box")
    else:
        if read_rps > single_box_cap:
            plan.append("add read replicas" if read_rps <= replica_read_cap
                        else "add replicas + read cache")
        if write_rps > single_box_cap:
            plan.append("shard writes (SD7)")   # replicas don't scale writes
    return plan

print(scaling_decision(500, 300))       # ['single box']
print(scaling_decision(5000, 400))      # ['add read replicas']
print(scaling_decision(9000, 2000))     # ['add replicas + read cache', 'shard writes (SD7)']

Knowledge check check yourself