System Design · Chapter SD5

HLD: architecture & caching

High-level design from "what happens on a page load" to a working gateway+cache+load-balancer edge — the client→LB→app→cache/DB shape of every web system, built in runnable Python.

⏱️ ~3 hours🧪 5 labs🎯 Beginner→Expert

🌱 Start here — from zero High-level design, from scratch — no infrastructure background needed — we start with "what happens when you load a web page?"

High-level design (HLD) is drawing the big boxes of a system and how requests flow between them. Almost every web app has the same shape: clients → a load balancer → many identical app servers → a cache and database. This chapter builds that mental model and then implements the interesting pieces — load balancing and an LRU cache — in runnable Python.

The words you'll hear (in plain terms):

Term	What it actually means
client / server	the client (browser/app) asks; the server answers.
load balancer	spreads incoming requests across many servers.
app server	runs your code; you run many identical copies.
cache	fast storage of results so you don't recompute/re-fetch.
API gateway	a front door that does auth, rate limiting, and routing.

What you need before starting:

Basic Python; the agent/web chapters help but aren't required.
Nothing to install; the labs are plain Python.
Curiosity about how big sites actually work.

New to the topic? Read this box, then take the chapters in order — each section is tagged essential → expert so you always know the depth you're at.

Learning objectives

Describe the client → LB → app → cache/DB shape of every web system.
Implement load-balancing strategies in Python.
Explain what an API gateway centralizes.
Build an LRU cache and reason about cache placement + invalidation.

▶ Runnable companionEvery code block here is also saved under code/sd5-hld-architecture/ — pure Python / sqlite3, runs with no setup.

1 · The shape of a web system essential

Start with the picture. A request travels: client → load balancer → one of N identical app servers → cache (fast) or database (source of truth). HLD is choosing and connecting these boxes; the rest of the chapter implements the interesting ones.

🗺️ How to read this diagram

This is the single most important picture in the whole chapter: the shape that almost every website on the internet is built from. Read it left to right — that is the exact path one request takes, from your browser all the way to the data.

Clients (the leftmost box) are the browsers and phone apps that people use. They ask for things ("show me the home page"); they never hold the real data themselves.
The arrow between boxes is a request travelling over the network. Follow the arrows in order — each box hands the request to the next one.
The Load balancer is a traffic cop. Many identical copies of your program are running, and the load balancer spreads incoming requests across them so no single one gets swamped.
The App servers (N) box is your actual code, running as N identical copies (N just means "several"). They are marked stateless: they keep no memory of their own, so any copy can handle any request — which is what makes it safe to add more of them.
The Cache + DB box on the right is where the real data lives (state lives here). The database (DB) is the permanent source of truth; the cache is fast temporary storage that remembers recent answers so you don't have to ask the slow database every time.

In short: A page load is just this line of boxes, read left to right: your browser asks → the load balancer picks a free app server → that server answers, using the fast cache when it can and the database when it must. Every later section just zooms in on one of these boxes.

Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.

Step 1 · A request, modeled in Python

request_flow.py# A tiny model of the flow: request -> pick a server -> serve.
class Server:
    def __init__(self, name): self.name = name
    def handle(self, req): return f"{self.name} handled {req}"

servers = [Server("app1"), Server("app2"), Server("app3")]
# (the "load balancer" that picks a server is Step 2)
print(servers[0].handle("GET /home"))

app1 handled GET /home

▶ How this works

Before wiring anything fancy, this lab models the flow above as a few lines of Python so you can see it work. A class here is just a template for making objects; each Server object is one of those "app server" boxes from the diagram.

class Server defines what a server is. __init__ is the setup step that runs when you create one — it just remembers the server's name (like app1).
handle(self, req) is what a server does with a request: it returns a sentence saying which server handled which request. req is the request (e.g. "GET /home" means "fetch the home page").
servers = [Server("app1"), Server("app2"), Server("app3")] creates three identical servers and puts them in a list — the "N app servers" from the picture.
The last line calls servers[0].handle("GET /home") — it hands the request to the first server (index 0) by hand. Choosing which server automatically is the load balancer's job, coming up next.

What the output means: It prints app1 handled GET /home — the first server acknowledging it dealt with the request.

Try this: Change servers[0] to servers[2] and re-run: now app3 answers. Picking the index by hand every time is exactly the chore a load balancer removes.

2 · Load balancers — spread the work essential

A load balancer decides which server handles each request, so no server is overwhelmed and dead servers are skipped. Let's implement the common strategies.

Step 2 · Balancing strategies

load_balancer.pyimport itertools, random

class RoundRobin:
    def __init__(self, servers): self._it = itertools.cycle(servers)
    def pick(self): return next(self._it)

class LeastConnections:
    def __init__(self, servers): self.load = {s: 0 for s in servers}
    def pick(self):
        s = min(self.load, key=self.load.get); self.load[s] += 1; return s
    def release(self, s): self.load[s] -= 1

servers = ["s1", "s2", "s3"]
rr = RoundRobin(servers)
print("round-robin:", [rr.pick() for _ in range(5)])
lc = LeastConnections(servers)
picks = [lc.pick() for _ in range(3)]      # each new conn goes to the least-loaded
print("least-conn:", picks)

round-robin: ['s1', 's2', 's3', 's1', 's2']
least-conn: ['s1', 's2', 's3']

▶ How this works

This lab implements the load balancer box for real — two classic strategies for deciding which server gets the next request. Each is a small class with a pick() method that hands back the chosen server.

RoundRobin just takes turns: server 1, then 2, then 3, then back to 1, forever. itertools.cycle(servers) makes an endless loop over the list, and next(self._it) pulls the next one each time pick() is called. Simple and fair when all servers are equally powerful.
LeastConnections is smarter: it sends the request to whichever server is currently doing the least work. self.load is a dictionary counting how many active requests each server has.
In pick(), min(self.load, key=self.load.get) finds the server with the smallest count, then self.load[s] += 1 records that it just got one more job. release(s) subtracts one when a request finishes.
The bottom lines run each strategy a few times and print the picks so you can see the round-robin rotation versus the least-connections choices.

What the output means: round-robin: ['s1', 's2', 's3', 's1', 's2'] shows the plain rotation. least-conn: ['s1', 's2', 's3'] — because every server starts idle, it spreads the first three requests one each.

Try this: Call lc.release('s1') after a few picks, then lc.pick() again — s1 now looks least-loaded, so it gets chosen. That is how a real balancer favours servers that just freed up.

Strategy	Picks	Use when
Round-robin	next in rotation	servers roughly equal
Least-connections	fewest active requests	uneven request durations
Consistent hashing	same key → same server	sticky cache/session (SD7)

3 · API gateway — the front door intermediate

An API gateway sits in front of your services and handles cross-cutting concerns once — auth, rate limiting (your SD4 limiter), routing, logging — so each service stays focused on business logic.

🗺️ How to read this diagram

This diagram zooms into a new box that sits in front of your services: the API gateway, the system's single front door. Read it left to right again — every request must pass through the middle box before it can reach any service behind it.

The Client on the left is the same as before — but notice it has just one entry point to talk to. It doesn't need to know about the individual services hiding behind the gateway.
The arrow into the gateway is every incoming request funnelling through one guarded door instead of scattering to many services directly.
The API gateway (middle) does the repetitive jobs once, for everyone: auth (checking you're allowed in), limit (rate limiting — blocking someone hammering the system), and route (deciding which service should answer). Doing these in one place means each service doesn't have to repeat them.
The Service A / B / C box on the right are the focused services — each handles one job (orders, users, etc.) and can trust that anything reaching it has already been authenticated and routed correctly.

In short: Think of a building's front desk: one guard checks everyone's badge and points them to the right room, so each room's staff can just do their work. The gateway is that front desk for your services.

Step 3 · A minimal gateway (auth + route)

gateway.pyclass Gateway:
    def __init__(self, routes): self.routes = routes   # path -> handler
    def handle(self, path, token):
        if token != "secret":                          # cross-cutting: auth
            return (401, "unauthorized")
        handler = self.routes.get(path)
        if not handler:
            return (404, "not found")
        return (200, handler())                        # route to the service

gw = Gateway({"/orders": lambda: "order list", "/users": lambda: "user list"})
print(gw.handle("/orders", "secret"))
print(gw.handle("/orders", "wrong"))

(200, 'order list')
(401, 'unauthorized')

▶ How this works

Here is that gateway front door in code. A Gateway object is given a routing table — a dictionary mapping each URL path to the function that answers it — and its handle method enforces auth and routing before letting a request through.

__init__(self, routes) stores the routing table. routes maps a path like "/orders" to a handler — a small function that produces the answer for that path.
In handle(path, token), the first check is auth: if token != "secret" the caller didn't send the right password, so it returns (401, "unauthorized") and stops. (401 is the web's code for "not logged in".)
Next, self.routes.get(path) looks up a handler for the requested path. If there isn't one, it returns (404, "not found") — 404 means "no such page".
Only if both checks pass does it call the handler and return (200, handler()) — 200 means "OK, here's your answer". The tuple is (status code, body), exactly how real web responses are shaped.

What the output means: (200, 'order list') for the request with the right token, then (401, 'unauthorized') for the one with the wrong token — the gateway let the first through and blocked the second.

Try this: Call gw.handle("/missing", "secret") — a valid token but an unknown path, so you get (404, 'not found'). Notice auth is checked before routing: you can't probe paths without logging in first.

4 · Caching — the biggest performance lever advanced

A cache stores expensive results close to where they're needed. The core question is eviction: when full, what do you drop? LRU (Least Recently Used) is the workhorse.

Step 4 · An LRU cache from scratch

lru_cache.pyfrom collections import OrderedDict

class LRUCache:
    def __init__(self, capacity): self.cap = capacity; self._d = OrderedDict()
    def get(self, key):
        if key not in self._d: return None
        self._d.move_to_end(key); return self._d[key]
    def put(self, key, value):
        if key in self._d: self._d.move_to_end(key)
        self._d[key] = value
        if len(self._d) > self.cap: self._d.popitem(last=False)

c = LRUCache(2)
c.put("a", 1); c.put("b", 2)
c.get("a")                    # 'a' now most-recent
c.put("c", 3)                 # evicts 'b'
print(c.get("b"), c.get("a"), c.get("c"))   # None 1 3

None 1 3

▶ How this works

This builds a cache — the fast-storage box from the first diagram. A cache has limited room, so the key question is eviction: when it's full, what do you throw out? LRU (Least Recently Used) throws out whatever hasn't been touched for the longest, on the bet that recent things will be needed again.

OrderedDict is a dictionary that remembers the order items were added. We use that order as a "most-recently-used" ranking: freshest at the end, stalest at the front.
get(key) returns None if the key isn't stored. If it is, self._d.move_to_end(key) bumps it to the end (marking it as just used) before returning its value.
put(key, value) stores the value and moves it to the end. Then the crucial line: if len(self._d) > self.cap — if we've exceeded the capacity, popitem(last=False) removes the item at the front, i.e. the least recently used one.
The demo makes a cache of size 2, stores a and b, then reads a (making a the freshest). Adding c overflows the cache, so the stalest key — b — is evicted.

What the output means: None 1 3 — asking for b returns None (it was evicted), while a (1) and c (3) are still there. Reading a earlier is exactly what saved it.

Try this: Remove the c.get("a") line and re-run. Now a is the stalest, so adding c evicts a instead — and the output becomes 2 None 3. That one access changed who got dropped.

The hard part is invalidationCache layers: client → CDN → gateway → app memory → shared cache (Redis) → DB. Storing is easy; knowing when cached data is stale is the hard part. Common strategies: TTL (expire after N seconds) and write-through (update cache on write).

5 · Expert — assemble a mini architecture expert

Wire the pieces together the way a real request server would: the gateway rate-limits (SD4), checks the cache (Step 4), and on a miss picks a backend via the load balancer (Step 2).

Step 5 · Gateway + cache + LB, end to end (runs standalone)

edge.pyfrom collections import OrderedDict
import itertools

class LRUCache:
    def __init__(self, cap): self.cap=cap; self._d=OrderedDict()
    def get(self, k):
        if k not in self._d: return None
        self._d.move_to_end(k); return self._d[k]
    def put(self, k, v):
        self._d[k]=v; self._d.move_to_end(k)
        if len(self._d)>self.cap: self._d.popitem(last=False)

class Backend:
    def __init__(self, name): self.name=name
    def fetch(self, key): return f"{self.name}:value:{key}"

class Edge:
    """Gateway that caches reads and load-balances misses to backends."""
    def __init__(self, backends):
        self.cache = LRUCache(100)
        self._lb = itertools.cycle(backends)
    def get(self, key):
        hit = self.cache.get(key)
        if hit is not None:
            return ("cache", hit)
        server = next(self._lb)                 # round-robin on miss
        val = server.fetch(key)
        self.cache.put(key, val)
        return ("origin:" + server.name, val)

edge = Edge([Backend("b1"), Backend("b2")])
print(edge.get("x"))     # miss -> b1
print(edge.get("x"))     # hit  -> cache
print(edge.get("y"))     # miss -> b2

('origin:b1', 'b1:value:x')
('cache', 'b1:value:x')
('origin:b2', 'b2:value:y')

▶ How this works

The finale wires the whole edge together the way a real system does: one Edge object that checks the cache first, and only if the answer isn't cached does it load-balance the request to a backend and remember the result. It reuses the LRU cache and the round-robin idea from earlier.

Edge.__init__ sets up two things: an LRUCache(100) (room for 100 answers) and itertools.cycle(backends) — the round-robin rotation over the backend servers.
In get(key), it first calls self.cache.get(key). If that returns something (hit is not None), it's a cache hit: return ("cache", hit) immediately — fast, no backend touched.
If the cache had nothing (a miss), server = next(self._lb) picks the next backend in rotation, server.fetch(key) gets the real value, and self.cache.put(key, val) saves it so the next request for the same key is a hit.
It returns a label saying where the answer came from — "cache" or "origin:" + server.name — so you can watch the request's path.

What the output means: First get("x") is a miss → ('origin:b1', 'b1:value:x'). The second get("x") is now a hit → ('cache', 'b1:value:x') (no backend). get("y") is a new miss → ('origin:b2', ...), and note round-robin sent it to the next backend, b2.

Try this: Call edge.get("x") a third time — still a cache hit, still instant. This tiny loop (check cache → on miss, balance to a backend and cache the result) is the exact pattern a CDN and the capstone's URL shortener use to serve hot data fast.

You already use caching everywherePrompt caching (IC3) and RAG retrieval are caching too. The LRU + edge you just built is the same idea the capstone's URL shortener uses to serve hot redirects.

Exercise SD5.1 — Extend the edge

Context: A real edge is more than a cache — it enforces per-client limits and expires stale data. Extending your composed request path with a rate limiter and a TTL cache is how the pieces from earlier chapters come together.

Your task: Add your rate limiter to the edge so each client is limited before the cache check, give the LRU a TTL by storing (value, expires_at) and expiring stale entries, and print which path each request takes.

Requirements:

Rate-limit each client before the cache is consulted
Store each cache entry with an expiry timestamp and treat expired entries as misses
Preserve LRU eviction alongside the new TTL expiry
Classify every request as rate-limited, cache-hit, or origin and print that label
Demonstrate all three outcomes across a sequence of requests

💡 Hint: Check the rate limit first and short-circuit before touching the cache; on a cache read, compare now against the stored expires_at before treating the entry as a hit.

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · Round-robin load balancerBeginner

Context: A load balancer spreads work across a pool of identical servers, and round-robin is its simplest policy. It's the assumption behind the "many identical app servers" box in almost every high-level design.

Your task: Implement round-robin selection over N servers and show it cycling through them in order and wrapping around.

Requirements:

Hold the list of servers and a rotating index
Return the next server on each call and advance the index
Wrap around to the first server after the last (modulo the pool size)
Demonstrate a sequence of calls cycling through all servers and repeating
Understand the assumption: servers are identical and requests are similar in cost

💡 Hint: An index modulo the pool length, incremented each call, is the whole policy; it only makes sense when every server can serve every request equally.

Show solution

The simplest LB policy, in a few lines:

class RoundRobin:
    def __init__(self, servers):
        self.servers = servers
        self.i = 0
    def next(self):
        s = self.servers[self.i % len(self.servers)]
        self.i += 1
        return s

lb = RoundRobin(["app-1", "app-2", "app-3"])
print([lb.next() for _ in range(5)])
# ['app-1', 'app-2', 'app-3', 'app-1', 'app-2']

Round-robin assumes servers are identical and requests are similar — the assumption behind the "many identical app servers" box in the HLD shape.

Exercise 2 · Little's Law: how many servers?Intermediate

Context: Capacity questions are answerable on paper before you provision anything. Little's Law ties concurrency, arrival rate, and latency together, turning "how many servers?" into arithmetic.

Your task: Using Little's Law, compute the in-flight concurrency a service faces at a given arrival rate and latency, then divide by a per-server concurrency limit to get the server count.

Requirements:

Compute concurrency as arrival rate times latency (L = λ × W)
Divide concurrency by the per-server limit to size the fleet
Round the server count up (you can't run a fraction of a server)
Return both the concurrency and the server count
Note that real provisioning adds headroom for spikes and failures on top

💡 Hint: Little's Law is just L = λ × W; the server count is ceil(L / per_server), and the extra headroom is a judgement call layered on top.

Show solution

Little's Law: concurrency L = arrival rate λ × latency W:

def servers_needed(rps, latency_s, per_server_concurrency):
    concurrency = rps * latency_s          # Little's Law: L = λ * W
    import math
    return concurrency, math.ceil(concurrency / per_server_concurrency)

conc, n = servers_needed(rps=2000, latency_s=0.2, per_server_concurrency=100)
print(f"in-flight requests: {conc:.0f}")   # 400
print(f"servers needed: {n}")              # 4

2000 rps × 0.2 s = 400 concurrent requests; at 100 each you need 4 servers (plus headroom for spikes and failures).

Exercise 3 · LRU cache in front of a slow storeAdvanced

Context: Caching is the single biggest performance lever in most architectures: it converts repeated reads of a slow store into memory lookups. An LRU cache in front of a slow DB is the canonical shape.

Your task: Put an LRU cache in front of a mock slow database, count the actual database hits, and show the cache absorbing repeated reads.

Requirements:

Implement a small capacity-bounded LRU cache (get/put with eviction)
Route reads through the cache, falling back to the slow store on a miss
Populate the cache on a miss so the next identical read is served from memory
Count how many reads actually reached the database
Show repeated keys are served from cache, so DB hits are far fewer than total reads

💡 Hint: Only misses should increment the DB-hit counter; the demonstration is that six reads with repeats produce far fewer than six DB hits.

Show solution

The cache turns repeated reads into memory lookups:

from collections import OrderedDict

class Cache:
    def __init__(self, cap): self.cap, self.d = cap, OrderedDict()
    def get(self, k):
        if k in self.d: self.d.move_to_end(k); return self.d[k]
        return None
    def put(self, k, v):
        self.d[k] = v; self.d.move_to_end(k)
        if len(self.d) > self.cap: self.d.popitem(last=False)

db_hits = 0
def slow_db(k):
    global db_hits; db_hits += 1
    return f"row-{k}"

cache = Cache(2)
def read(k):
    v = cache.get(k)
    if v is None:
        v = slow_db(k); cache.put(k, v)
    return v

for k in [1, 1, 2, 1, 3, 1]:
    read(k)
print("db_hits:", db_hits)   # 3  (only 1,2,3 hit the DB; repeats served from cache)

Six reads, three DB hits — the repeated key 1 is served from memory. At scale this is the difference between a healthy DB and a melted one.

Exercise 4 · Cache hit-rate → effective latencyExpert

Context: Once you have a cache, its hit rate is the metric that governs user-visible latency. Effective latency is a weighted average of cache and miss latency, so you can compute exactly what hit rate an SLA demands.

Your task: Given a hit rate, a cache latency, and a miss latency, compute the effective average latency, then solve for the hit rate needed to meet a latency SLA.

Requirements:

Model effective latency as h × hit_ms + (1 - h) × miss_ms
Evaluate it across several hit rates to show the curve
Rearrange the formula to solve for the hit rate that meets a target latency
Report the required hit rate as a percentage
Draw the conclusion that pushing hit rate from 90% to 99% cuts effective latency several-fold

💡 Hint: It's a weighted average; to hit an SLA of T, set the expression equal to T and solve algebraically for h.

Show solution

Effective latency is a weighted average — cheap to model:

def eff_latency_ms(hit_rate, hit_ms=1.0, miss_ms=50.0):
    return hit_rate * hit_ms + (1 - hit_rate) * miss_ms

for h in (0.0, 0.5, 0.9, 0.99):
    print(f"hit={h:.0%} -> {eff_latency_ms(h):.2f} ms")
# hit=0%  -> 50.00 ms
# hit=90% -> 5.90 ms
# hit=99% -> 1.49 ms

# Hit rate needed for a 10ms SLA:
import math
# 10 = h*1 + (1-h)*50  ->  h = (50-10)/(50-1)
h_needed = (50 - 10) / (50 - 1)
print(f"need hit rate >= {h_needed:.1%}")   # ~81.6%

Going from 90% to 99% hit rate cuts effective latency 4x — showing why cache hit rate is the metric to optimize.

Exercise 5 · Assemble the edge (gateway + LB + cache)Professional

Context: A high-level design is only real once the boxes compose into a request path. Assembling gateway auth, a cache lookup, and round-robin dispatch shows each component doing exactly its job.

Your task: Compose a miniature request path — gateway auth check, then cache lookup, then round-robin to an app server, recording the result — modelled offline as composed functions.

Requirements:

Reject unauthenticated requests fast at the gateway (e.g. a 401)
Check the cache before dispatching and serve a hit without touching an app server
On a miss, pick an app server round-robin and record which one handled it
Populate the cache after a miss so the next identical request is a hit
Show the response reporting whether it was served by cache or by a specific server

💡 Hint: Each stage fails fast or short-circuits — auth before cache, cache before dispatch — so a repeated authenticated request never reaches an app server twice.

Show solution

The client→gateway→LB→app→cache shape, in miniature:

from collections import OrderedDict

class Cache:
    def __init__(self, cap): self.cap, self.d = cap, OrderedDict()
    def get(self, k):
        if k in self.d: self.d.move_to_end(k); return self.d[k]
    def put(self, k, v):
        self.d[k] = v
        if len(self.d) > self.cap: self.d.popitem(last=False)

servers = ["app-1", "app-2"]; rr = {"i": 0}
cache = Cache(100)

def gateway(request):
    if not request.get("token"):
        return {"status": 401}
    key = request["path"]
    hit = cache.get(key)
    if hit:
        return {"status": 200, "served_by": "cache", "body": hit}
    s = servers[rr["i"] % len(servers)]; rr["i"] += 1
    body = f"handled by {s}"
    cache.put(key, body)
    return {"status": 200, "served_by": s, "body": body}

print(gateway({"path": "/x"}))                 # {'status': 401}
print(gateway({"path": "/x", "token": "t"}))   # served_by app-1
print(gateway({"path": "/x", "token": "t"}))   # served_by cache

Auth fails fast at the gateway; the second identical request is served from cache without touching an app server — each box does exactly its job.

Exercise 6 · Capacity plan for a launchIndustry scenario

Context: Launch planning is where all the estimation tools converge into one provisioning decision: given peak load and a cache hit rate, do the app tier and the database both fit within their ceilings?

Your task: Produce a back-of-envelope capacity plan that, from peak rps, latency, cache hit rate, per-server limits, and a DB ceiling, outputs the servers needed, the DB rps after caching, and whether it fits.

Requirements:

Size app servers from concurrency (peak rps × latency) over the per-server limit
Compute post-cache DB load as peak_rps × (1 - hit_rate)
Compare DB load against the ceiling to decide whether it fits
Return concurrency, app-server count, DB rps after cache, and the fit verdict
Show that lowering the hit rate can flip the verdict from fits to doesn't-fit

💡 Hint: The cache is what shrinks DB load from the full arrival rate down to just the misses; re-run with a lower hit rate to surface the decision the plan is really making.

Show solution

One calculator that turns launch assumptions into a provisioning decision:

import math

def capacity_plan(peak_rps, latency_s, hit_rate, per_server, db_ceiling_rps):
    concurrency = peak_rps * latency_s
    app_servers = math.ceil(concurrency / per_server)
    db_rps = peak_rps * (1 - hit_rate)          # only misses hit the DB
    fits = db_rps <= db_ceiling_rps
    return {
        "concurrency": round(concurrency),
        "app_servers": app_servers,
        "db_rps_after_cache": round(db_rps),
        "db_fits": fits,
    }

plan = capacity_plan(peak_rps=5000, latency_s=0.15, hit_rate=0.9,
                     per_server=120, db_ceiling_rps=1000)
print(plan)
# {'concurrency': 750, 'app_servers': 7, 'db_rps_after_cache': 500, 'db_fits': True}

The cache drops DB load from 5000 to 500 rps — the reason it fits under the 1000 rps ceiling. Change hit_rate to 0.7 and it no longer fits, which is the decision the plan surfaces.

✓ Checkpoint — you can move on when you can…

Draw the client → LB → app → cache/DB architecture.
Implement round-robin and least-connections balancing.
Explain what an API gateway centralizes.
Build an LRU cache and wire a gateway+cache+LB edge.

Knowledge check check yourself

✓ Knowledge check

In the canonical client → load balancer → app servers → cache/DB shape, the app servers are described as "stateless." What does statelessness mean here, and why is it what makes it safe to add more app servers?

Show answer

Stateless means the app servers keep no memory of their own, so any identical copy can handle any request. Because no server holds unique per-request state, you can freely add more copies and the load balancer can route a request to any of them.

✓ Knowledge check

The lesson calls caching "the biggest performance lever" but warns that "the hard part is invalidation." What is the core eviction question an LRU cache answers, and what two strategies does the lesson name for knowing when cached data is stale?

Show answer

The eviction question is: when the cache is full, what do you drop? LRU (Least Recently Used) drops whatever hasn't been touched for the longest. For staleness, the lesson names TTL (expire after N seconds) and write-through (update the cache on write).

Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.

Learning objectives

1 · The shape of a web system essential

2 · Load balancers — spread the work essential

3 · API gateway — the front door intermediate

4 · Caching — the biggest performance lever advanced

5 · Expert — assemble a mini architecture expert

Exercise SD5.1 — Extend the edge

🪜 Practice ladder beginner → industry

✓ Checkpoint — you can move on when you can…

Knowledge check check yourself

More in System design