AI EngineeringZero to ProductionHome·About·Contact
System Design · Chapter SD9

Memory & concurrency

Two hard problems every system inherits from the OS: how memory is faked so each process thinks it owns the machine (virtual memory + paging), and what goes wrong when threads share it (races, deadlock). Runnable page-replacement and banker's-algorithm simulations turn the theory into fault counts and safe/unsafe verdicts.

⏱️ ~3.5 hours🧪 6 labs🎯 Beginner→Expert
🌱 Start here — from zeroMemory and concurrency, from scratch. Two illusions the OS maintains: every process believes it has the whole machine's memory to itself (virtual memory), and many threads seem to run at once. This chapter shows how paging fakes the first — and how sharing memory between threads breaks the second, producing race conditions and deadlock. We simulate the page-replacement algorithms and the banker's algorithm so you can measure faults and prove a state safe. The sims are models, not a real MMU or scheduler.

Learning objectives

  • Explain virtual memory, pages, and why page faults happen.
  • Simulate FIFO, LRU and optimal replacement and compare fault counts.
  • Diagnose a race condition and fix it with a mutex or a semaphore.
  • State the four Coffman conditions and run the banker's algorithm to check safety.
  • Reason about producer-consumer and bounded buffers.
▶ Runnable companionAll labs are standard-library Python. Concurrency labs use real threading, so the race demo is genuinely non-deterministic; the paging and banker's labs are deterministic models of the algorithms.
Cross-link — the Python angleThis chapter is the OS-level view of concurrency. The Python-specific consequences (the GIL, why threads don't parallelise CPU work in CPython, and when to reach for multiprocessing or async) are covered in A2 · Memory & concurrency/GIL. Read them together.

1 · Virtual memory & paging

Each process is given a virtual address space: a private, contiguous-looking range of addresses starting at zero. The OS and the CPU's MMU (memory management unit) translate those virtual addresses to real physical RAM addresses. Memory is handled in fixed-size chunks called pages (physical slots are frames). The mapping lives in a page table. When a program touches a page that isn't in RAM, the CPU raises a page fault; the OS fetches the page from disk into a free frame and resumes the program.

Virtual addr Page table Frame in RAM
Why virtual memory is worth itIt gives isolation (process A cannot read process B's memory — the SD8 protection theme again), lets programs use more memory than physically exists (paging out to disk), and simplifies programming (every process sees the same clean address layout).
Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.
Step 1 · A page table + fault, modeled
paging.py# MODEL: translate virtual pages to physical frames; a miss is a page fault.
class MMU:
    def __init__(self, num_frames):
        self.table = {}                 # virtual page -> frame
        self.free = list(range(num_frames))
        self.faults = 0
    def access(self, page):
        if page in self.table:
            return ("hit", self.table[page])
        self.faults += 1                # page fault: not resident
        if self.free:
            frame = self.free.pop(0)
            self.table[page] = frame
            return ("fault->loaded", frame)
        return ("fault->no-frame", None)  # eviction needed (see Step 2)

mmu = MMU(num_frames=2)
for pg in [1, 1, 2, 3]:
    print(pg, mmu.access(pg))
print("total faults:", mmu.faults)
1 ('fault->loaded', 0)
1 ('hit', 0)
2 ('fault->loaded', 1)
3 ('fault->no-frame', None)
total faults: 3

2 · Page replacement — FIFO, LRU, optimal

When RAM is full and a new page must load, the OS must evict one. Which? Three reference policies: FIFO (evict the oldest-loaded page), LRU (evict the least-recently-used page — the SD5 cache idea, now for RAM), and optimal (evict the page used furthest in the future). Optimal is unimplementable in reality (it needs the future) but is the theoretical floor every real policy is measured against.

Step 2 · A page-replacement simulator (model)
page_replace.py# MODEL: count page faults for a reference string under three policies.
def simulate(refs, frames, policy):
    mem, faults = [], 0
    for i, page in enumerate(refs):
        if page in mem:
            if policy == "LRU":         # mark as most-recently used
                mem.remove(page); mem.append(page)
            continue
        faults += 1
        if len(mem) < frames:
            mem.append(page); continue
        if policy == "FIFO":
            victim = mem[0]
        elif policy == "LRU":
            victim = mem[0]             # front = least recently used
        else:  # OPTIMAL: page used furthest in the future (or never)
            future = refs[i + 1:]
            victim = max(mem, key=lambda p: future.index(p)
                         if p in future else float("inf"))
        mem.remove(victim); mem.append(page)
    return faults

refs = [7, 0, 1, 2, 0, 3, 0, 4, 2, 3, 0, 3, 2]
for pol in ("FIFO", "LRU", "OPTIMAL"):
    print(f"{pol:8s} faults={simulate(refs, 3, pol)}")
FIFO     faults=10
LRU      faults=9
OPTIMAL  faults=7
Bélády's anomaly — more frames, more faultsFIFO has a famous pathology: adding more frames can sometimes increase the fault count. LRU and optimal are "stack algorithms" and never suffer this. It is a good reminder that intuitive policies can behave non-monotonically.
PolicyEvictsNeedsReality
FIFOoldest loadedload ordercheap; can hit Bélády's anomaly
LRUleast recently usedaccess historygreat in practice; approximated in HW
Optimalused furthest aheadthe futureunimplementable; a lower-bound yardstick

3 · The concurrency problem & race conditions

When two threads share memory and at least one writes, the result can depend on the timing of their interleaving — a race condition. The classic example is counter += 1, which is really three steps (read, add, write). If two threads interleave those steps, an update can be lost. The buggy region is the critical section: code that must not run concurrently.

Step 3 · A real race (non-deterministic)
race.py# REAL threads: this genuinely races. counter += 1 is read-modify-write,
# so concurrent increments lose updates. Output varies per run.
import threading

counter = 0
def bump():
    global counter
    for _ in range(100_000):
        counter += 1            # NOT atomic: read, +1, write

threads = [threading.Thread(target=bump) for _ in range(4)]
for t in threads: t.start()
for t in threads: t.join()
print("counter:", counter, "(expected 400000)")
counter: 400000 (expected 400000)   # or LESS, and it varies per run
Why you might see 400000 anyway (the GIL)In CPython the GIL can make short increments look atomic sometimes, so you may see the correct total on some runs and a smaller number on others — which is exactly what makes races so dangerous: they hide until they don't. See A2 for why the GIL both mitigates and complicates this.

4 · Mutexes & semaphores

A mutex (mutual-exclusion lock) lets exactly one thread into a critical section at a time; others wait. A semaphore generalises it to a counter that admits up to N threads — useful for limiting concurrency (e.g. "at most 3 downloads at once"). A mutex is a semaphore with N=1.

Step 4 · Fixing the race with a mutex
mutex.py# REAL threads: the lock serialises the critical section, so no update is lost.
import threading

counter = 0
lock = threading.Lock()
def bump():
    global counter
    for _ in range(100_000):
        with lock:              # only one thread in here at a time
            counter += 1

threads = [threading.Thread(target=bump) for _ in range(4)]
for t in threads: t.start()
for t in threads: t.join()
print("counter:", counter)     # always 400000
counter: 400000
Semaphore = a counting mutexUse threading.Semaphore(3) to cap concurrency at three. Acquire before the guarded work, release after (a with block does both). The mutex above is just the N=1 case.

5 · Deadlock — the four conditions & the banker's algorithm

A deadlock is a cycle of threads each holding a resource the next one needs, so none can proceed. Coffman showed four conditions must all hold for deadlock to be possible: mutual exclusion, hold-and-wait, no preemption, and circular wait. Break any one and deadlock becomes impossible. The banker's algorithm avoids deadlock by only granting a request if the resulting state is still safe — i.e. some ordering exists in which every process can finish.

Coffman conditionMeaningOne way to break it
Mutual exclusiona resource is held exclusivelymake resources shareable where possible
Hold and waithold one, wait for anotherrequest all resources up front
No preemptioncan't force a releaseallow the OS to reclaim resources
Circular waita cycle of waitsimpose a global lock ordering
Step 5 · Banker's algorithm — safety check (model)
bankers.py# MODEL of the banker's algorithm: is the current allocation state SAFE?
# Safe = there exists an order in which all processes can finish.
def is_safe(available, maximum, allocation):
    n = len(maximum)                       # processes
    need = [[maximum[i][j] - allocation[i][j] for j in range(len(available))]
            for i in range(n)]
    work = list(available)
    finish = [False] * n
    order = []
    made_progress = True
    while made_progress:
        made_progress = False
        for i in range(n):
            if not finish[i] and all(need[i][j] <= work[j]
                                     for j in range(len(work))):
                for j in range(len(work)):   # this process can finish; it frees its
                    work[j] += allocation[i][j]  # resources back to the pool
                finish[i] = True
                order.append(i)
                made_progress = True
    return all(finish), order

# 3 resource types, 5 processes (classic textbook state)
available  = [3, 3, 2]
maximum    = [[7,5,3],[3,2,2],[9,0,2],[2,2,2],[4,3,3]]
allocation = [[0,1,0],[2,0,0],[3,0,2],[2,1,1],[0,0,2]]
safe, order = is_safe(available, maximum, allocation)
print("safe?", safe)
print("safe sequence:", order)
safe? True
safe sequence: [1, 3, 4, 0, 2]
How to read the safe sequenceThe algorithm repeatedly finds a process whose remaining need fits in the free pool, "runs" it, and returns its held resources to the pool — enabling the next. If every process can be finished this way, the state is safe. Grant a request only if the hypothetical post-grant state is still safe.

6 · Producer–consumer & bounded buffers

A producer thread makes items; a consumer thread uses them; a bounded buffer sits between. The synchronisation problem: the producer must block when the buffer is full, the consumer must block when it is empty, and the buffer itself must not be corrupted by concurrent access. Python's queue.Queue solves all three with an internal lock and condition variables — it is the canonical safe bounded buffer.

Step 6 · Producer–consumer with a bounded queue
producer_consumer.py# REAL threads: Queue(maxsize=...) blocks the producer when full and the
# consumer when empty, and is internally thread-safe.
import threading, queue

buf = queue.Queue(maxsize=5)
DONE = object()

def producer():
    for i in range(10):
        buf.put(i)              # blocks if buffer is full
    buf.put(DONE)

def consumer(out):
    while True:
        item = buf.get()        # blocks if buffer is empty
        if item is DONE:
            break
        out.append(item)

out = []
p = threading.Thread(target=producer)
c = threading.Thread(target=consumer, args=(out,))
p.start(); c.start(); p.join(); c.join()
print("consumed:", out)
consumed: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
This is the pattern behind every work queueThe SD7 message queue, a thread pool, and a Kafka consumer group are all producer–consumer at heart: decouple the rate of production from the rate of consumption with a buffer, and use a lock/condition (or the broker) to keep it correct.

✓ Checkpoint — you can move on when you can…

  • Explain what a page fault is and trace an address through the page table.
  • Run the page-replacement sim and say why optimal beats LRU beats FIFO here.
  • Show a race condition and fix it with a mutex.
  • State the four Coffman conditions and run the banker's algorithm.
  • Implement producer–consumer with a bounded, thread-safe buffer.

Knowledge check

check yourself
✓ Knowledge check

Optimal page replacement produces the fewest faults yet no real OS uses it. Why not — and what is it actually useful for?

Show answer
Optimal evicts the page used furthest in the future, which requires knowing the future reference string — impossible at runtime. Its value is as a lower bound: it tells you the best any policy could achieve on a workload, so you can judge how close LRU or FIFO gets.
✓ Knowledge check

A deadlock needs all four Coffman conditions to hold. A team fixes a deadlock by making every thread acquire locks in a fixed global order (always lock A before B). Which condition does that break, and why does breaking one suffice?

Show answer
It breaks circular wait: if every thread acquires locks in the same total order, no cycle of "A waits for B waits for A" can form. Because all four conditions are necessary for deadlock, eliminating any single one makes deadlock impossible — lock ordering is the cheapest of the four to enforce in practice.

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · Count page faults for a reference stringBeginner

Context: Page-fault counting is the fundamental measurement of a memory system; the reference string is the workload.

Your task: Given a reference string and a frame count, count the page faults under FIFO — the simplest replacement policy.

Requirements:

  • A page already resident is a hit (no fault)
  • A miss is a fault; if a free frame exists, just load it
  • If full, evict the oldest-loaded page (FIFO)
  • Return the total fault count for the string

💡 Hint: Model memory as a list; evict index 0, append the new page — that is FIFO order.

Show solution

FIFO is a queue of resident pages — evict the front, append the newcomer:

def fifo_faults(refs, frames):
    mem, faults = [], 0
    for page in refs:
        if page in mem:
            continue                 # hit
        faults += 1                  # fault
        if len(mem) < frames:
            mem.append(page)
        else:
            mem.pop(0)               # evict oldest-loaded
            mem.append(page)
    return faults

print(fifo_faults([7,0,1,2,0,3,0,4,2,3,0,3,2], 3))   # 10

Ten faults on this classic string with three frames. FIFO ignores how often a page is used, which is why LRU usually does better.

Exercise 2 · Compare FIFO, LRU and optimalIntermediate

Context: The three reference policies bracket what is achievable; comparing them on one string shows the value of recency and the unreachable optimum.

Your task: Extend your simulator to also do LRU and optimal, and print all three fault counts for the same reference string and frame count.

Requirements:

  • LRU: on a hit, move the page to the most-recently-used position
  • LRU eviction: remove the least-recently-used (front) page
  • Optimal: evict the page whose next use is furthest in the future (or never)
  • Print FIFO, LRU and optimal fault counts side by side

💡 Hint: For optimal, look at refs[i+1:] and pick the resident page with the largest next-use index.

Show solution

One driver, three policies distinguished only by how the victim is chosen:

def simulate(refs, frames, policy):
    mem, faults = [], 0
    for i, page in enumerate(refs):
        if page in mem:
            if policy == "LRU":
                mem.remove(page); mem.append(page)
            continue
        faults += 1
        if len(mem) < frames:
            mem.append(page); continue
        if policy in ("FIFO", "LRU"):
            victim = mem[0]          # oldest / least-recently-used at front
        else:                        # OPTIMAL
            future = refs[i+1:]
            victim = max(mem, key=lambda p: future.index(p)
                         if p in future else float("inf"))
        mem.remove(victim); mem.append(page)
    return faults

refs = [7,0,1,2,0,3,0,4,2,3,0,3,2]
for pol in ("FIFO", "LRU", "OPTIMAL"):
    print(pol, simulate(refs, 3, pol))

Prints FIFO 10, LRU 9, OPTIMAL 7. Recency (LRU) beats load-order (FIFO); the unreachable optimum (7) is the yardstick for how good either really is.

Exercise 3 · Demonstrate a race and fix itAdvanced

Context: A lost-update race is the canonical concurrency bug; seeing it and then fixing it with a lock is the core lesson of shared-memory concurrency.

Your task: Write a program where several threads increment a shared counter without a lock (showing it can under-count), then add a mutex and show the count is always correct.

Requirements:

  • Run enough iterations that the race can manifest
  • Show the unlocked total can be less than the expected total
  • Add a mutex around the critical section
  • Show the locked total is always exactly correct
  • Explain why counter += 1 is not atomic

💡 Hint: The critical section is exactly the read-modify-write of the counter; the lock must span all three sub-steps.

Show solution

The same workload with and without a mutex:

import threading

def run(use_lock):
    counter = 0
    lock = threading.Lock()
    def bump():
        nonlocal counter
        for _ in range(200_000):
            if use_lock:
                with lock:
                    counter += 1
            else:
                counter += 1
    ts = [threading.Thread(target=bump) for _ in range(4)]
    for t in ts: t.start()
    for t in ts: t.join()
    return counter

expected = 4 * 200_000
print("no lock:", run(False), "expected", expected)   # may be < expected
print("with lock:", run(True), "expected", expected)  # always == expected

counter += 1 compiles to read, add, write; without the lock two threads can read the same value and one increment is lost, so the unlocked run can fall short (it varies per run, and the GIL can sometimes mask it). The mutex makes the three sub-steps indivisible, so the locked run is always exact.

Exercise 4 · Banker's algorithm: grant or deny a requestExpert

Context: The banker's algorithm is deadlock avoidance in practice: it only grants a resource request if the resulting state can still be completed by some ordering.

Your task: Implement a request check that tentatively grants a resource request, runs the safety check, and grants only if the resulting state is safe — otherwise rolls back and denies.

Requirements:

  • Reject immediately if the request exceeds the process's declared need
  • Reject if the request exceeds currently available resources
  • Tentatively apply the grant, then run the safety check
  • Grant if safe; otherwise roll back and deny
  • Demonstrate one granted and one denied request

💡 Hint: Deep-copy (or carefully restore) the state before the tentative grant so a denied request leaves everything unchanged.

Show solution

Request handling wraps the safety check with two fast rejections and a tentative grant:

import copy

def is_safe(available, maximum, allocation):
    n, m = len(maximum), len(available)
    need = [[maximum[i][j]-allocation[i][j] for j in range(m)] for i in range(n)]
    work, finish = list(available), [False]*n
    progress = True
    while progress:
        progress = False
        for i in range(n):
            if not finish[i] and all(need[i][j] <= work[j] for j in range(m)):
                for j in range(m): work[j] += allocation[i][j]
                finish[i] = True; progress = True
    return all(finish)

def request(pid, req, available, maximum, allocation):
    m = len(available)
    need = [maximum[pid][j]-allocation[pid][j] for j in range(m)]
    if any(req[j] > need[j] for j in range(m)):
        return "DENY: exceeds declared need"
    if any(req[j] > available[j] for j in range(m)):
        return "DENY: not enough available now"
    av2 = [available[j]-req[j] for j in range(m)]
    al2 = copy.deepcopy(allocation)
    for j in range(m): al2[pid][j] += req[j]
    return "GRANT" if is_safe(av2, maximum, al2) else "DENY: would be unsafe"

available  = [3,3,2]
maximum    = [[7,5,3],[3,2,2],[9,0,2],[2,2,2],[4,3,3]]
allocation = [[0,1,0],[2,0,0],[3,0,2],[2,1,1],[0,0,2]]
print("P1 asks [1,0,2]:", request(1, [1,0,2], available, maximum, allocation))
print("P0 asks [0,2,2]:", request(0, [0,2,2], available, maximum, allocation))

Prints P1 asks [1,0,2]: GRANT (the post-grant state is still safe) and P0 asks [0,2,2]: DENY: would be unsafe. The tentative state is built on copies, so a denied request leaves the real allocation untouched — that is deadlock avoidance: never enter a state you cannot guarantee to exit.

Exercise 5 · A concurrency limiter with a semaphoreProfessional

Context: Bounding concurrency is a daily production need — connection pools, download caps, rate control. A counting semaphore is the primitive.

Your task: Build a limiter that lets at most N worker threads into a guarded region at once, and record the maximum observed concurrency to prove the cap holds.

Requirements:

  • Use a semaphore initialised to N
  • Track a live in-region counter and the maximum it reaches
  • Run many more than N workers so the cap is actually exercised
  • Assert the observed maximum never exceeds N
  • Explain how a mutex is the N=1 special case

💡 Hint: Guard the in-region counter with its own small lock so the max is recorded correctly under contention.

Show solution

A semaphore caps how many of the 50 workers are inside the region at once:

import threading, time, random

N = 3
sem = threading.Semaphore(N)
live = 0
peak = 0
meter = threading.Lock()

def worker():
    global live, peak
    with sem:                       # at most N threads past this point
        with meter:
            live += 1; peak = max(peak, live)
        time.sleep(random.uniform(0.005, 0.02))
        with meter:
            live -= 1

ts = [threading.Thread(target=worker) for _ in range(50)]
for t in ts: t.start()
for t in ts: t.join()
print("peak concurrency:", peak, "cap:", N)
assert peak <= N

The peak never exceeds N because the semaphore admits at most N holders; a separate small lock protects the meter so the peak is recorded correctly under contention. A mutex is exactly this with N = 1.

Exercise 6 · A thread-safe bounded work queue with shutdownIndustry scenario

Context: Real services run a producer feeding a pool of consumers through a bounded queue, and must shut down cleanly without losing or double-processing work.

Your task: Build a bounded producer/multi-consumer pipeline that processes every item exactly once and shuts every consumer down cleanly after the work is exhausted.

Requirements:

  • Use a bounded thread-safe queue so a fast producer applies backpressure
  • Fan out to several consumer threads
  • Ensure every item is processed exactly once (no loss, no duplication)
  • Use a sentinel (one per consumer) or task-done tracking to signal shutdown
  • Show the union of consumer outputs equals the produced set

💡 Hint: One sentinel per consumer is the simplest correct shutdown: each consumer that sees a sentinel exits, so N sentinels stop N consumers.

Show solution

A bounded queue, a pool of consumers, and one sentinel per consumer for a clean shutdown:

import threading, queue

NUM_CONSUMERS = 3
work = queue.Queue(maxsize=8)      # bounded -> backpressure on the producer
results = []
results_lock = threading.Lock()
SENTINEL = object()

def producer(n):
    for i in range(n):
        work.put(i)                # blocks when full
    for _ in range(NUM_CONSUMERS):
        work.put(SENTINEL)         # one poison pill per consumer

def consumer():
    while True:
        item = work.get()
        if item is SENTINEL:
            work.task_done(); break
        with results_lock:
            results.append(item * item)
        work.task_done()

N = 100
p = threading.Thread(target=producer, args=(N,))
cs = [threading.Thread(target=consumer) for _ in range(NUM_CONSUMERS)]
p.start()
for c in cs: c.start()
p.join()
for c in cs: c.join()
print("processed:", len(results), "of", N)
print("exactly once:", sorted(results) == [i*i for i in range(N)])

Prints processed: 100 of 100 and exactly once: True. The bounded queue applies backpressure so a fast producer cannot exhaust memory; the per-consumer sentinels guarantee every worker stops exactly once and no real item is mistaken for a shutdown signal. This is the shape of every production work-queue.

© 2026 studybydoing.in · AI Engineering: Zero to Production · All rights reserved. · About · Privacy Policy · Terms · Contact
Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.
© studybydoing.in