Computer organization

1 · The von Neumann architecture essential

Here's something that sounds impossible until you see how it's done. You download a program you've never run — a game, a compiler, an AI model — and your computer, which was built years before that program existed, just runs it. No rewiring, no new chip. The same machine that ran your spreadsheet this morning runs the new thing this afternoon. How can one fixed piece of hardware do genuinely new things it was never designed for? The answer is a single idea from 1945, and once you get it, the whole box stops being magic.

Think of a player piano versus a musician. An old music box is built to play one tune — the melody is baked into the metal drum, and to change the song you'd have to rebuild the machine. A human pianist is different: hand them any sheet of music and they play it, because the instructions (the notes) are separate from the performer (the hands). The von Neumann insight is to make the computer the pianist, not the music box: keep the instructions in changeable memory, right alongside the data they work on, and build one machine that reads whatever instructions it finds and does them. Swap the sheet music, get a new song, same pianist.

The common misconception is that a computer "knows" how to run your specific program — that somewhere inside is a Photoshop chip and a browser chip. It doesn't. There is one dumb loop that reads a number, treats it as a command, does that command, and reads the next number. Photoshop is just a very long, very clever list of those numbers sitting in memory. That's also why the same layout enables both the wonders and the horrors of computing: because code is just data, a program can write other programs (compilers, JITs), modify itself, or — if you're careless — be overwritten by data pretending to be code (that's the root of a whole class of security bugs). The blur between "code" and "data" is the feature and the footgun.

The von Neumann architecture (1945) is the blueprint for almost every computer: a single memory holds both instructions and data (the stored-program idea), a CPU reads and executes them, and I/O connects to the outside world. Because code is just data in memory, a program can be loaded, replaced, even generated at runtime — that is why software is soft.

2 · Inside the CPU essential

Picture the most focused person you've ever worked with — and also the most forgetful. That's the CPU. It can do exactly one small thing at a time with blinding speed, but it has almost no memory of its own; it has to be told, step by step, where to find each number and where to put each result. If you've ever watched a line cook at a tiny station — one cutting board, two hands, ingredients passed to them one at a time from a big fridge across the kitchen — you already understand the shape of a processor. The genius isn't in remembering; it's in doing one thing, fast, forever.

Here's the part that surprises people: the CPU does not reach into your gigabytes of RAM and compute there. It can only operate on a handful of tiny slots called registers — think three or four spaces on that cutting board. To add two numbers living in memory, the control unit must first carry them into registers, let the ALU add, then carry the answer back out. Every part in the table below exists to serve that one rhythm: fetch a number in, do arithmetic, put a number back. The ALU is the knife (it does the math and logic — literally the adder circuits you built in df7); the registers are the cutting board; the control unit is the head chef reading the order and telling everyone what to do next; the program counter is the finger on the recipe marking which step comes next; and the bus is the runner ferrying ingredients between the station and the fridge.

The mistake to avoid is imagining the CPU as one big brain that "understands" your program. It understands nothing — it decodes one instruction, does the mechanical thing that instruction names, and forgets it happened. All the intelligence lives in the ordering of instructions, not in the chip. Keep that division of labour in mind as you read each part below: nobody here is smart, but together, running billions of times a second, they run everything you've ever used.

Registers are to the ALU what your hands are to you: the only place work actually happens. Everything else is about getting data into a register, operating, and putting the result back.

3 · Fetch–decode–execute: simulate a tiny CPU advanced

The first time this clicked for me, it was a little disappointing — and then thrilling. I had always assumed a processor was doing something profound, some inscrutable dance too clever to follow. Then I saw the actual loop it runs, and it fit in about six lines. That was the deflating part. The thrilling part came right after: this tiny loop is all there is. Every app, every operating system, every neural network you'll ever run is this same tediously simple cycle, repeated a few billion times a second. There is no second, smarter mechanism hiding behind it. This is the whole engine.

The loop has four beats, and the everyday analogy is following a recipe with your finger on the page. Fetch: look at the step your finger is on. Decode: work out what it's asking ("add the eggs" — okay, that means this action). Execute: actually do it. Advance: move your finger to the next line. Then repeat. The "finger on the page" is the program counter (PC) — a single number holding which instruction comes next — and that humble number is secretly where all the power lives. Normally the finger just slides down one line at a time, which gives you a boring straight-line program. But what if a step could say "move your finger back up to line 2"? Now you've got a loop. And "if the bowl is empty, skip to line 9"? That's an if. Every loop and branch you've ever written is really just an instruction that writes a new value into the PC instead of letting it tick forward.

The subtlety that trips beginners is exactly that PC-advance step. It's easy to forget that after most instructions the PC quietly increments by one — and that jumps work precisely by overriding that increment. If you add a jump instruction but still let the PC increment afterward, your jump lands one step past where you meant, and the bug is maddening because the code "looks" right. Watch for that in the lab and the exercises below. For now, meet the model: we build a working (if deliberately tiny) CPU with 5 instructions and run a program that computes (3 + 4) × 2 and stores it, printing the state after every beat so you can watch the cycle turn.

tiny_cpu.py# Instruction set: (op, arg). Registers: ACC (accumulator) + a small RAM.
#   LOAD  v  -> ACC = v
#   ADD   v  -> ACC = ACC + v
#   MUL   v  -> ACC = ACC * v
#   STORE a  -> RAM[a] = ACC
#   HALT     -> stop
program = [
    ("LOAD", 3),
    ("ADD", 4),
    ("MUL", 2),
    ("STORE", 0),
    ("HALT", None),
]

def run(program):
    PC, ACC, RAM = 0, 0, [None] * 8       # program counter, accumulator, memory
    while True:
        op, arg = program[PC]             # FETCH
        # DECODE + EXECUTE
        if op == "LOAD":  ACC = arg
        elif op == "ADD": ACC += arg
        elif op == "MUL": ACC *= arg
        elif op == "STORE": RAM[arg] = ACC
        elif op == "HALT": break
        print(f"PC={PC} {op:<5} arg={arg}  -> ACC={ACC}")
        PC += 1                            # advance
    return RAM

ram = run(program)
print("RAM[0] =", ram[0])

PC=0 LOAD  arg=3  -> ACC=3
PC=1 ADD   arg=4  -> ACC=7
PC=2 MUL   arg=2  -> ACC=14
PC=3 STORE arg=0  -> ACC=14
RAM[0] = 14

4 · The memory hierarchy intermediate

Here's a question that exposes how most of us secretly picture memory. If registers are so blindingly fast, why doesn't the computer just make all its memory out of registers and be done with it? A machine with 16 GB of register-speed storage would be a monster. The answer is brutal and physical: you can't. Fast memory is expensive, hot, and takes up huge amounts of chip area, so you can only afford a tiny amount of it. Cheap memory is slow. There is no material on Earth that is simultaneously enormous, instant, and affordable — so every computer ever built quietly makes a compromise, and that compromise is the memory hierarchy.

The analogy that makes the numbers stick is your own desk. The papers in your hands (registers) you can use instantly. What's on the desk (L1 cache) takes a half-second reach. A desk drawer (L2) is a couple of seconds. The bookshelf across the room (L3) is a short walk. Going down the hall to the archive (main RAM) is a real trip. And fetching a box from a warehouse in another city (disk or network) is a different order of experience entirely — you plan your whole day around not having to do it. Crucially, these aren't gentle 20%-slower steps: each layer down is roughly ten to a hundred times slower than the one above. That's why the table shows latencies as orders of magnitude, not tidy numbers.

The trap here is treating "memory access" as one flat cost — as if reading any variable costs the same. It doesn't, and the gap is enormous: an L1 hit is around a nanosecond, a trip to RAM around a hundred. The whole game the hardware plays — and the reason the next two sections exist — is to keep the data you're about to need in the fast layers, so you rarely pay the down-the-hall or another-city price. The CPU checks the fast layers first and only falls through to slower ones on a miss. Read the numbers below as ratios you should feel in your gut, not guarantees: exact figures vary widely by chip, but "RAM is ~100× slower than L1, disk is astronomically slower than RAM" is stable enough to reason with — and, as section 6's latency-budget exercise shows, it's exactly how senior engineers locate a performance problem.

5 · Caching & locality of reference advanced

Two programs. Same answer. Same number of additions. One runs several times faster than the other. Nothing about the algorithm changed — not the Big-O, not the operation count — yet one is sluggish and one is snappy. When engineers first hit this it feels like a bug in reality: how can doing the identical work cost wildly different amounts? This section is the answer, and it's one of the most practical things in the whole chapter, because it turns "my code is mysteriously slow" into "my code is fighting the cache" — a problem you can actually fix.

Here's the intuition. Imagine a librarian fetching books for you from a back room. Fetching one book takes a while, so she's clever: every time you ask for a book, she brings back the whole shelf it was sitting on, betting you'll want its neighbours next. If your requests wander the shelf in order, she almost never has to go back — the next book is already on the desk. That bet is locality, and it comes in two flavours: temporal ("you'll probably want that same book again soon") and spatial ("you'll probably want the book right next to it"). A real cache does exactly this: it doesn't fetch one byte, it fetches a whole cache line (typically 64 bytes) at once. So touching data right next to what you just touched is nearly free — it's already on the desk — while jumping to a random far-off address forces a fresh trip to the back room at full RAM latency.

The mistake almost everyone makes is assuming the order in which you walk through data doesn't matter — that a loop is a loop. It matters enormously. Walking memory in the order it's physically laid out means every fetched line gets fully used before the next one is needed (you read all 16 books off the shelf she brought). Skipping around — grabbing one item, then leaping thousands of addresses away — means each expensive line delivers a single useful item before you abandon it and force another trip. That is the entire performance difference between the two programs I opened with, and there's a textbook way to see it happen, which is exactly what the demonstration below sets up.

The canonical demonstration: summing a 2-D array row-major (along rows, the way it's laid out in memory) touches consecutive addresses and rides the cache lines; column-major jumps by a whole row each step and thrashes the cache. The code below measures both on your machine:

locality.pyimport time

N = 2000
# a flat list acts as an N x N matrix, row-major: element (r, c) at r*N + c
matrix = [0] * (N * N)

def sum_row_major():
    s = 0
    for r in range(N):
        base = r * N
        for c in range(N):
            s += matrix[base + c]        # consecutive addresses (cache-friendly)
    return s

def sum_col_major():
    s = 0
    for c in range(N):
        for r in range(N):
            s += matrix[r * N + c]       # jumps N elements each step (cache-hostile)
    return s

for name, fn in (("row-major", sum_row_major), ("col-major", sum_col_major)):
    t0 = time.perf_counter()
    fn()
    print(f"{name}: {time.perf_counter() - t0:.3f} s")
# Row-major is typically faster. Exact numbers depend on your CPU,
# cache sizes, and Python build -- run it and compare the two.

6 · A taste of assembly professional

You write c = a + b. That's one line. It reads like arithmetic — three symbols and a plus sign, done. But the CPU cannot do that in one step; nothing in the machine understands the word "a" or the idea of a named variable. Somewhere between your tidy line and the silicon, that single expression gets translated into the several literal, mechanical moves the processor actually performs. Assembly is where you get to read that translation — it's the last human-readable layer before raw binary, a thin set of names sitting directly on top of the CPU's real instructions. You don't usually write it, but learning to read a few lines is like a doctor learning to read an X-ray: you finally see the bones under the skin.

The reason to care isn't nostalgia for the old days — it's that assembly makes the abstractions from this whole chapter concrete. Your single line of arithmetic doesn't map to one CPU action; it maps to the exact rhythm you've now seen three times: bring the operands into registers, let the ALU compute, carry the result back to memory. The variables a, b, c are just addresses; the CPU shuttles their contents through a register, adds, and shuttles the answer back. Seeing that spelled out is the moment the tiny CPU in section 3, the registers in section 2, and the memory in section 4 all snap together into one picture.

The misconception worth killing here is that "assembly is a scary, alien language you'll never touch." In practice it's the opposite: it's simpler than Python — just move, add, store, jump — merely more verbose, because it hides nothing. And it does show up in a real career: when you profile a hot loop and stare at the disassembly to see why the compiler didn't vectorize it, when a debugger drops you into instructions because the source is optimized away, or when you're squeezing latency out of an inference kernel. Below is roughly what c = a + b becomes on a register machine (x86-flavoured pseudocode). Read it slowly and notice: it is exactly our tiny CPU — move into a register, add, store back.

add.asmmov  eax, [a]     ; load variable a from memory into register eax
add  eax, [b]     ; eax = eax + b   (the ALU does the add)
mov  [c], eax     ; store the register result back into variable c

Compare to our toy CPU: mov eax,[a] is LOAD, add eax,[b] is ADD, mov [c],eax is STORE. Real ISAs (x86, ARM, RISC-V) have hundreds of instructions, but the shape — move to register, compute, store — is the same one you simulated.

Knowledge check check yourself

# PC=0 LOAD 5 -> ACC=5
# PC=1 ADD 10 -> ACC=15
# PC=2 ADD 10 -> ACC=25
# PC=3 HALT    -> stop
print('PC path: 0,1,2,3  final ACC = 25')

program = [('LOAD',20),('SUB',5),('SUB',3),('HALT',None)]
def run(prog):
    PC, ACC = 0, 0
    while True:
        op, arg = prog[PC]
        if   op=='LOAD': ACC = arg
        elif op=='ADD':  ACC += arg
        elif op=='SUB':  ACC -= arg
        elif op=='HALT': break
        PC += 1
    return ACC
print(run(program))   # 12

prog = [
    ('LOAD', 3),   # 0
    ('PRINT', None), # 1
    ('SUB', 1),    # 2
    ('JZ', 6),     # 3  if ACC==0 jump to HALT
    ('JMP', 1),    # 4  else loop back to PRINT
    ('NOP', None), # 5
    ('HALT', None),# 6
]
def run(p):
    PC, ACC = 0, 0
    while True:
        op, arg = p[PC]
        if   op=='LOAD':  ACC = arg
        elif op=='SUB':   ACC -= arg
        elif op=='PRINT': print('ACC =', ACC)
        elif op=='JZ':
            if ACC == 0: PC = arg; continue
        elif op=='JMP':   PC = arg; continue
        elif op=='HALT':  break
        PC += 1
run(prog)   # prints ACC = 3, then 2, then 1

import time
N = 2000
m = [0] * (N*N)
def row():
    s = 0
    for r in range(N):
        b = r*N
        for c in range(N): s += m[b+c]
    return s
def col():
    s = 0
    for c in range(N):
        for r in range(N): s += m[r*N+c]
    return s
row(); col()                       # warm up
t=time.perf_counter(); row(); tr=time.perf_counter()-t
t=time.perf_counter(); col(); tc=time.perf_counter()-t
print(f'row {tr:.3f}s  col {tc:.3f}s  ratio {tc/tr:.2f}x')
# absolute times are machine-dependent; expect ratio > 1 (row faster)

# SLOW: inner loop jumps a full row each step
def copy_slow(src, dst, N):
    for c in range(N):
        for r in range(N):
            dst[r*N + c] = src[r*N + c]

# FAST: inner loop walks consecutive addresses
def copy_fast(src, dst, N):
    for r in range(N):
        base = r*N
        for c in range(N):
            dst[base + c] = src[base + c]

N = 4
src = list(range(N*N)); dst = [0]*(N*N)
copy_fast(src, dst, N)
print(dst == src)   # True -- same result, cache-friendly order

# Approximate orders of magnitude (machine-dependent):
#   L1 hit   ~1 ns        RAM     ~100 ns
#   L3 hit   ~10 ns       SSD     ~10-100 us
#   disk seek ~ms         network ~ms+
#
# Reasoning:
# * A 2-3x CPU-bound slowdown with no I/O change -> likely locality/cache
#   (data grew past L2/L3, or an access pattern regressed). Fix: profile,
#   check working-set size and loop/access order.
# * A jump from ~us to ~ms in p99 -> crossed into disk or network territory
#   (cache miss on a hot dataset, an added remote call, a slow query).
#   Fix: trace the request, look for new I/O, check cache hit-rate.
print('classify by the MAGNITUDE of the regression, then trace that layer')