Transformers & the Path to Large Language Models
This is where the whole module lands. The transformer replaced recurrence with attention, unlocked parallel training at massive scale, and — through pretraining and fine-tuning — became the LLM you've used all course. Understand this chapter and claude-opus-4-8 stops being a black box.
Learning objectives
- Explain self-attention intuitively — every token looking at every other.
- Name the transformer's key parts: attention, multi-head, positional encoding, feed-forward, stacked layers.
- Explain why the transformer scaled where RNNs couldn't.
- Walk the path from architecture → pretraining → fine-tuning/alignment → the LLM.
- Connect transformer facts to LLM behavior you already know (context, tokens, hallucination).
Self-attention: the core idea advanced
The RNN's flaw (K4) was forcing information through a single hidden state, one step at a time. Self-attention throws that out: for each token, the model looks directly at every other token in the sequence and decides how much each one matters for understanding this one. No chain, no bottleneck — every position can reach every other in a single step.
This is the single most important picture in the whole chapter. It shows self-attention: how the model figures out what the word "it" refers to. The sentence is at the top; the coloured word it sits in the middle box, and arrows drop from it down to the other words it is "looking at".
- The box in the middle is the word the model is currently trying to understand:
it. Think of it as the word asking a question: "which earlier word do I stand for?" - Each arrow is
itlooking at another word and giving it a weight — a score for how relevant that word is. Every token looks at every other token; the arrows just show the ones that matter here. - The numbers in the lower boxes are those weights:
animal (0.7)is much bigger thanstreet (0.1)ortired (0.1). So the model decidesitmostly means animal — even though animal sits far back in the sentence. - The whole row of weights adds up to about 1 — the model is splitting 100% of its attention across the words, spending most of it on the one that matters. That is the "weighted look at the whole sentence" the caption calls self-attention.
In short: each word gets to look directly at every other word and decide how much each one matters. No word is too far away to reach — which is exactly the long-distance problem the older models could not solve.
The transformer's parts advanced
The transformer wraps self-attention with a few more components. You don't need to implement them, but naming them makes model behavior legible.
| Component | Job |
|---|---|
| Self-attention | Each token gathers relevant context from all others — the core |
| Multi-head attention | Run attention several times in parallel, each "head" learning a different kind of relationship (syntax, coreference, topic…) |
| Positional encoding | Attention alone is order-blind (like bag-of-words, K2!) — positional info is added so the model knows token order |
| Feed-forward layers | Per-token processing after attention mixes context — where much "knowledge" lives |
| Residuals + normalization | Engineering that makes very deep stacks trainable and stable |
| Stacked layers | Dozens+ of attention+FFN blocks — depth builds abstraction, shallow→deep meaning |
Why the transformer won advanced
The 2017 paper's title — "Attention Is All You Need" — was the claim that you could drop recurrence entirely. It won because it solved both RNN walls (K4) simultaneously.
This diagram contrasts the old way (left, red) with the new way (right, green) of reading a sentence, and shows why the transformer won. Read it as two side-by-side pictures, then the two summary lines at the bottom.
- Left, red — "RNN: sequential." The little boxes are words, joined by arrows in a chain. The arrows mean the model must process one word at a time, in order — it can't start word 3 until it finishes word 2. Slow, and it forgets things that happened far back in the chain.
- Right, green — "Transformer: parallel." The boxes have no chain between them: the model reads all the words at once. That is what "all positions at once" means, and it is why a transformer can use a big GPU (many calculations at the same time) instead of waiting in line.
- The first bottom line adds the other win: any token attends directly to any other — so a word at the end can look straight back at a word at the start, with no "long-range fade" (nothing gets forgotten along a chain).
- The last blue line is the payoff: with both old limits gone, you can train the model on trillions of words. Speed + memory-across-distance = the ability to train at massive scale.
In short: the transformer beat the older model by fixing two problems at once — it reads everything in parallel (fast) and lets any word see any other word directly (no forgetting). That combination is what made huge LLMs possible.
Lab K5.1 · Architecture → pretraining → the LLM expert
The transformer is an architecture. Turning it into an LLM takes two more stages. This pipeline is the answer to "where does an LLM come from?"
This is a pipeline: read it left to right, following the arrows. It answers "where does an LLM like Claude actually come from?" — the transformer you just learned about is only the first box; two more stages turn it into a helpful assistant.
- Box 1 — "Transformer architecture." The blueprint from this chapter — attention, stacked layers. On its own it knows nothing yet; it's an empty engine.
- Arrow, then Box 2 — "Pretrain: predict next token." The engine is fed enormous amounts of text and practises one game over and over: guess the next word. Doing this at huge scale is where its raw knowledge and fluency come from.
- Arrow, then Box 3 — "Fine-tune / align (RLHF)." The fluent-but-raw model is then trained on examples and human/AI feedback to follow instructions and be helpful, honest, and harmless — turning a text-predictor into an assistant.
- Arrow, then Box 4 — "LLM · Claude." The finished product you actually call. The bottom caption sums the flow up: capability comes from scale, then it's shaped to be helpful.
In short: an LLM is built in three steps — build the transformer, pretrain it to predict the next word on massive text, then align it to be a helpful assistant. Each arrow is one step forward in that recipe.
| Stage | What happens | Produces |
|---|---|---|
| Pretraining | Next-token prediction (K4) on trillions of tokens of text | A "base model" — fluent, knowledgeable, but not a helpful assistant |
| Instruction tuning | Fine-tune on examples of following instructions | A model that responds to requests, not just continues text |
| Alignment (RLHF/CAI) | Train on human/AI preferences toward helpful, honest, harmless | The assistant behavior you interact with (C1) |
Encoder, decoder, both expert
The original transformer had an encoder and a decoder (for translation, K4). Different LLM families use different slices — worth knowing because it explains what a model is good at.
| Type | Reads… | Best at | Example use |
|---|---|---|---|
| Encoder-only | The whole input at once (bidirectional) | Understanding: classification, embeddings | BERT-style; the embedding models behind RAG (K2, Ch 3) |
| Decoder-only | Left-to-right, predicting next token | Generation | The GPT/Claude family — chat, code, the LLMs you use |
| Encoder-decoder | Encode input, decode output | Transforming one sequence to another | Translation, some summarization models |
Transformer facts explain LLM behavior expert
The whole point of this module: knowing the machine explains the behavior you've worked with all course. Every LLM quirk traces to something here.
| LLM behavior you know | Transformer explanation |
|---|---|
| Finite, costly context window | Attention is O(n²) in length — more tokens cost quadratically (C1, Ch 3) |
| Billed per token; sees subwords | Input is tokenized to subwords (A5), then embedded (K2); compute scales with token count |
| Hallucination | It's a next-token predictor (K4) — it generates plausible continuations, not verified facts (why RAG & citations, Ch 3) |
| In-context learning (few-shot) | Attention lets examples in the prompt directly influence next-token prediction (Ch 2) |
| Sensitive to prompt wording/order | Every token attends to every other; phrasing changes the whole context representation |
| Helpful/honest/harmless persona | From alignment (RLHF/CAI), not the base model (C1) |
Common pitfalls expert
| Pitfall | Fix |
|---|---|
| Thinking the transformer "understands" like a person | It's a scaled next-token predictor; capability ≠ comprehension |
| Assuming attention knows word order | It's order-blind; positional encoding adds order |
| Believing bigger context is free | Attention is O(n²) — long context costs quadratically |
| Confusing pretraining with the assistant behavior | Alignment (RLHF/CAI) creates the helpful persona (C1) |
| Treating fluency as accuracy | Prediction ≠ truth — ground & verify (Ch 3, Ch 5) |
| Thinking architecture alone made LLMs | Scale (data + compute + params) is the decisive factor |
Exercises expert
Exercise K5.1 — Attention by intuition
Context: The Winograd-style sentence "The trophy didn't fit in the suitcase because it was too big" is the classic test of whether a model can resolve a pronoun using the whole sentence — the leap self-attention makes that n-grams cannot.
Your task: Explain what "it" refers to in that sentence, how the reference flips when "big" becomes "small", and why self-attention resolves it while an n-gram model (from K4) cannot.
Requirements:
- State the referent for both the "big" and the "small" version
- Explain how self-attention lets "it" weigh the whole sentence, adjective included
- Explain why a fixed-window n-gram has no way to reach back to the nouns
- Frame this as the concrete "context leap" the chapter is about
💡 Hint: Ask yourself what information the pronoun needs to see, and how far away in the sentence that information sits.
Show what to look for
"big" → it = trophy; "small" → it = suitcase. Attention lets "it" weigh the whole sentence including the adjective, resolving the reference. An n-gram sees only a tiny fixed window and has no way to reach back to "trophy"/"suitcase" or reason about the adjective. That's the context leap.
Exercise K5.2 — Explain the O(n²) cost
Context: Self-attention compares every token with every other token, so its cost grows with the square of the sequence length. That single fact explains a lot of the economics you have met elsewhere in the course.
Your task: In two sentences, explain to a peer why doubling the context window more than doubles attention cost, and connect it to pricier long-context API calls (C1) and to why RAG chunks documents (Ch 3).
Requirements:
- Name the quadratic (O(n²)) relationship between length and cost explicitly
- Show why 2× the tokens is ~4× the pairwise comparisons
- Tie it to why long-context calls cost more
- Tie it to why RAG splits documents into chunks instead of stuffing everything
💡 Hint: Count the pairwise interactions, not the tokens — that is where the squaring comes from.
Exercise K5.3 — Trace one behavior to the machine
Context: The whole point of this chapter is the "it's not magic" muscle: every LLM behaviour you have hit traces down to a mechanical fact about transformers.
Your task: Pick one LLM behaviour you have experienced in this course (hallucination, few-shot working, a prompt-order effect, or the token bill) and trace it all the way down to a transformer fact from this chapter.
Requirements:
- Name a specific behaviour you actually observed
- Connect it to a concrete mechanism — attention, softmax, context length, or position
- Make the causal chain explicit, from mechanism to observed behaviour
- Avoid hand-waving: the link should be mechanical, not "the model is smart"
💡 Hint: Prompt-order effects and the token bill both point straight at attention over a finite, quadratic context — good places to start.
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: Attention weights are a probability distribution over tokens, and that distribution comes from a softmax over raw compatibility scores. Getting the softmax numerically right is the first brick in the whole transformer.
Your task: Implement a numerically stable softmax(scores) and confirm the returned weights form a valid probability distribution.
Requirements:
- Subtract the max score before exponentiating so large inputs don't overflow
- Normalize by the sum of exponentials so the output sums to 1.0
- Work on a plain Python list of floats using only the stdlib
- Demonstrate on a few scores and print that the weights sum to ~1.0
- Handle negative and positive scores in the same call
💡 Hint: The stability trick is a single subtraction — shifting every score by the maximum leaves the softmax unchanged but keeps exp() in range.
Show solution
Subtract the max for stability, exponentiate, normalize. Runnable, stdlib:
import math
def softmax(scores):
m = max(scores)
exps = [math.exp(s - m) for s in scores] # subtract max -> no overflow
Z = sum(exps)
return [e / Z for e in exps]
w = softmax([2.0, 1.0, 0.1, -1.0])
print([round(x, 3) for x in w])
print("sum =", round(sum(w), 6)) # 1.0 -- a valid probability distributionSoftmax turns raw compatibility scores into a distribution — the attention weights that decide how much each token contributes.
Context: Scaled dot-product attention is the core operation of every transformer: softmax(q·kᵀ / √d) · V. Building it for one query over a handful of key/value pairs makes the mechanism concrete.
Your task: Implement single-query attention that scores the query against each key, scales, softmaxes, and returns the weighted sum of the values.
Requirements:
- Score each key by the dot product with the query
- Divide every score by
√dwhere d is the query dimension - Convert scores to weights with your stable softmax
- Return both the attention weights and the blended output vector
- Show that a query aligned with one key pulls the output toward that key's value
💡 Hint: The output is a convex combination of the value vectors — weight each value by its softmax score and sum component-wise.
Show solution
Score against each key, scale by √d, softmax, weight the values. Runnable:
import math
def softmax(scores):
m = max(scores); exps = [math.exp(s - m) for s in scores]; Z = sum(exps)
return [e / Z for e in exps]
def attention(q, keys, values):
d = len(q)
scores = [sum(qi * ki for qi, ki in zip(q, k)) / math.sqrt(d) for k in keys]
w = softmax(scores)
out = [0.0] * len(values[0])
for wi, v in zip(w, values):
for j in range(len(v)):
out[j] += wi * v[j]
return w, out
q = [1.0, 0.0]
keys = [[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]
values = [[10.0, 0.0], [0.0, 10.0], [5.0, 5.0]]
w, out = attention(q, keys, values)
print("weights:", [round(x, 3) for x in w])
print("output :", [round(x, 3) for x in out])
# the query aligns with key 0 (and 2), so the output leans toward their valuesThe scaling by √d keeps scores from growing with dimension (which would saturate softmax) — a small but load-bearing correctness detail.
Context: The √d divisor in attention looks arbitrary until you watch score magnitude grow with dimension. That growth pushes softmax into saturation, which is exactly why the transformer paper added the scaling.
Your task: Show numerically that unscaled dot-product scores grow with dimension and drive softmax toward a one-hot distribution, while dividing by √d keeps it usable.
Requirements:
- Generate random query/key vectors at several dimensions (e.g. 4, 64, 512)
- Report the magnitude of the largest unscaled score at each dimension
- Compare the peak softmax weight with and without the √d scaling
- Make it reproducible with a fixed random seed
- Draw the conclusion: score magnitude scales roughly with √d
💡 Hint: Watch the maximum softmax weight as d grows — unscaled it creeps toward 1.0 (saturation), scaled it stays moderate.
Show solution
Compare unscaled score magnitude at small vs large d, and softmax sharpness. Runnable:
import math, random
def softmax(scores):
m = max(scores); e = [math.exp(s - m) for s in scores]; Z = sum(e)
return [x / Z for x in e]
def demo(d, seed=0):
rng = random.Random(seed)
q = [rng.gauss(0, 1) for _ in range(d)]
ks = [[rng.gauss(0, 1) for _ in range(d)] for _ in range(3)]
raw = [sum(a*b for a, b in zip(q, k)) for k in ks]
scaled = [s / math.sqrt(d) for s in raw]
return max(abs(s) for s in raw), softmax(raw), softmax(scaled)
for d in (4, 64, 512):
mag, un, sc = demo(d)
print(f"d={d:>3}: max|score|={mag:7.2f} "
f"unscaled max-w={max(un):.3f} scaled max-w={max(sc):.3f}")
# score magnitude grows ~sqrt(d); unscaled softmax saturates toward 1.0Unscaled scores grow with √d, so softmax collapses onto one token and gradients vanish; dividing by √d keeps the distribution usable — exactly why the transformer paper added it.
Context: A transformer processes all tokens in parallel with no recurrence, so position must be injected explicitly. Sinusoidal encodings give the model a continuous, parameter-free notion of where each token sits.
Your task: Implement sinusoidal positional encoding for a position and model dimension, then show that nearby positions produce more similar vectors than distant ones.
Requirements:
- Even dimensions use sine, odd dimensions use cosine
- Frequencies are geometrically spaced across the dimensions
- The function returns a vector of length
d_model - Use cosine similarity to compare a few positions
- Demonstrate that adjacent positions are more similar than far-apart ones
💡 Hint: The frequency for dimension i follows 1 / 10000**(2*(i//2)/d_model) — geometric spacing is what makes the position signal smooth and extrapolatable.
Show solution
Even dims use sin, odd dims cos, at geometrically spaced frequencies. Runnable:
import math
def positional_encoding(pos, d_model):
pe = []
for i in range(d_model):
freq = 1.0 / (10000 ** (2 * (i // 2) / d_model))
pe.append(math.sin(pos * freq) if i % 2 == 0 else math.cos(pos * freq))
return pe
def cos_sim(a, b):
dot = sum(x*y for x, y in zip(a, b))
na = math.sqrt(sum(x*x for x in a)); nb = math.sqrt(sum(y*y for y in b))
return dot / (na * nb)
d = 16
p0, p1, p10 = (positional_encoding(p, d) for p in (0, 1, 10))
print(f"sim(pos0, pos1) = {cos_sim(p0, p1):.3f}")
print(f"sim(pos0, pos10) = {cos_sim(p0, p10):.3f}")
# adjacent positions are more similar than distant ones -- a smooth position signalSinusoidal encodings give the model a continuous, extrapolatable notion of position without any learned parameters or recurrence.
Context: The lesson maps a task to a transformer family: encoder-only for understanding, decoder-only for generation, encoder-decoder for transduction. Knowing which to reach for is a routine architecture decision.
Your task: Build a selector that, given a task name, returns the right architecture family and a representative model.
Requirements:
- Cover classification/embedding → encoder-only
- Cover generation/chat → decoder-only
- Cover translation/summarization → encoder-decoder
- Return both the family and an example model per task
- Fall back to a sensible default for an unknown task
💡 Hint: A dictionary keyed by task name is enough — the teaching point is the mapping itself, and that modern LLMs are overwhelmingly decoder-only.
Show solution
Route the task to the architecture family and a representative model. Runnable:
def architecture_for(task):
table = {
"classification": ("encoder-only", "BERT-style"),
"embedding/search": ("encoder-only", "sentence encoders"),
"text generation": ("decoder-only", "GPT/Llama-style"),
"chat": ("decoder-only", "GPT/Llama-style"),
"translation": ("encoder-decoder", "T5/BART-style"),
"summarization": ("encoder-decoder", "T5/BART-style"),
}
return table.get(task, ("decoder-only", "default for open-ended tasks"))
for t in ["classification", "text generation", "translation"]:
fam, model = architecture_for(t)
print(f"{t:>16} -> {fam:<16} ({model})")Encoders read (bidirectional, great for understanding); decoders write (causal, great for generation); encoder-decoder transduces one sequence into another. Modern LLMs are overwhelmingly decoder-only.
Context: Multi-head attention is the transformer's core mixing step: several attention heads run in parallel over different learned projections, and their outputs concatenate. Modeling it in stdlib shows why heads can attend differently.
Your task: Assemble multi-head attention: project the query/keys/values per head, run scaled dot-product attention in each head, and concatenate the head outputs.
Requirements:
- Each head has its own projection applied to q, k, and v
- Run your single-query scaled dot-product attention inside each head
- Use different projections so the heads attend to different dimensions
- Concatenate the per-head outputs into one vector
- Print each head's weights to show they differ, then the concatenated output
💡 Hint: Give two toy heads projections that isolate different input dimensions — you'll see them place attention on different keys before you concatenate.
Show solution
Each head projects q/k/v, attends, and the outputs concatenate. Runnable (toy projections, stdlib):
import math
def softmax(s):
m = max(s); e = [math.exp(x - m) for x in s]; Z = sum(e)
return [x / Z for x in e]
def head_attention(q, keys, values):
d = len(q)
scores = [sum(a*b for a, b in zip(q, k)) / math.sqrt(d) for k in keys]
w = softmax(scores)
out = [sum(wi * v[j] for wi, v in zip(w, values)) for j in range(len(values[0]))]
return w, out
def project(vec, W):
return [sum(vec[i] * W[i][j] for i in range(len(vec))) for j in range(len(W[0]))]
# two heads with different projections -> they attend to different things
Wq = {0: [[1, 0], [0, 0]], 1: [[0, 0], [0, 1]]} # head 0 sees dim0, head 1 sees dim1
q = [1.0, 1.0]
keys = [[1.0, 0.0], [0.0, 1.0]]
values = [[9.0, 0.0], [0.0, 9.0]]
concat = []
for h in (0, 1):
qh = project(q, Wq[h])
kh = [project(k, Wq[h]) for k in keys]
w, out = head_attention(qh, kh, values)
print(f"head {h}: weights={[round(x,2) for x in w]}")
concat += out
print("concat output:", [round(x, 2) for x in concat])Multiple heads let the model attend to different relationships in parallel (syntax, coreference, position); concatenating their outputs is the transformer's core mixing step. A trained model learns the projections — labeled needs-libs/GPU.
✓ Checkpoint — you can move on when you can…
- Explain self-attention as every token weighting every other.
- Name the transformer's parts and what multi-head & positional encoding do.
- Say why it scaled where RNNs couldn't (parallel + direct access).
- Walk architecture → pretraining → alignment → the LLM.
- Trace real LLM behavior (context cost, hallucination) to transformer facts.
Knowledge check check yourself
Explain self-attention using the query/key/value mechanism, and say which two prior-chapter problems it solves.
Show answer
Attention is O(n²) in sequence length. What practical LLM realities does this quadratic cost directly explain?