Transformers & the Path to Large Language Models

Self-attention: the core idea advanced

The RNN's flaw (K4) was forcing information through a single hidden state, one step at a time. Self-attention throws that out: for each token, the model looks directly at every other token in the sequence and decides how much each one matters for understanding this one. No chain, no bottleneck — every position can reach every other in a single step.

Start with the moment this idea clicks. Read the sentence "the animal didn't cross the street because it was tired," then ask yourself: what does "it" mean? You knew instantly — the animal. But notice what your brain just did. It didn't read left-to-right and hope to still be holding "animal" in working memory eight words later; it reached back across the whole sentence at once and connected "it" to the noun that fits. Every model before the transformer struggled with exactly this — the further apart two related words sat, the more likely the connection was lost. Self-attention is the mechanism that lets a machine do what you just did, without a chain to walk down.

Here is the plain idea, before any math. Imagine each word in the sentence gets to ask a question about the others and receive weighted answers back. "It" asks, roughly, "which earlier word am I standing in for?" — and every other word answers with a relevance score. "Animal" scores high; "street" and "tired" score low. The word's new, richer meaning is then a blend of all the others, mixed in proportion to those scores. Run that for every word simultaneously and each token walks away understanding its own place in the sentence's web of relationships. That is the entire trick, and it is why we call it self-attention: the sequence is paying attention to itself.

Why this is a leap and not just a tweak. The RNN of K4 was like reading a book through a mail-slot — one word slides past at a time, and to remember page one by page three hundred you must cram everything into a single fixed-size note (the hidden state). Two things go wrong: it is slow (you cannot start word three until you have finished word two), and it forgets (early words fade as the note is overwritten). Self-attention removes the mail-slot entirely. Every token sees every other token directly, in one parallel step — no chain to traverse, no bottleneck note to overflow. That single change fixes both of K4's walls at once, which is the drum this whole chapter beats.

The common misconception to kill early: attention is not the model "looking things up in a database" or running a search. It is a learned, weighted average. Nothing is fetched and copied; instead every value vector is blended in, with the most weight going to the most relevant. And the weights are not hand-coded rules — they are learned from data, which is why a well-trained model discovers on its own that pronouns tend to attend to nouns and verbs to their subjects. Hold that framing — a differentiable, learned blend of the whole context, computed for every token in parallel — and the diagram below will read like confirmation of something you already understand.

The transformer's parts advanced

The transformer wraps self-attention with a few more components. You don't need to implement them, but naming them makes model behavior legible.

Here's why this section is worth more than a quick skim. Attention on its own is powerful but incomplete — like a brilliant engine with no chassis, no transmission, and no steering. Left alone, raw self-attention has two glaring gaps that would make a real language model useless. First, it is completely order-blind: to attention, "dog bites man" and "man bites dog" are the same bag of words attending to each other (the exact failure that sank bag-of-words in K2). Second, attention only mixes information between tokens — it doesn't do much thinking with the information once gathered. The extra parts in the table below are the chassis and transmission that turn the engine into a working car.

Read the table as a set of answers to "what would go wrong if this piece were missing?" Positional encoding answers the order-blindness gap — it stamps each token with a signal of where it sits so the model can tell first from last. Multi-head attention answers "one kind of relationship isn't enough": rather than a single attention pass, the model runs several in parallel, and each head is free to specialize — one might track grammatical subject-verb links, another might track which pronoun refers to which noun, another topic. It's the difference between one person reading a contract and a whole team of specialists reading it at once. Feed-forward layers are where per-token "processing" happens after attention has gathered the context — a small neural network applied to each position, and, perhaps surprisingly, where a large share of the model's factual knowledge is stored.

The last two rows are pure engineering, but they are the reason the whole thing is trainable. Residual connections add each layer's input back to its output — a shortcut that lets gradients flow cleanly through dozens of layers instead of vanishing (the very problem that capped RNN depth in K4). Normalization keeps the numbers in each layer in a sane range so training doesn't blow up. And then you stack the block — attention plus feed-forward, wrapped in residuals and norm — dozens or hundreds of times. Depth is what builds abstraction: early layers pick up surface patterns (word shapes, simple syntax), and deeper layers assemble those into meaning, intent, and long-range structure. The mental model to keep: one transformer block is a small, repeatable unit; a transformer is that unit stacked deep, and an LLM is that stack made enormous.

Why the transformer won advanced

The 2017 paper's title — "Attention Is All You Need" — was the claim that you could drop recurrence entirely. It won because it solved both RNN walls (K4) simultaneously.

The stakes here are the entire modern AI industry, so it's worth understanding why this architecture and not another. Plenty of clever sequence models came before it, and many were more elaborate. The transformer won not because it was the smartest design but because it was the one you could scale — and in this field, scalability beats cleverness almost every time. The story is really about a single word: parallelism. An RNN is fundamentally sequential; its recurrence means step n literally cannot begin until step n−1 finishes. That is a death sentence on modern hardware, because a GPU is a machine with thousands of cores built to do thousands of things at the same time. Feeding it a strictly one-at-a-time computation is like hiring a thousand-person crew and letting only one work while the rest watch.

Self-attention has no such ordering constraint. Because every token attends to every other in one shot, you can compute all positions simultaneously — the whole sequence goes through the GPU at once, saturating those thousands of cores. That is the left-vs-right contrast in the diagram below: a chain you must walk versus a block you process in parallel. And the practical consequence is staggering: a training run that would have taken an RNN months could be done in days, which meant researchers could afford to train on far more data with far bigger models. Scale stopped being impossible and became merely expensive — and "expensive but possible" is exactly the regime where breakthroughs happen.

The second win rides along for free. Because any token reaches any other directly — one attention hop, not a hundred recurrent steps — the "long-range fade" that made RNNs forget the start of a paragraph by its end simply disappears. A word at position 500 looks straight back at position 1 with no signal loss. So the transformer didn't trade one weakness for another; it removed both of K4's walls at once. The lesson worth carrying forward: when the field bet everything on transformers, it wasn't betting on a uniquely brilliant idea — it was betting that parallel + direct-access would let scale do the heavy lifting. That bet paid off so completely that the next section's pipeline (architecture → pretrain → align) is really just "take this scalable engine and pour in as much data and compute as money allows."

Lab K5.1 · Architecture → pretraining → the LLM expert

The transformer is an architecture. Turning it into an LLM takes two more stages. This pipeline is the answer to "where does an LLM come from?"

This is the section that clears up the most common confusion beginners carry about LLMs. People conflate three very different things — the architecture, the trained-on-everything base model, and the helpful assistant you chat with — and treat them as one magical object. They are not. They are three distinct stages, built one on top of the next, and separating them explains almost every "why does it do that?" question you will ever ask. Think of it like a car factory: first you design the engine (the architecture), then you build and run it in raw form (pretraining), then you tune it, add the interior, and make it pleasant to drive (alignment). Each stage produces something real but unfinished until the last.

Stage one, the architecture, is everything you learned above — attention, multi-head, positional encoding, stacked blocks. On its own it is an empty engine: the wiring is there, but the millions of weights are random, so it knows nothing. Stage two, pretraining, is where all the raw capability comes from, and the surprising part is how simple the task is. The model plays one game, trillions of times: given the text so far, predict the next token — the exact next-token objective from K4, just run at unimaginable scale. Nobody labels the data; the text is its own answer key (the next word is always sitting right there). To get good at that one game across the whole internet, the model is forced to absorb grammar, facts, styles, reasoning patterns, even a rough world-model — because all of those help it guess the next word. The output is a base model: fluent, knowledgeable, and completely unhelpful as an assistant, because it only knows how to continue text, not to answer you.

Stage three, alignment, is what turns that raw predictor into Claude. Two things happen: instruction tuning shows it thousands of examples of "a request, and a good response," teaching it to treat your input as something to answer rather than continue; then preference-based training (RLHF, and Anthropic's Constitutional-AI-style methods) shapes it toward being helpful, honest, and harmless (C1). The subtle but crucial point — and a great myth-buster — is that alignment mostly surfaces and steers capability that pretraining already created; it rarely teaches new knowledge. That's why the assistant's personality is a thin, changeable layer over a vast base, and why the same base model can be aligned into very different-feeling products. Keep the three stages separate in your head and the whole rest of the chapter — base capability vs. assistant behavior, hallucination vs. persona — falls neatly into place.

Encoder, decoder, both expert

The original transformer had an encoder and a decoder (for translation, K4). Different LLM families use different slices — worth knowing because it explains what a model is good at.

This is the section that stops you from reaching for the wrong tool. A recurring beginner mistake is assuming "transformer" means "chatbot like Claude" — so people try to make a chat model produce search embeddings, or expect an embedding model to hold a conversation, and are baffled when it goes badly. The fix is realizing the transformer is a Lego brick, not a finished toy: the same self-attention block can be assembled three different ways, and each assembly is genuinely good at a different job. Knowing which is which turns "why doesn't this model do what I want?" into an architecture-selection decision you can make on purpose.

The dividing line is one deceptively simple question: can a token look at the tokens that come after it, or only the ones before? An encoder reads bidirectionally — every token attends to the entire input, both left and right — which is perfect for understanding a fixed piece of text you already have in full. That's why encoder-only models (BERT-style) power the embedding and classification jobs behind RAG (K2, K3): to summarize a sentence into one meaning vector, it helps to see the whole sentence at once. A decoder reads causally — each token may only attend to what came before it, never ahead — because its job is to generate the next token, and peeking at the future would be cheating (during training the answer would leak in). That left-to-right constraint is exactly the K4 next-token setup, which is why every chat and coding LLM you use, Claude included, is decoder-only.

The third assembly, encoder-decoder, bolts both together: an encoder fully digests an input sequence, then a decoder generates a fresh output sequence while attending back to that digest. This shines when the output is a transformation of a specific input rather than an open-ended continuation — translation (read the French, write the English) and some summarization models being the classic cases. The reason generative LLMs mostly abandoned this two-piece design in favor of pure decoders is scale and simplicity: it turns out a big enough decoder-only model, trained on enough text, can learn to translate and summarize just by predicting the next token, so the extra encoder machinery wasn't worth the complexity. Same brick, three assemblies — and now the mismatch surprises disappear, because you can name what each shape was built to do.

Transformer facts explain LLM behavior expert

The whole point of this module: knowing the machine explains the behavior you've worked with all course. Every LLM quirk traces to something here.

This is the section that earns the whole chapter its keep. Everything before it was mechanism; this is the payoff — the moment the machine stops being abstract and starts explaining things you have actually felt as a user of these models. The reason it matters so much is that mystery breeds superstition. Engineers who don't understand the transformer end up treating the LLM like a temperamental oracle — muttering incantations, adding "please" and "you are an expert," hoping. Engineers who do understand it reason about behavior mechanically and fix problems at the root. The table below is a Rosetta Stone between the two views: every column-two entry is a fact from this chapter; every column-one entry is a behavior you've bumped into.

Walk a couple of the rows to feel the pattern, because the pattern is the real lesson. Take the finite, costly context window: it's not an arbitrary product limit someone chose to annoy you — it falls straight out of attention being O(n²) in length, so doubling the input quadruples the work, and at some point that's simply too expensive to serve. Or take the biggest one, hallucination: the model was built and trained to do exactly one thing — predict a plausible next token — and a plausible continuation is not the same thing as a true one. The model has no built-in sense of "I don't actually know this"; it will confidently produce fluent text either way. That's not a bug you can prompt away; it's the defining trait of a next-token predictor, and it is why the course keeps insisting on grounding (RAG), evaluation, and guardrails.

The mental habit to build here — and to carry into every chapter that follows — is the "trace it to the machine" reflex. Whenever a model does something surprising, don't stop at "it's smart" or "it's dumb." Ask: which mechanism produced this? Prompt-order sensitivity? That's every token attending to every other, so rephrasing genuinely changes the whole representation. Few-shot examples working? That's attention letting the examples in your prompt directly steer the next-token prediction. The helpful, careful persona? That's the thin alignment layer (C1), not the base model. Once you can complete that chain from observed behavior back to a concrete transformer fact, the LLM stops being magic and becomes an engineering component with knowable failure modes — which is precisely the mindset the rest of this course is built to give you.

Common pitfalls expert

Exercises expert

import math

def softmax(scores):
    m = max(scores)
    exps = [math.exp(s - m) for s in scores]   # subtract max -> no overflow
    Z = sum(exps)
    return [e / Z for e in exps]

w = softmax([2.0, 1.0, 0.1, -1.0])
print([round(x, 3) for x in w])
print("sum =", round(sum(w), 6))   # 1.0 -- a valid probability distribution

import math

def softmax(scores):
    m = max(scores); exps = [math.exp(s - m) for s in scores]; Z = sum(exps)
    return [e / Z for e in exps]

def attention(q, keys, values):
    d = len(q)
    scores = [sum(qi * ki for qi, ki in zip(q, k)) / math.sqrt(d) for k in keys]
    w = softmax(scores)
    out = [0.0] * len(values[0])
    for wi, v in zip(w, values):
        for j in range(len(v)):
            out[j] += wi * v[j]
    return w, out

q = [1.0, 0.0]
keys   = [[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]
values = [[10.0, 0.0], [0.0, 10.0], [5.0, 5.0]]
w, out = attention(q, keys, values)
print("weights:", [round(x, 3) for x in w])
print("output :", [round(x, 3) for x in out])
# the query aligns with key 0 (and 2), so the output leans toward their values

import math, random

def softmax(scores):
    m = max(scores); e = [math.exp(s - m) for s in scores]; Z = sum(e)
    return [x / Z for x in e]

def demo(d, seed=0):
    rng = random.Random(seed)
    q = [rng.gauss(0, 1) for _ in range(d)]
    ks = [[rng.gauss(0, 1) for _ in range(d)] for _ in range(3)]
    raw = [sum(a*b for a, b in zip(q, k)) for k in ks]
    scaled = [s / math.sqrt(d) for s in raw]
    return max(abs(s) for s in raw), softmax(raw), softmax(scaled)

for d in (4, 64, 512):
    mag, un, sc = demo(d)
    print(f"d={d:>3}: max|score|={mag:7.2f}  "
          f"unscaled max-w={max(un):.3f}  scaled max-w={max(sc):.3f}")
# score magnitude grows ~sqrt(d); unscaled softmax saturates toward 1.0

import math

def positional_encoding(pos, d_model):
    pe = []
    for i in range(d_model):
        freq = 1.0 / (10000 ** (2 * (i // 2) / d_model))
        pe.append(math.sin(pos * freq) if i % 2 == 0 else math.cos(pos * freq))
    return pe

def cos_sim(a, b):
    dot = sum(x*y for x, y in zip(a, b))
    na = math.sqrt(sum(x*x for x in a)); nb = math.sqrt(sum(y*y for y in b))
    return dot / (na * nb)

d = 16
p0, p1, p10 = (positional_encoding(p, d) for p in (0, 1, 10))
print(f"sim(pos0, pos1)  = {cos_sim(p0, p1):.3f}")
print(f"sim(pos0, pos10) = {cos_sim(p0, p10):.3f}")
# adjacent positions are more similar than distant ones -- a smooth position signal

def architecture_for(task):
    table = {
        "classification":   ("encoder-only",    "BERT-style"),
        "embedding/search": ("encoder-only",    "sentence encoders"),
        "text generation":  ("decoder-only",    "GPT/Llama-style"),
        "chat":             ("decoder-only",    "GPT/Llama-style"),
        "translation":      ("encoder-decoder", "T5/BART-style"),
        "summarization":    ("encoder-decoder", "T5/BART-style"),
    }
    return table.get(task, ("decoder-only", "default for open-ended tasks"))

for t in ["classification", "text generation", "translation"]:
    fam, model = architecture_for(t)
    print(f"{t:>16} -> {fam:<16} ({model})")

import math

def softmax(s):
    m = max(s); e = [math.exp(x - m) for x in s]; Z = sum(e)
    return [x / Z for x in e]

def head_attention(q, keys, values):
    d = len(q)
    scores = [sum(a*b for a, b in zip(q, k)) / math.sqrt(d) for k in keys]
    w = softmax(scores)
    out = [sum(wi * v[j] for wi, v in zip(w, values)) for j in range(len(values[0]))]
    return w, out

def project(vec, W):
    return [sum(vec[i] * W[i][j] for i in range(len(vec))) for j in range(len(W[0]))]

# two heads with different projections -> they attend to different things
Wq = {0: [[1, 0], [0, 0]], 1: [[0, 0], [0, 1]]}   # head 0 sees dim0, head 1 sees dim1
q = [1.0, 1.0]
keys   = [[1.0, 0.0], [0.0, 1.0]]
values = [[9.0, 0.0], [0.0, 9.0]]
concat = []
for h in (0, 1):
    qh = project(q, Wq[h])
    kh = [project(k, Wq[h]) for k in keys]
    w, out = head_attention(qh, kh, values)
    print(f"head {h}: weights={[round(x,2) for x in w]}")
    concat += out
print("concat output:", [round(x, 2) for x in concat])

Knowledge check check yourself