Neural Language Models & Sequence Modelling
A language model does one thing: predict the next token. This chapter is how that idea became neural — from counting n-grams, to RNNs that carry a memory across a sequence, to the bottleneck that eventually forced the invention of attention. It's the direct run-up to the transformer.
Learning objectives
- Define a language model as next-token prediction — the objective behind LLMs.
- Explain n-gram models and why they can't scale to real context.
- Understand RNNs/LSTMs: a hidden state that carries memory through a sequence.
- Name the two fatal limits — vanishing gradients and sequential (un-parallelizable) computation.
- See how the RNN bottleneck motivated attention (K5).
claude-opus-4-8 is trained on. Understanding RNNs and why they hit a wall is what makes the transformer's design (K5) feel inevitable rather than arbitrary.What a language model is advanced
Strip away the mystique: a language model assigns probabilities to sequences of tokens, which reduces to one repeated question — given the tokens so far, what's the next one? Everything an LLM does (chat, code, summaries) is this next-token prediction, run over and over, one token at a time.
This is the one idea the whole chapter is built on: a language model reads the words so far and guesses what word comes next. Read it strictly left to right.
- The left box is your input — the sentence so far, with a blank at the end:
"the cat sat on the ___". That trailing blank is what the model has to fill in. - The arrow feeds that text into the middle box labelled
LM(the language model). An arrow always means "this goes into here". - The right side is the model's answer — but notice it is not a single word. It is a ranked list of candidates with numbers:
mat 0.4,floor 0.2,roof 0.05. Those numbers are probabilities (they add up to 1 across the whole vocabulary); a higher number means the model thinks that word is more likely to come next. - To actually generate text you pick one word from that list, stick it onto the end of the sentence, and run the whole thing through again — over and over, one word at a time.
In short: Reading the model = text in, list of likely next words out. Repeat that loop and you get sentences. Everything a modern LLM does is this one step, done extremely well.
Where it started: n-gram models advanced
The earliest statistical language models were n-grams: estimate the next word's probability by counting how often each word followed the previous n−1 words in a big corpus. A bigram model looks back one word; a trigram, two.
| n-gram limit | Consequence |
|---|---|
| Fixed, tiny window | A trigram sees 2 words back — no real context or long-range meaning |
| Combinatorial explosion | Counting all 5-grams needs astronomically much data; most never appear (sparsity) |
| No generalization | "black cat" seen, "dark cat" unseen → zero probability, despite similar meaning (the K2 problem) |
The RNN: a memory that walks the sequence advanced
A Recurrent Neural Network processes tokens one at a time, maintaining a hidden state — a vector that summarizes everything seen so far. At each step it combines the current token with the previous hidden state to produce a new hidden state and a prediction. In principle, that state can carry information from any distance.
h passes from each step to the next, so token 4's prediction can (in theory) depend on token 1. That was the breakthrough over n-grams: variable-length memory. But two things flow from that left-to-right chain — a memory problem and a speed problem — and both proved fatal.
This shows an RNN reading a sentence one word at a time. The trick to reading it: the same little machine is drawn once per word, so you can watch its memory get handed forward.
- Along the bottom are the input words in order —
the,cat,sat,on. Time flows left to right, one word per step. - The short upward arrows (from
x1…x4) feed each word up into the box above it. So box 1 gets "the", box 2 gets "cat", and so on. - The green boxes
h1 h2 h3 h4are the hidden state — think of it as the model's running memory, a scratchpad summarising everything read so far. - The key part is the horizontal arrows between the boxes: each step passes its memory to the next step. That is why the box reading "on" can still know about "the cat" from the start — the memory was carried along the chain.
- The catch (see the caption): the memory can only move one step at a time, in order. Nothing can jump ahead or be computed in parallel — and that single fact becomes the fatal flaw two sections later.
In short: An RNN = a reader with a notepad. It reads one word, updates the notepad, passes it on. Great for carrying memory; slow because it can never skip ahead.
LSTMs: patching the memory problem expert
Plain RNNs forget fast — over a long sequence, early information washes out of the hidden state (the vanishing gradient problem: the training signal shrinks toward zero as it propagates back through many steps). LSTMs (and GRUs) added gates — learned mechanisms that decide what to keep, forget, and output — letting the network hold information much longer. LSTMs powered the best NLP for years: translation, speech, text generation.
The two walls that killed RNNs expert
Even LSTMs hit hard ceilings. Two limitations — one about memory, one about hardware — are the exact problems the transformer (K5) was designed to eliminate.
This diagram is just two labelled boxes side by side — the two problems that eventually killed RNNs. There are no arrows; each box is one independent problem to read on its own.
- Box 1 · Long-range fade: as the sentence gets long, the memory of the earliest words gets fainter and fainter until it is effectively gone. Even the improved LSTM version (next section) only softens this — it doesn't cure it.
- Box 2 · Sequential compute: because each step needs the previous step's memory before it can run, the model must go word-by-word in order. It cannot do many words at once.
- Why box 2 hurts so much: modern GPUs (the chips used to train these models) are fast precisely because they do thousands of things in parallel. A model forced to go strictly one step at a time can't use that power, so it is painfully slow to train at large scale.
- Read the two boxes as "memory problem" (box 1) and "speed problem" (box 2). The next chapter's transformer is designed specifically to fix both.
In short: Two walls: RNNs forget distant words, and they can't be sped up by parallel hardware. The second wall is the one that made RNNs a dead end for big models.
| Limit | Why it's fatal | What fixes it (K5) |
|---|---|---|
| Long-range dependency | Meaning that spans a paragraph gets lost — the K1 context problem, still unsolved | Attention: every token can look directly at every other, at any distance |
| Sequential computation | Token-by-token processing can't use parallel GPUs → can't scale training | Attention processes all positions at once — fully parallelizable |
Sequence-to-sequence & the birth of attention expert
Before transformers, the standard architecture for translation was seq2seq: an RNN encoder compresses the whole input sentence into a single fixed vector, and an RNN decoder generates the output from it. The flaw is obvious once you see it — everything the input said must squeeze through one vector.
This shows the old translation setup (seq2seq) and the fix that led to modern models. Follow the solid arrows left to right first, then look at the curved dashed arrow on top.
- Encoder RNN (left) reads the whole input sentence and squeezes it down into the tiny middle box marked
1 vec— a single fixed-size summary ("vector") of the entire input. - Decoder RNN (right) takes that one summary and writes out the translation, word by word.
- The problem (the "bottleneck"): everything the input said has to fit through that one small box. For a long sentence, detail gets lost — like summarising a paragraph in five words and then trying to rebuild the paragraph from just those five words.
- The curved dashed arrow on top (highlighted in amber) is attention, the fix: it lets the decoder reach back and look at all of the encoder's per-word states, not just the single squeezed summary — and focus on the words that matter for each output word.
- That amber arrow is the hinge of the whole module: attention worked so well as an add-on that researchers dropped the RNN entirely and built a model out of attention alone — the transformer (K5).
In short: Encoder squeezes the sentence into one box, decoder unpacks it — but the squeeze loses detail. Attention lets the decoder peek back at the full input, and that idea becomes the transformer.
What survived into the LLM era expert
| RNN-era idea | Where it lives now |
|---|---|
| Next-token prediction objective | Unchanged — it's exactly how LLMs are trained & generate |
| Embeddings as input | Kept — transformers embed tokens too (K2) |
| Attention | Promoted from add-on to the core mechanism (K5) |
| Hidden-state recurrence | Dropped — replaced by parallel self-attention |
| Encoder/decoder structure | Kept & generalized — transformers come in encoder, decoder, or both |
Common pitfalls expert
| Pitfall | Fix |
|---|---|
| Thinking LLMs are more than next-token predictors at core | That objective is the core; capability emerges from scale |
| Assuming n-grams "just needed more data" | Sparsity & no-generalization are structural, not data-size |
| Believing RNNs failed only on memory | The parallelization limit is what really blocked scale |
| Thinking attention was invented with transformers | It began as a seq2seq add-on; transformers made it everything |
| Dismissing the lineage as irrelevant | It explains context windows, token costs, and why transformers won |
Exercises expert
Exercise K4.1 — Next-token by hand
Context: Building and sampling a bigram model by hand makes the fixed-window limitation tangible: locally fine, globally incoherent.
Your task: Build a tiny bigram model over a paragraph, then generate by repeatedly sampling the next word — and explain the incoherence in terms of the fixed window.
Requirements:
- Count word pairs across the paragraph
- Generate by sampling the next word from those counts
- Observe that each adjacent pair looks plausible
- Explain the global drift as the one-token memory window
💡 Hint: Each pair is locally fine, but with memory of only one token there is no topic or long-range agreement — the gap RNNs and attention were built to close.
Show what to look for
Each pair looks fine ("the cat", "cat sat") but the output drifts with no memory beyond one word — no topic, no long-range agreement. That incoherence is exactly the context limitation that RNNs, then attention, were built to fix.
Exercise K4.2 — Why can't we parallelize an RNN?
Context: The single architectural difference between an RNN and a transformer — sequential state versus parallel attention — is why transformers could scale.
Your task: Explain to a peer, in two sentences, why an RNN must process tokens in order while a transformer need not, then state why that difference mattered for training LLMs at scale.
Requirements:
- State that an RNN's hidden state forces strictly sequential processing
- State that a transformer attends over all positions in parallel
- Connect the parallelism to feasible large-scale training
- Keep it to a peer-level, two-sentence explanation
💡 Hint: The recurrent dependency chains each step to the last; attention removes that chain, and parallel compute is what unlocked scale.
Exercise K4.3 — Trace the lineage
Context: Tracing the progression from n-grams to transformers — one problem solved and one left open at each step — is the map that ties the whole module together.
Your task: Draw the progression n-gram → RNN → LSTM → seq2seq+attention → transformer, and for each step write the one problem it solved and the one it left open.
Requirements:
- Lay out the five stages in order
- For each stage, name the one problem it solved
- For each stage, name the one problem it left open
- Show how each open problem motivates the next stage
💡 Hint: Each arrow should read as "solved X, but left Y" — and Y is exactly what the next model was invented to address.
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: A language model, at heart, assigns a probability to the next token. A bigram counter is that idea in its simplest form — counting instead of a neural net.
Your task: Build a bigram counter over a corpus and turn the counts into next-token probabilities given a previous token.
Requirements:
- Count
(prev → next)token pairs across the corpus - Add sentence-boundary markers so starts and ends are modeled
- Normalize the counts per previous token into a probability distribution
- Report the next-token distribution for a given previous token
💡 Hint: The essence of a language model is a distribution over the next token; here it comes straight from normalized counts.
Show solution
Count (prev→next) pairs, normalize per prev. Runnable:
from collections import Counter, defaultdict
def train_bigram(corpus):
counts = defaultdict(Counter)
for sent in corpus:
toks = ["<s>"] + sent.split() + ["</s>"]
for a, b in zip(toks, toks[1:]):
counts[a][b] += 1
return counts
def next_probs(counts, prev):
c = counts[prev]; total = sum(c.values())
return {w: round(n / total, 3) for w, n in c.most_common()}
corpus = ["the cat sat", "the cat ran", "the dog sat"]
counts = train_bigram(corpus)
print("after 'the':", next_probs(counts, "the")) # cat 0.667, dog 0.333
print("after 'cat':", next_probs(counts, "cat")) # sat 0.5, ran 0.5This is the essence of a language model — a distribution over the next token — just with counting instead of a neural net.
Context: Sampling from the bigram model reveals its limit: it remembers only one token back, so generated text stays locally plausible but wanders globally — the context limitation RNNs were built to fix.
Your task: Generate a sentence by repeatedly sampling the next token from the bigram distribution until the end-of-sentence marker.
Requirements:
- Start from the sentence-start marker
- Sample the next token proportional to its counts
- Stop at the end-of-sentence token or a max length
- Show that the output drifts, having memory of only one prior token
💡 Hint: Because each step conditions on a single previous token, the text has no long-range memory — that wandering is the motivation for real recurrence.
Show solution
Sample proportionally to counts; stop at the end token. Runnable:
import random
from collections import Counter, defaultdict
def train_bigram(corpus):
counts = defaultdict(Counter)
for sent in corpus:
toks = ["<s>"] + sent.split() + ["</s>"]
for a, b in zip(toks, toks[1:]):
counts[a][b] += 1
return counts
def generate(counts, seed=0, max_len=12):
rng = random.Random(seed)
cur, out = "<s>", []
for _ in range(max_len):
choices, weights = zip(*counts[cur].items())
cur = rng.choices(choices, weights=weights)[0]
if cur == "</s>":
break
out.append(cur)
return " ".join(out)
counts = train_bigram(["the cat sat", "the cat ran fast", "the dog sat"])
print(generate(counts, seed=1))
print(generate(counts, seed=7))Sampling reveals the model's limits: a bigram only remembers one token back, so it wanders — motivating models with real memory (RNNs).
Context: An RNN carries a hidden state through the sequence; that state is its memory, blending each new input with a running summary of everything before it.
Your task: Implement one RNN cell's forward pass — h_t = tanh(W_x·x + W_h·h_<t-1> + b) — over a short sequence in pure Python.
Requirements:
- Step a hidden state through the sequence one timestep at a time
- At each step combine the input projection and the previous-state projection
- Apply a
tanhsquashing so the state stays bounded - Return the hidden state at every timestep
💡 Hint: The hidden state is the whole idea: each step folds the new input into a running summary — the mechanism, minus the learned weights.
Show solution
h_t = tanh(W_x·x + W_h·h_{t-1} + b), stepped over the sequence. Runnable, stdlib math:
import math
def matvec(M, v):
return [sum(mij * vj for mij, vj in zip(row, v)) for row in M]
def add(a, b):
return [x + y for x, y in zip(a, b)]
def rnn_forward(inputs, Wx, Wh, b, h0):
h = h0
states = []
for x in inputs:
pre = add(add(matvec(Wx, x), matvec(Wh, h)), b)
h = [math.tanh(z) for z in pre] # squashing keeps state bounded
states.append(h)
return states
Wx = [[0.5, 0.0], [0.0, 0.5]]
Wh = [[0.1, 0.0], [0.0, 0.1]]
b = [0.0, 0.0]
seq = [[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]] # 3 timesteps, 2 features
for t, h in enumerate(rnn_forward(seq, Wx, Wh, b, [0.0, 0.0])):
print(f"t={t}: h = {[round(x, 3) for x in h]}")The hidden state h is the RNN's memory: each step blends the new input with the running summary — the mechanism, minus the learned weights.
Context: Plain RNNs hit a wall: gradients backpropagated through many tanh steps multiply per-step factors below 1 and decay geometrically, so signal from far-back tokens vanishes.
Your task: Model gradient magnitude across timesteps to show why long-range dependencies vanish in a deep RNN.
Requirements:
- Multiply a per-step factor down the chain, one factor per timestep
- Use a factor below 1 to mimic
tanhsaturation - Trace how the gradient scale shrinks as the step count grows
- Show it becomes negligible after enough steps, killing long dependencies
💡 Hint: A per-step factor under 1 compounds exponentially with distance; LSTMs add a gated cell state to keep a near-1 path for the gradient.
Show solution
Backprop multiplies per-step factors (<1 for tanh saturation); the product decays geometrically. Runnable:
def gradient_flow(steps, per_step_factor):
g = 1.0
trace = []
for t in range(1, steps + 1):
g *= per_step_factor # chain rule multiplies each step
trace.append((t, g))
return trace
print("factor 0.6 (typical tanh region):")
for t, g in gradient_flow(20, 0.6):
if t % 4 == 0:
print(f" after {t:>2} steps: gradient scale = {g:.2e}")
# by step 20 the gradient is ~10^-5: the signal from far-back tokens is goneWith per-step factors below 1, the gradient decays exponentially with distance — so plain RNNs cannot learn dependencies more than a handful of tokens back. LSTMs add a gated cell state to keep a near-1 path.
Context: Seq2seq crammed an entire input into one fixed vector — the bottleneck attention removed by letting the decoder read every input position directly.
Your task: Model the information loss of a fixed encoder vector versus attention's per-token access as input length grows.
Requirements:
- Treat the fixed vector as holding only about a capacity's worth of tokens
- Show its faithful recall degrades as the input grows past that capacity
- Model attention as retaining full recall regardless of length
- Contrast the two across several input lengths
💡 Hint: The fixed vector overflows on long inputs; attention keeps full recall by re-reading every position — the idea transformers took to its conclusion.
Show solution
Model 'capacity' as a fixed budget for the encoder vector vs attention re-reading each token. Runnable:
def fixed_vector_recall(input_len, vector_capacity=8):
# a fixed vector can faithfully hold ~capacity tokens; the rest degrade
kept = min(input_len, vector_capacity)
return kept / input_len
def attention_recall(input_len):
return 1.0 # attention can look back at every token, no fixed bottleneck
for n in (5, 8, 20, 100):
print(f"len={n:>3}: fixed-vector recall={fixed_vector_recall(n):.2f} "
f"attention recall={attention_recall(n):.2f}")
# long inputs overflow the fixed vector; attention keeps full recallThe fixed encoder vector degrades as inputs grow; attention removes the bottleneck by letting the decoder read every input position — the idea transformers took to its conclusion.
Context: Several RNN-era ideas live on in transformers while others were replaced. A concept map of what survived versus what was swapped out is the spine of the whole module.
Your task: Build a lookup that, for each RNN-era idea, reports whether it survived into the LLM era or was replaced.
Requirements:
- Encode the survival map as data keyed by idea
- Mark the next-token objective, embeddings, teacher forcing, and attention as survivors
- Mark recurrent hidden state, the fixed context vector, and sequential-only compute as replaced
- Look up and report the status for a given idea
💡 Hint: The through-line: the objective and embeddings carried over, but recurrence was replaced by attention so compute could parallelize across tokens.
Show solution
Encode the lesson's survival map as data. Runnable:
LEGACY = {
"next-token objective": "SURVIVED -- LLMs still predict the next token",
"word embeddings": "SURVIVED -- input/output embedding tables",
"teacher forcing": "SURVIVED -- used in pretraining",
"attention": "SURVIVED & CENTRAL -- transformers are attention",
"recurrent hidden state": "REPLACED -- self-attention, no sequential state",
"fixed context vector": "REPLACED -- attention over all positions",
"sequential-only compute": "REPLACED -- transformers parallelize over tokens",
}
def status(idea):
return LEGACY.get(idea, "unknown -- check the lesson")
for idea in ["next-token objective", "recurrent hidden state", "attention"]:
print(f"{idea:>24}: {status(idea)}")The through-line: the objective and embeddings carried over; the recurrence was replaced by attention so compute could parallelize across tokens.
✓ Checkpoint — you can move on when you can…
- Define a language model as next-token prediction and connect it to LLMs.
- Explain why n-grams can't scale (window, sparsity, no generalization).
- Describe an RNN's hidden state and what LSTMs' gates fixed.
- Name the two RNN walls — long-range fade and sequential compute.
- Explain how the seq2seq bottleneck gave birth to attention.
Knowledge check check yourself
RNNs hit two 'walls'. Which one does the lesson say really mattered for scaling LLMs, and why?
Show answer
How did attention originally arise in seq2seq, and what was the pivotal insight that turned it into the transformer?