Neural Language Models & Sequence Modelling

What a language model is advanced

Here's a claim that sounds too small to be true, so hold onto it. The thing that writes your emails, debugs your code, and drafts your legal memos is, underneath, doing exactly one thing on a loop: guessing the next word. Not "understanding," not "reasoning about your intent" — predicting the next token, then predicting the next one given that, then the next. If you believe nothing else from this chapter, believe that, because every strange behaviour you'll ever see from an LLM — its fluency, its confident wrong answers, its sensitivity to how you phrase things — falls straight out of that one fact.

The reason to open the whole sequence-models chapter here, rather than with an RNN diagram, is that the goal never changed — only the machinery did. N-grams, RNNs, LSTMs, transformers: every architecture in this chapter is a different attempt at the same job. So if you fix the job firmly in your head first, the rest of the chapter reads as a story of engineers building better and better next-token predictors, each one fixing the last one's weakness. Miss this framing and the models look like an arbitrary parade of acronyms; keep it and they line up as one clear arc.

An analogy that makes it click: think of the predictive-text bar above your phone keyboard. You type "I'll be there in five" and it offers "minutes." That little suggester is a language model — a weak one, looking back only a word or two. Now imagine that same suggester having read most of the public internet, tracking not two words but two thousand, and offering not one guess but a full ranked probability over every word it knows. That's an LLM. Same idea, wildly different scale — and scale, as you'll see by the end, is the entire plot.

The common mistake is to assume that because the output is so capable, the objective must be something grander than "predict the next token" — that there's a hidden reasoning module, or a fact database, bolted on. There isn't. Capability is an emergent property of doing next-token prediction extremely well over a staggering amount of text; the objective stays humble even as the behaviour grows sophisticated. Keep the humble objective in view and you'll never be surprised that the model can hallucinate a citation as fluently as it quotes a real one — to a next-token predictor, both are just plausible continuations.

Strip away the mystique: a language model assigns probabilities to sequences of tokens, which reduces to one repeated question — given the tokens so far, what's the next one? Everything an LLM does (chat, code, summaries) is this next-token prediction, run over and over, one token at a time.

Where it started: n-gram models advanced

Before there were neural networks, there was counting — and counting got you surprisingly far. If you wanted a machine to finish the sentence "peanut butter and ___," you didn't need it to understand sandwiches. You just needed a giant tally of every phrase people had ever written, so you could look up what word most often followed "peanut butter and" and answer "jelly." That's the whole idea behind n-gram models, and it's worth meeting them first because their failures are precisely what motivated everything that came after. You can't appreciate why an RNN's memory was a breakthrough until you've felt how little memory an n-gram has.

Here's the mental model. An n-gram model estimates the probability of the next word by looking back at a fixed number of previous words — and only that many. A bigram remembers one word back; a trigram, two. It's like trying to follow a conversation through a keyhole that only ever shows you the last two words spoken: you can guess the next word plausibly, but you have no idea what the paragraph is about. "The sat cat" and "the cat sat" are equally invisible to a model whose entire universe is two-word windows.

The natural reaction — and it's the common mistake — is to think, "fine, just use a bigger window, count 5-grams or 10-grams instead." This is where the wall is, and it's a wall of arithmetic, not of effort. Every word you add to the window multiplies the number of possible phrases by the size of the vocabulary. With a 50,000-word vocabulary, 5-grams have more possible combinations than there are atoms in your body, and the overwhelming majority of them never appear even once in any corpus you could collect. So the counts you'd need are almost all zero — the model has no data for nearly every real phrase, a problem called sparsity. Widening the window doesn't buy you memory; it buys you an ocean of zeros.

And there's a second, subtler flaw that no amount of data fixes: n-grams have no sense that words mean similar things. If the model has seen "black cat" a thousand times but never "dark cat," it treats "dark cat" as impossible — probability zero — even though you and I know the two phrases are near-identical in meaning. The model has no notion of similarity because it only ever counted exact strings. That's the generalization gap from K2, and it's structural: counting can't share evidence between related words. The table below lays out these three limits — window, sparsity, no generalization — and each one is a door that neural networks (embeddings for meaning, a hidden state for memory) were built to walk through.

The earliest statistical language models were n-grams: estimate the next word's probability by counting how often each word followed the previous n−1 words in a big corpus. A bigram model looks back one word; a trigram, two.

The RNN: a memory that walks the sequence advanced

The n-gram's fatal flaw was amnesia: a window two words wide. So the obvious question researchers asked was: what if the model kept a running summary — a little notebook it updated with every word it read, carrying meaning forward from any distance? That notebook is the single idea behind the Recurrent Neural Network, and it was a genuine breakthrough. For the first time a model could, in principle, connect a pronoun on line ten back to the name that introduced it on line one. Fixed windows were gone.

Picture a person reading a novel aloud with a sticky note in hand. Each word they read, they glance at the note, update it with a terse summary of the story so far, and read on. They never re-read the whole book — the note is their memory of everything before this point. An RNN works exactly like that: the sticky note is called the hidden state (a vector of numbers), and at every token the network does one small computation — blend the current word with the current note to produce an updated note plus a prediction for what comes next. Read a word, update the state, predict, repeat. The state is a compressed memory of the entire history, refreshed one step at a time.

The word "recurrent" is worth pausing on, because it names the whole trick: the network's output at each step is fed back in as an input to the next step. It's the same handful of weights applied over and over down the sequence — one small machine, reused at every position, threading its memory through. That reuse is elegant and it's why the diagram below draws the box once per word: it really is the same box each time, just shown at successive moments as the hidden state flows left to right.

But hold two things in mind as you read on, because they're the seeds of the RNN's downfall. First, that sticky note is small and gets overwritten every step — early words fade as newer ones pile on, which becomes the memory problem in the LSTM and limits sections. Second, and more quietly lethal: because each note depends on the previous note, the model cannot skip ahead. Word five's computation literally cannot begin until word four's is finished. That strict left-to-right chain, so natural for a human reader, is exactly what will make RNNs impossible to train fast on parallel hardware — the wall that ultimately ended them. The breakthrough and the fatal flaw are the same feature, seen from two sides.

A Recurrent Neural Network processes tokens one at a time, maintaining a hidden state — a vector that summarizes everything seen so far. At each step it combines the current token with the previous hidden state to produce a new hidden state and a prediction. In principle, that state can carry information from any distance.

LSTMs: patching the memory problem expert

A plain RNN's sticky note has a cruel property: it's rewritten in full at every single step. Read enough words and whatever was on it at the start is smeared into oblivion — the model that could in principle remember token one has, in practice, forgotten it by token twenty. This isn't just hand-waving; it has a precise mechanical cause called the vanishing gradient problem, and it's worth understanding because it explains why the fix took the shape it did.

Here's the intuition without the calculus. To learn, the network sends an error signal backward through every step of the chain, adjusting weights. But that backward signal gets multiplied by a factor at each step, and those factors are usually less than one. Multiply a number below one by itself twenty times and it collapses toward zero — so by the time the signal has travelled back twenty words, it's a rounding error. The network literally cannot feel that word one mattered to word twenty; the lesson never reaches that far. Long-range learning dies not because the idea is impossible but because the training signal evaporates on the way.

The LSTM (Long Short-Term Memory) and its lighter cousin the GRU fix this with a beautiful piece of engineering: a second memory track — a "cell state" you can picture as a conveyor belt running straight through the sequence, largely untouched. Instead of overwriting everything each step, the network learns small gates — valves, really — that decide what to add to the belt, what to erase from it, and what to read off it for the current prediction. Because information can ride the belt across many steps without being repeatedly mangled, both the memory and the gradient survive far longer. The gates are learned, so the model discovers on its own that "the subject of this sentence" is worth keeping while "that comma five words ago" is not.

This is the same deliberate keep-versus-discard logic you'll meet again in long-term agent memory (M4) and LangGraph checkpointers (L5) — deciding what deserves to persist is a recurring theme in this course, and the LSTM is where it first appears in a neural net. But note the honest framing in the callout below, because it's the point of this whole section: LSTMs patched the memory problem, they didn't cure it. Distant information still fades, just more slowly, and — crucially — the conveyor belt did nothing at all about the other flaw, the strict left-to-right chain that blocks parallelism. LSTMs bought the RNN era years of dominance in translation and speech, but they left the deeper wall standing.

Plain RNNs forget fast — over a long sequence, early information washes out of the hidden state (the vanishing gradient problem: the training signal shrinks toward zero as it propagates back through many steps). LSTMs (and GRUs) added gates — learned mechanisms that decide what to keep, forget, and output — letting the network hold information much longer. LSTMs powered the best NLP for years: translation, speech, text generation.

The two walls that killed RNNs expert

This is the section that explains why the entire field pivoted — so it's the one to read slowly. The stakes are enormous: had these two walls not existed, there might have been no transformer, no GPT, no Claude, and this course would be about tuning LSTMs. The reason we have LLMs at all is that engineers eventually decided these two limits couldn't be patched away and needed a fundamentally different architecture. Understanding them is what turns the transformer from "a clever thing that appeared in 2017" into "the obvious answer to two specific, painful problems."

The two walls are easy to name and easy to confuse, so let's be precise. Wall one is memory: even with the LSTM's conveyor belt, information from far back still fades over a long enough sequence. Better than a plain RNN, but not solved — ask an LSTM about something a paragraph ago and it often can't reach it. This one people expect. Wall two is speed, and it's the one that actually mattered: because each hidden state depends on the previous one, the model must compute the sequence strictly in order, one token after another. It physically cannot work on token five while it's still figuring out token four.

Why is that second wall the killer? Here's the analogy that makes it land. Imagine you have a hundred cooks and a thousand meals to prepare. If each meal is independent, you hand them out and all hundred cooks work at once — done in a flash. But if every meal must be cooked in sequence because each one uses the previous meal's leftover sauce, then 99 of your cooks stand idle while one works down the line. That's an RNN on a modern GPU. GPUs are those hundred cooks — thousands of processors built to do enormous numbers of calculations simultaneously. An architecture chained to strict left-to-right order can't hand them parallel work, so it wastes the very hardware that makes large-scale training possible. You can't train on trillions of tokens across thousands of GPUs if your model insists on going one token at a time.

The common mistake here is to remember only wall one — to think RNNs failed because they "forgot things." Memory was the famous problem, but it was partly fixable and had been partly fixed. The parallelization wall was the structural dead end: no amount of gating changes the fact that recurrence is inherently sequential. So as you read the diagram and table below, keep the emphasis where it belongs — the transformer's headline victory in the next chapter is not "it remembers better" (though it does), it's "it computes every position at once," and that is what unlocked the scale that made everything else possible.

Even LSTMs hit hard ceilings. Two limitations — one about memory, one about hardware — are the exact problems the transformer (K5) was designed to eliminate.

Sequence-to-sequence & the birth of attention expert

This section is where the hero of the whole LLM story quietly walks on stage — not as a headline act, but as a bug fix. Attention, the mechanism that would go on to define the transformer and every model built on it, wasn't invented to revolutionize AI. It was invented to patch an annoying weakness in machine translation. That's a genuinely useful thing to know, because it tells you how real breakthroughs happen: someone fixes a concrete, unglamorous problem, and only later does everyone realize the fix was the actual point.

First, the setup. Translating a sentence is a "sequence in, sequence out" problem, so the RNN-era solution was the seq2seq architecture: one RNN, the encoder, reads the entire source sentence and boils it down to a single fixed-length vector; a second RNN, the decoder, reads that vector and generates the translation word by word. Clean, symmetrical, and — for short sentences — it worked.

The flaw is one you can feel physically. Imagine being asked to read a full paragraph, then repeat it perfectly to someone else using only a single index card to carry between them. For one sentence, fine. For a dense paragraph, the card can't hold it all — you write frantically, run out of room, and the far end reconstructs a blurry, lossy version. That index card is the fixed encoder vector, and this is the bottleneck: no matter how long or rich the input, everything must squeeze through one same-sized vector. Translation quality was observed to fall off a cliff as sentences got longer, for exactly this reason.

The fix, introduced around 2014–2015, was attention, and the idea is intuitive once you drop the single-card constraint. What if the decoder, instead of relying only on that one squeezed summary, could look back at the encoder's notes for every input word — and, for each word it's about to produce, decide which input words are most relevant right now? Translating the verb of the sentence? Pay attention to the source verb. Translating a noun? Look back at that noun. It's the difference between reciting from a blurry index card and being allowed to glance back at the original paragraph, focusing on the line that matters for the word you're writing.

Attention worked so well as a bolt-on that it raised a radical question, and that question is the entire bridge to the next chapter: if attention is doing the real work of connecting relevant words across any distance — and it does so by looking at all positions at once, with no left-to-right chain — then why keep the RNN at all? What if you threw out the recurrence entirely and built the whole model from attention? That thought, published in 2017 as "Attention Is All You Need," is the transformer. Notice it demolishes both walls at once: attention reaches any distance directly (memory), and it processes every position simultaneously (parallelism). The rest of this course lives downstream of that single idea.

Before transformers, the standard architecture for translation was seq2seq: an RNN encoder compresses the whole input sentence into a single fixed vector, and an RNN decoder generates the output from it. The flaw is obvious once you see it — everything the input said must squeeze through one vector.

What survived into the LLM era expert

It's tempting to read this chapter as "RNNs lost, transformers won, forget the losers" — and that would be the wrong lesson. Revolutions in engineering almost never throw everything away; they keep the parts that were right and replace only the part that was broken. The transformer is a perfect case study: it inherited far more from the RNN era than it discarded. Sorting out exactly what carried over and what got swapped is worth doing deliberately, because it's how you build an accurate mental model of a modern LLM instead of a superstitious one.

Start with what survived unchanged, because it's the most important and the most surprising. The next-token prediction objective — the very first idea in this chapter — is exactly how today's largest models are trained and how they generate. Nothing about that goal changed from the n-gram counting table to claude-opus-4-8; only the predictor got better. Embeddings survived too: transformers still turn tokens into vectors first, the same trick from K2. And the encoder/decoder shape survived and generalized — transformers come as encoder-only, decoder-only, or both, depending on the task.

Then there's the idea that got promoted: attention went from a humble seq2seq add-on to the load-bearing pillar of the entire architecture. And exactly one big idea got dropped — recurrence, the hidden state passed step to step. That's the whole substitution in a sentence: transformers kept the RNN era's goal (predict the next token) and its inputs (embeddings) and replaced its mechanism (sequential recurrence) with parallel self-attention. The table makes this survived-versus-replaced split concrete, and the callout that follows it is the honest coda: recurrence isn't dead everywhere — it still has niches, and modern state-space models are quietly reviving the idea with fixes for its old flaws — but for general language modeling at scale, the swap was decisive.

Common pitfalls expert

Exercises expert

from collections import Counter, defaultdict

def train_bigram(corpus):
    counts = defaultdict(Counter)
    for sent in corpus:
        toks = ["<s>"] + sent.split() + ["</s>"]
        for a, b in zip(toks, toks[1:]):
            counts[a][b] += 1
    return counts

def next_probs(counts, prev):
    c = counts[prev]; total = sum(c.values())
    return {w: round(n / total, 3) for w, n in c.most_common()}

corpus = ["the cat sat", "the cat ran", "the dog sat"]
counts = train_bigram(corpus)
print("after 'the':", next_probs(counts, "the"))   # cat 0.667, dog 0.333
print("after 'cat':", next_probs(counts, "cat"))   # sat 0.5, ran 0.5

import random
from collections import Counter, defaultdict

def train_bigram(corpus):
    counts = defaultdict(Counter)
    for sent in corpus:
        toks = ["<s>"] + sent.split() + ["</s>"]
        for a, b in zip(toks, toks[1:]):
            counts[a][b] += 1
    return counts

def generate(counts, seed=0, max_len=12):
    rng = random.Random(seed)
    cur, out = "<s>", []
    for _ in range(max_len):
        choices, weights = zip(*counts[cur].items())
        cur = rng.choices(choices, weights=weights)[0]
        if cur == "</s>":
            break
        out.append(cur)
    return " ".join(out)

counts = train_bigram(["the cat sat", "the cat ran fast", "the dog sat"])
print(generate(counts, seed=1))
print(generate(counts, seed=7))

import math

def matvec(M, v):
    return [sum(mij * vj for mij, vj in zip(row, v)) for row in M]

def add(a, b):
    return [x + y for x, y in zip(a, b)]

def rnn_forward(inputs, Wx, Wh, b, h0):
    h = h0
    states = []
    for x in inputs:
        pre = add(add(matvec(Wx, x), matvec(Wh, h)), b)
        h = [math.tanh(z) for z in pre]     # squashing keeps state bounded
        states.append(h)
    return states

Wx = [[0.5, 0.0], [0.0, 0.5]]
Wh = [[0.1, 0.0], [0.0, 0.1]]
b  = [0.0, 0.0]
seq = [[1.0, 0.0], [0.0, 1.0], [1.0, 1.0]]   # 3 timesteps, 2 features
for t, h in enumerate(rnn_forward(seq, Wx, Wh, b, [0.0, 0.0])):
    print(f"t={t}: h = {[round(x, 3) for x in h]}")

def gradient_flow(steps, per_step_factor):
    g = 1.0
    trace = []
    for t in range(1, steps + 1):
        g *= per_step_factor          # chain rule multiplies each step
        trace.append((t, g))
    return trace

print("factor 0.6 (typical tanh region):")
for t, g in gradient_flow(20, 0.6):
    if t % 4 == 0:
        print(f"  after {t:>2} steps: gradient scale = {g:.2e}")
# by step 20 the gradient is ~10^-5: the signal from far-back tokens is gone

def fixed_vector_recall(input_len, vector_capacity=8):
    # a fixed vector can faithfully hold ~capacity tokens; the rest degrade
    kept = min(input_len, vector_capacity)
    return kept / input_len

def attention_recall(input_len):
    return 1.0   # attention can look back at every token, no fixed bottleneck

for n in (5, 8, 20, 100):
    print(f"len={n:>3}: fixed-vector recall={fixed_vector_recall(n):.2f}  "
          f"attention recall={attention_recall(n):.2f}")
# long inputs overflow the fixed vector; attention keeps full recall

LEGACY = {
    "next-token objective": "SURVIVED -- LLMs still predict the next token",
    "word embeddings":      "SURVIVED -- input/output embedding tables",
    "teacher forcing":      "SURVIVED -- used in pretraining",
    "attention":            "SURVIVED & CENTRAL -- transformers are attention",
    "recurrent hidden state": "REPLACED -- self-attention, no sequential state",
    "fixed context vector": "REPLACED -- attention over all positions",
    "sequential-only compute": "REPLACED -- transformers parallelize over tokens",
}
def status(idea):
    return LEGACY.get(idea, "unknown -- check the lesson")

for idea in ["next-token objective", "recurrent hidden state", "attention"]:
    print(f"{idea:>24}: {status(idea)}")

Knowledge check check yourself