NLP Foundations & Text Processing
LLMs didn't appear from nowhere — they're the current peak of decades of natural language processing. This module walks that arc so the models you already use stop being magic. It starts at the base: what NLP is, why human language is genuinely hard for computers, and the text-processing pipeline that underlies everything after.
This section opens the black box: how text becomes numbers a model can process, and how the transformer architecture (the 'T' in GPT) turns those numbers into predictions. It's the theory behind everything else — useful for debugging, for interviews, and for genuinely understanding the tools you've been using.
The words you'll hear (in plain terms):
| Term | What it actually means |
|---|---|
| NLP | Natural Language Processing — getting computers to work with human language. |
| token / embedding | text split into chunks (tokens), each turned into a vector of numbers (embedding). |
| transformer | the neural-network architecture behind modern LLMs. |
| attention | the mechanism that lets a model weigh which words matter to each other. |
| neural network | layers of math that learn patterns from data. |
What you need before starting:
- This is the most theoretical section — helpful but not required to build apps.
- Comfort with basic math (vectors) and Python; NumPy (DA1) helps.
- Curiosity about 'why', not just 'how'.
New to the topic? Read this box, then take the chapters in order — each section is tagged essential → expert so you always know the depth you're at.
Learning objectives
- Define NLP and the classic task families it covers.
- Explain why natural language is hard — ambiguity, context, scale.
- Run the classic preprocessing pipeline: normalize → tokenize → stem/lemmatize → stop-words.
- Know which classic steps LLMs made optional — and which still matter.
- Place this module's arc: rules → statistics → neural → transformers → LLMs.
What NLP is essential
Natural Language Processing is getting computers to work with human language — understanding it, generating it, and transforming it. It's a decades-old field; LLMs are its latest, most general tool. Classic NLP is a catalog of tasks, and it helps to see them because an LLM now does most of them with one model.
| Task family | Example | An LLM does this via… |
|---|---|---|
| Classification | Spam? Sentiment? Topic? | A prompt or structured output (Ch 2, K3) |
| Extraction | Find names, dates, entities (NER) | Schema-constrained extraction (Ch 2) |
| Generation | Summarize, translate, write | Its core capability |
| Retrieval / search | Find relevant documents | Embeddings + RAG (Ch 3) |
| Sequence labeling | Tag each word (POS, chunks) | Rarely needed directly now |
Why language is hard for computers essential
Numbers are precise; language is slippery. The difficulties below are exactly what made NLP a decades-long research field — and what the transformer's context-handling finally tackled at scale.
| Challenge | Example | Why it's hard |
|---|---|---|
| Ambiguity | "I saw her duck" (bird? crouch?) | Same words, multiple meanings — needs context |
| Context dependence | "It" refers to what, three sentences back? | Meaning spans distance (why context windows matter) |
| Variation | "big" / "large" / "huge" / "enormous" | Many surface forms, similar meaning |
| Idiom & implication | "It's raining cats and dogs" | Meaning ≠ the literal words |
| Scale & sparsity | Infinite possible sentences | Can't enumerate; must generalize |
The historical arc essential
NLP moved through four eras, each fixing the previous one's ceiling. This module is a walk up that ladder — knowing where you are on it makes every technique make sense.
This is a timeline, read left to right. Each rising block is an era in how computers handled language, and each one got better than the one before it. Height roughly means "how capable" — the boxes climb as you move right in time.
- Rules (leftmost, shortest) — the earliest approach: humans hand-wrote grammar rules by hand ("if the sentence looks like X, do Y"). Powerful but brittle — you can't write a rule for every sentence.
- Statistical — instead of rules, count words and model probabilities ("how often does this word follow that one?"). This is the era of counting, covered in K2/K3.
- Neural (RNN) — a neural network learns the useful features itself from data instead of being told them. RNNs read a sentence one word at a time, in order (K4).
- Transformers → LLMs (rightmost, tallest) — the breakthrough is attention: the model looks at the whole sentence at once and weighs which words matter to each other. Scaled up, this became the LLMs you use today (K5).
- The line along the bottom is the shared timeline; the small labels under each box (
hand-written,count & probability,learn features,attention) name the one big idea of each era.
In short: Each era hit a ceiling that the next era was invented to break. You are climbing this same ladder as you go through modules K1 → K5.
Lab K1.1 · The classic preprocessing pipeline intermediate
Before statistical/ML models could touch text, they needed it cleaned into consistent units. This pipeline is foundational NLP — and you still use pieces of it for search, classical models, and data cleaning (B1).
This is a pipeline — a conveyor belt. Messy text enters on the left, flows through four cleaning stations (the arrows show the direction), and comes out on the right as a tidy list of tokens (word-chunks) that older NLP models can work with.
- raw text (start) — the original sentence, exactly as a human wrote it, with capitals, punctuation, and filler words.
- normalize — make everything consistent: lowercase the letters and strip punctuation, so "Run", "run", and "run!" all become the same thing.
- tokenize — split the cleaned string into separate units (here, words). Each unit is a token.
- stop-words / stem·lemma — the last station does two jobs: throw away high-frequency filler words ("the", "is"), and shrink each remaining word to a root form ("running" → "run").
- tokens (end) — the clean word list that comes out. This is the input that classic text representations (K2) and models (K3) actually consume.
In short: Follow one word through the belt: "Runners!" → normalize → "runners" → tokenize → a token → stem → "runner". The lab below runs exactly these four steps in Python.
preprocess.pytext = "The Runners were running quickly through the streets!"
# 1 · normalize — lowercase, strip punctuation
norm = "".join(c for c in text.lower() if c.isalnum() or c.isspace())
# 2 · tokenize — split into words (real tools handle edge cases)
tokens = norm.split()
# ['the','runners','were','running','quickly','through','the','streets']
# 3 · remove stop-words — drop low-signal filler
STOP = {"the", "were", "through", "a", "is"}
content = [t for t in tokens if t not in STOP]
# ['runners','running','quickly','streets']
# 4 · stem / lemmatize — reduce to a root form
# stemming (crude): 'runners'->'runner', 'running'->'run' (chops suffixes)
# lemmatizing (smart): 'running'->'run', 'better'->'good' (uses a dictionary)
This is the classic NLP preprocessing pipeline in ~10 lines of plain Python — the same four stations from the diagram above, turning one messy sentence into a clean list of word-roots. There's no library and no AI here; it's deliberately simple so you can see each step happen.
- Start with raw text.
text = "The Runners were running quickly through the streets!"— note the capital letters, the!, and filler words liketheandwere. That's the mess we're cleaning up. - Step 1 · normalize.
text.lower()makes everything lowercase, then the"".join(c for c in ... if c.isalnum() or c.isspace())keeps only letters, digits, and spaces — dropping the!. Result: one clean lowercase string. - Step 2 · tokenize.
norm.split()breaks that string into a list of words wherever there's a space. The comment shows the result — 8 word tokens, including two copies ofthe. - Step 3 · remove stop-words.
STOPis a set of filler words. The line[t for t in tokens if t not in STOP]keeps only words not in that set, droppingthe,were, andthrough. Down to 4 meaningful words. - Step 4 · stem / lemmatize. These last lines are comments (explanation, not runnable code) because doing it properly needs a library. Stemming crudely chops endings (
running→run); lemmatizing uses a dictionary to find the true base (better→good). Stemming is fast but sometimes wrong; lemmatizing is accurate but slower.
What the output means: After steps 1–3 the sentence "The Runners were running quickly through the streets!" becomes the token list ['runners','running','quickly','streets'] — lowercased, de-punctuated, and stripped of filler. Step 4 would then reduce running to its root run.
Try this: Add a word to the STOP set — say "quickly" — and predict how content changes before you run it. This is exactly the kind of aggressive cleaning the warning box later tells you not to do before sending text to an LLM.
| Step | What & why |
|---|---|
| Normalize | Lowercase, strip punctuation/accents so "Run", "run", "run!" match |
| Tokenize | Split into units (words). The hard part — see the note below |
| Stop-words | Drop high-frequency low-meaning words ("the", "is") for count-based models |
| Stemming | Crudely chop suffixes to a root ("running"→"run") — fast, sometimes wrong |
| Lemmatization | Use grammar/dictionary to find the true base ("better"→"good") — accurate, slower |
Tokenization: then vs now intermediate
Splitting text into units sounds trivial and isn't — is "don't" one token or two? What about "New York", URLs, emoji, Chinese (no spaces)? Classic NLP used word-tokenizers with lots of rules. Modern LLMs use subword tokenization (BPE), which sidesteps most of it — and that shift explains real LLM behavior.
token+ization, not the whole word and not individual letters. This subword scheme (BPE — built from scratch in A5) handles any word (even unseen ones) with a fixed vocabulary. It's why you're billed per token (Ch 2/C1), why models sometimes miscount letters in a word, and why token ≠ word. The classic tokenization problem didn't vanish — BPE solved it more generally.What still matters vs what LLMs replaced intermediate
LLMs made much of the classic pipeline optional — but not all of it. Knowing the difference saves you both from re-inventing what the model already does and from throwing away tools that are still the right choice.
| Classic step | Status with LLMs |
|---|---|
| Manual stop-word removal, stemming | Mostly gone — the model handles morphology; don't strip text before sending it |
| Word tokenization rules | Replaced by learned subword (BPE) tokenizers |
| Text normalization / cleaning | Still matters — garbage in, garbage out; clean data before RAG/prompts (B1) |
| Understanding tokens | Essential — drives cost, context limits, chunking (Ch 3, C1) |
| Task framing (classify/extract/etc.) | Essential — it's how you structure prompts & evals (Ch 2, Ch 5) |
When classic NLP still wins advanced
The LLM isn't always the answer. For some jobs, a classic technique is faster, cheaper, and perfectly adequate — the same "match the tool to the task" judgment as model tiering (C1) and build-vs-adopt (I1).
| Reach for classic NLP when… | Reach for an LLM when… |
|---|---|
| Simple keyword search / exact match | Meaning-based search (embeddings, Ch 3) |
| High-volume, latency-critical, tiny budget | The task needs real language understanding |
| A well-defined pattern (regex, rules) | Ambiguity, nuance, or generation is involved |
| You need full determinism & explainability | Flexibility across varied inputs matters more |
Common pitfalls advanced
| Pitfall | Fix |
|---|---|
| Over-preprocessing text before an LLM | Send natural text; clean only real noise |
| Thinking LLMs "read letters/words" | They read subword tokens (A5); token ≠ word |
| Using an LLM for a regex-simple task | Classic tools are cheaper & deterministic |
| Ignoring the token concept | It drives cost, context limits, and chunking |
| Treating classic NLP as obsolete | Its ideas power LLMs; some tools still win |
Exercises advanced
Exercise K1.1 — Run the pipeline
Context: The sharpest way to feel why you don't classically preprocess before an LLM is to run both pipelines on the same text and compare what each keeps.
Your task: Run normalize → tokenize → stopwords → stem on a paragraph, then tokenize the same text with an LLM (subword) tokenizer and compare the units.
Requirements:
- Apply the full classic pipeline by hand or with a library
- Tokenize the identical text with a subword LLM tokenizer
- Note what the classic pipeline discards: case, punctuation, stopwords, word forms
- Observe that the subword tokenizer preserves all of it
💡 Hint: The contrast is the lesson: classic preprocessing throws away signal an LLM would rather keep.
Show what to look for
The classic pipeline drops case, punctuation, and stop-words and mangles word forms — losing signal. The LLM tokenizer keeps it all as subword tokens. That contrast is exactly why you don't classically preprocess before an LLM.
Exercise K1.2 — Classify the task
Context: Naming the classic task family behind a real problem is what lets you reach for the cheapest tool that solves it.
Your task: For five real problems (route a ticket, pull invoice fields, summarize a doc, find similar articles, tag parts of speech), name the classic NLP task family and how an LLM would handle each.
Requirements:
- Map each problem to its classic task family (classification, extraction, summarization, similarity/retrieval, sequence tagging)
- Sketch how an LLM would approach the same task
- Identify at least one task you would not hand to an LLM
- Justify that hold-out on cost, latency, or determinism grounds
💡 Hint: The odd one out is usually the high-volume, deterministic task where a classic method is faster and cheaper.
Exercise K1.3 — Classic or LLM?
Context: The same request can flip from a regex job to an LLM job the moment the target shifts from a fixed pattern to a subjective judgement.
Your task: Argue regex vs LLM for flagging messages that contain an order number of form ABC-12345, then redo the argument for flagging frustrated messages.
Requirements:
- For the fixed format, weigh cost, speed, and reliability of a regex
- Recognize the structured pattern is a clear regex win
- Show the decision flips once the target becomes subjective sentiment
- Explain why fuzzy, contextual meaning is where the LLM earns its cost
💡 Hint: A crisp, enumerable pattern favours regex; open-ended emotional judgement is exactly what a rule cannot capture.
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: The classic NLP pipeline begins by turning messy text into clean units. The very first steps — lowercase, split, strip punctuation — decide what signal survives before any model sees the words.
Your task: Write simple_tokens(text) that lowercases, splits on whitespace, and strips surrounding punctuation, then run it on a messy sentence.
Requirements:
- Lowercase the input before tokenizing
- Split on whitespace rather than a blunt
\W+split - Strip only edge punctuation so a contraction like
isn'tstays intact - Drop any empty tokens from the result
💡 Hint: The choice to strip edge-only punctuation is the whole point — it keeps in-word apostrophes that a naive split would shred.
Show solution
Pure stdlib; no NLTK needed for the basics. Runnable:
import re
def simple_tokens(text):
text = text.lower()
raw = text.split()
toks = [re.sub(r"^[^\w]+|[^\w]+$", "", w) for w in raw] # strip edge punct
return [t for t in toks if t]
print(simple_tokens("The QUICK, brown fox! Isn't it fast?"))
# ['the', 'quick', 'brown', 'fox', "isn't", 'it', 'fast']Note the pipeline choice: stripping only edge punctuation keeps isn't intact, unlike a blunt re.split(r"\W+").
Context: Stopword removal plus frequency counting is the bones of every classic NLP feature: strip the high-frequency, low-signal words and let the salient terms rise to the top.
Your task: Extend the tokenizer to remove a small stopword set, then count the remaining token frequencies and report the most common content words.
Requirements:
- Filter tokens against a small stopword set
- Count frequencies with something like
collections.Counter - Report the top few content terms (e.g.
most_common) - Show the stopwords are gone and the signal words dominate
💡 Hint: Removing stopwords before counting is what makes the salient terms — not the/a/is — win the frequency race.
Show solution
Use collections.Counter over filtered tokens. Runnable:
import re
from collections import Counter
STOP = {"the", "a", "an", "is", "it", "in", "of", "to", "and"}
def tokens(text):
return [re.sub(r"^[^\w]+|[^\w]+$", "", w) for w in text.lower().split()]
def content_freqs(text):
toks = [t for t in tokens(text) if t and t not in STOP]
return Counter(toks)
doc = "The cat sat on the mat. A cat is a cat, and the mat is warm."
print(content_freqs(doc).most_common(3))
# [('cat', 3), ('mat', 2), ('sat', 1)] -- stopwords gone, signal keptRemoving stopwords is a classic-NLP move: it strips high-frequency, low-signal words before counting so the salient terms dominate.
Context: Stemming collapses inflected forms (running/runs/runner) toward one root so they share a single feature — a crude but effective way to shrink a bag-of-words vocabulary.
Your task: Implement a small rule-based (suffix-stripping) stemmer and show it merges related word forms onto a shared stem.
Requirements:
- Strip common suffixes, trying the longest suffix first
- Guard against over-stemming very short stems (keep a minimum stem length)
- Handle a doubled-consonant case so
running→run - Demonstrate that inflections of one word collapse together
💡 Hint: This is a miniature Porter-style stripper; order the suffix rules longest-first so ational is tried before s.
Show solution
A miniature Porter-style suffix stripper — order matters (longest suffix first). Runnable:
def stem(word):
for suf in ("ational", "ization", "ingly", "edly", "ing", "ers", "er", "ed", "es", "s"):
if word.endswith(suf) and len(word) - len(suf) >= 3:
base = word[:-len(suf)]
if suf in ("ing", "ed") and base and base[-1] == base[-2:-1]:
base = base[:-1] # runn -> run (undo doubled consonant)
return base
return word
for w in ["running", "runs", "runner", "jumped", "jumping", "cats"]:
print(f"{w:>8} -> {stem(w)}")
# running/runs/runner -> run(n); jumped/jumping -> jump; cats -> catStemming is crude but effective for classic bag-of-words: it shrinks the vocabulary so inflected forms share one feature. Real stemmers (Porter/Snowball) live in NLTK — labeled needs-libs.
Context: The lesson's core judgement call: classic NLP beats an LLM when you need speed, determinism, cost control, or interpretability. Encoding that as a router makes the trade-off explicit.
Your task: Write pick_method(...) that chooses classic NLP vs an LLM from task constraints — latency budget, need for explanation, daily volume, and whether the meaning is subtle.
Requirements:
- Subtle / open-ended semantics pushes the choice toward the LLM
- A hard latency budget, auditability, or huge volume favours classic NLP
- Return both the recommended method and the reasons that drove it
- When no constraint dominates, suggest prototyping both
💡 Hint: Let the constraint profile decide: reach for classic NLP when determinism, latency, cost, or interpretability dominate; the LLM earns its keep only on subtle meaning.
Show solution
Match the constraint profile to the method the lesson recommends. Runnable:
def pick_method(latency_ms_budget, needs_explanation, volume_per_day, subtle_semantics):
reasons = []
if latency_ms_budget < 20: reasons.append("hard latency budget")
if needs_explanation: reasons.append("must show why (auditable)")
if volume_per_day > 5_000_000: reasons.append("cost at scale")
if subtle_semantics:
return "LLM (subtle meaning / open-ended)", []
if reasons:
return "classic NLP (regex/TF-IDF/linear model)", reasons
return "either -- prototype both", []
print(pick_method(10, True, 10_000_000, subtle_semantics=False))
print(pick_method(500, False, 1000, subtle_semantics=True))The lesson's point: reach for classic NLP when determinism, latency, cost, or interpretability dominate — LLMs earn their keep on subtle, open-ended meaning.
Context: Language is hard largely because of ambiguity — a single spelling can carry several senses. A toy detector over a sense dictionary makes that lexical ambiguity visible.
Your task: Build a detector that flags words carrying more than one sense from a small word→senses map, and report the ambiguous tokens in a sentence.
Requirements:
- Model senses as a word→list-of-senses dictionary
- Flag a token only when it maps to more than one sense
- Normalize tokens (case, trailing punctuation) before lookup
- Report each ambiguous word alongside its competing senses
💡 Hint: This is why bag-of-words plateaus: bank is one feature but two meanings, and only context can disambiguate it.
Show solution
Model ambiguity as a word→senses map; flag tokens with >1 sense. Runnable:
SENSES = {
"bank": ["riverside", "financial institution"],
"bat": ["animal", "sports equipment"],
"spring":["season", "coil", "water source"],
}
def ambiguities(sentence):
out = {}
for w in sentence.lower().split():
w = w.strip(".,!?")
if w in SENSES and len(SENSES[w]) > 1:
out[w] = SENSES[w]
return out
s = "I sat on the bank near the spring watching a bat."
for w, senses in ambiguities(s).items():
print(f"'{w}' is ambiguous: {senses}")This is why bag-of-words alone plateaus: bank is one feature but two meanings — resolving it needs context, which is what embeddings and transformers add.
Context: In production, preprocessing ships as one configurable component so training and serving stay in lockstep. Two divergent copies of this code is the classic cause of train/serve skew.
Your task: Package the pipeline as a small, testable class with toggles for lowercasing, stopwords, and stemming, plus a method that vectorizes text into token counts.
Requirements:
- Expose lowercase / stopwords / stem as explicit constructor knobs
- A single call runs the full tokenize → filter → stem pipeline
- Provide a
vectorizemethod returning a count vector (bag-of-words) - The class is self-contained and reusable as a shared library
💡 Hint: Keep every preprocessing decision behind one interface so the exact same transform runs at train time and at serve time.
Show solution
A small, testable class with explicit knobs — production-shaped. Runnable:
import re
from collections import Counter
class TextPipeline:
def __init__(self, lower=True, stopwords=None, stem=False):
self.lower = lower
self.stop = set(stopwords or [])
self.stem = stem
def _stem(self, w):
for suf in ("ing", "ed", "es", "s"):
if self.stem and w.endswith(suf) and len(w) - len(suf) >= 3:
return w[:-len(suf)]
return w
def __call__(self, text):
if self.lower: text = text.lower()
toks = [re.sub(r"^[^\w]+|[^\w]+$", "", w) for w in text.split()]
toks = [self._stem(t) for t in toks if t and t not in self.stop]
return toks
def vectorize(self, text):
return Counter(self(text))
pipe = TextPipeline(stopwords={"the", "a", "is"}, stem=True)
print(pipe("The cats are running and the dogs are jumping"))
print(pipe.vectorize("cats cats dogs").most_common())Shipping preprocessing as one configurable component keeps training and serving in lockstep — the classic cause of train/serve skew is two divergent copies of this code.
✓ Checkpoint — you can move on when you can…
- Define NLP and name its classic task families.
- Explain why language is hard: ambiguity, context, variation, scale.
- Run the normalize→tokenize→stop-words→stem/lemma pipeline.
- Say which classic steps LLMs replaced and which still matter.
- Place the rules→statistical→neural→transformer arc.
Knowledge check check yourself
The lesson warns against aggressive classic preprocessing (lowercasing, stripping punctuation, removing stop-words) before sending text to an LLM. Why is that a mistake?
Show answer
Why do modern LLMs use subword (BPE) tokenization rather than the word-tokenization rules of classic NLP, and what LLM behaviors does that explain?