NLP Foundations & Text Processing

What NLP is essential

Start with the moment the abstraction leaks. You paste a paragraph into an LLM, ask it to "pull out the invoice number and the due date," and it just… does it. No training, no model file, no rules — it feels like one seamless magic trick. But the day that request returns the wrong date, or costs ten times what you budgeted, or has to run a million times an hour, the single trick splinters into a dozen distinct problems — and each of those problems has a name that predates the LLM by decades. That catalog of names is NLP, and knowing it is what turns "the model did something" into "I know exactly which job I asked it to do and how to check it."

Here's the analogy that makes the field click. Think of NLP as medicine and the tasks below as its specialties — cardiology, dermatology, oncology. For fifty years you saw a different specialist for each complaint: a spam classifier here, a named-entity extractor there, a translation system somewhere else, each a separate model trained on separate data. The LLM is the eerily capable general practitioner who handles most of the specialties from one conversation. That's a genuine revolution — but the specialties didn't disappear. You still diagnose the problem in the old vocabulary ("this is a classification task, that is extraction"), because that framing is what tells you how to prompt it, how to evaluate it, and when to send it to a specialist after all.

The common mistake here is skipping the diagnosis. Beginners treat every text problem as one undifferentiated "ask the AI" blob, then can't explain why it works or fails. If you can't name the task family, you can't pick an eval for it (Ch 5), you can't reason about whether a cheaper tool would do the job, and you can't tell whether a wrong answer is a prompting bug or a genuinely hard case. Naming is not academic bookkeeping — it's the first move of every competent practitioner. So the definition, plainly: Natural Language Processing is getting computers to work with human language — understanding it, generating it, and transforming it. It's a decades-old field; LLMs are its latest, most general tool. Classic NLP is a catalog of tasks, and it helps to see them because an LLM now does most of them with one model.

Why language is hard for computers essential

Here's a question that should nag at you: if computers beat us at chess in 1997, why did fluent conversation take until the 2020s? Chess has more possible games than atoms in the universe — vastly "bigger" than a sentence. The answer is that chess is hard but crisp: fixed board, legal moves, one clear goal. Language is the opposite — it's small but slippery. Every rule has exceptions, every word bends with context, and the "correct" answer depends on who's speaking, to whom, about what, and what everyone already knows. That slipperiness, not size, is why language sat unsolved for half a century. This section is the honest tour of why — and once you feel each difficulty, the entire design of modern LLMs reads as a direct response to it.

The useful mental model is this: a computer is superb with things that are precise and stable — a bank balance means one thing, everywhere, forever. Language is neither. Take the sentence "I saw her duck." Is she lowering her head, or does she own a waterfowl? Same eight letters, two worlds apart, and nothing in the sentence itself decides it — only the surrounding situation does. A computer that reads text as fixed symbols is helpless here in a way it never is with numbers, because the symbol and the meaning have come apart. Everywhere you look in language you find this same gap: one form, many meanings (ambiguity); meaning that lives sentences away (context); many forms for one meaning (variation); meaning that flatly contradicts the literal words (idiom).

The mistake to avoid is thinking these are edge cases you could patch away. Early NLP tried exactly that — hand-write a rule for "duck," another for "bank," another for sarcasm — and drowned, because the exceptions are the language, not a fringe of it. The breakthrough was to stop fighting ambiguity and instead let a model use context to resolve it, which is precisely what "attention" (K5) does: weigh every other word to decide what this one means right now. Read the table below not as a list of annoyances but as the design brief for the transformer — each row is a problem so stubborn that solving it reshaped the whole field. The difficulties below are exactly what made NLP a decades-long research field — and what the transformer's context-handling finally tackled at scale.

The historical arc essential

Why does a history lesson belong in a hands-on AI course? Because the past isn't behind the LLM — it's inside it. Every era of NLP left a fossil in the model you use today: tokenization came from the rules era, "how likely is this next word" is the statistical era's core idea, learned representations came from the neural era, and attention is the transformer era's one big move. When an LLM surprises you — bills per token, miscounts letters, forgets something from earlier in a long chat — the explanation almost always lives in one of these layers. Knowing the arc means you're never reasoning from magic; you're reasoning from a lineage you can name.

The cleanest way to hold this in your head is that each era hit a ceiling, and the ceiling of one era became the founding problem of the next. Rules were precise but couldn't scale — nobody can hand-write a grammar for all of English, so they broke on the first sentence they hadn't anticipated. Statistics fixed brittleness by counting real usage instead of prescribing it, but counting words treats "big" and "large" as unrelated symbols and drowns in the sheer number of possible word combinations. Neural networks fixed that by learning that similar words should have similar representations — but the early ones (RNNs) had to read a sentence strictly left-to-right, one word at a time, and would forget the beginning by the end. Transformers fixed that by letting the model look at the whole sequence at once and weigh what matters. That's not four disconnected techniques; it's one long conversation, each answer exposing the next question.

The trap to sidestep is treating the earlier eras as "obsolete" and skipping them. They're not obsolete — they're the concepts the transformer absorbed and perfected, and the later modules (K2 statistical, K3 classification, K4 neural, K5 transformers) only make sense once you can see the ceiling each was built to break. Think of it like learning arithmetic before calculus: you don't use long division much anymore, but the ideas underneath it never left. This module is a walk up that ladder, and the payoff is that "why does the LLM behave like this?" almost always has an answer you can point to on the timeline below. NLP moved through four eras, each fixing the previous one's ceiling. This module is a walk up that ladder — knowing where you are on it makes every technique make sense.

Lab K1.1 · The classic preprocessing pipeline intermediate

Picture the problem a 1990s model actually faced. It's a spam filter, and it wants to learn that the word "free" signals junk mail. But in the raw inbox that word arrives as "Free", "FREE!", "free,", "free." and " free " — to a computer comparing exact strings, those are five different words, and the signal it needs gets shattered across all of them. The model can't learn anything until someone melts those five variants back into one. That melting-down is preprocessing, and this pipeline is the assembly line that does it: it takes text as humans carelessly type it and forges it into the clean, consistent units a counting model can actually work with.

The analogy that makes each stage obvious is a recycling plant. Mixed rubbish comes in the front; the plant washes it, sorts it into bins, throws out the worthless bits, and crushes each material down to a standard form so it can be reused. Preprocessing does the same four moves to text. Normalize is the wash — lowercase everything and strip punctuation so "Run", "run", and "run!" stop being strangers. Tokenize is the sort — chop the cleaned string into individual units (words) you can count. Stop-word removal is the throwing-out — words like "the" and "is" appear everywhere and so carry almost no signal about what a document is about, so they're discarded. Stemming/lemmatization is the crushing — "running", "runs", and "runner" all get pressed down toward one root so the model sees them as the same feature. Out the far end comes a tidy token list, and everything downstream (K2's representations, K3's classifiers) eats that output.

Now the honest caveat, because this is where the chapter turns. Every one of these steps throws information away on purpose — that was a feature when your model was too weak to use capitalization or punctuation, and it's a bug when your model is an LLM that reads exactly those cues for meaning. So learn this pipeline as history and as a still-useful tool for search and lightweight classical models — but hold the warning in your other hand: the aggressive cleaning you're about to run is precisely what you should not do before an LLM (the warning box later spells out why). Watch what each stage discards; that lost signal is the whole point of the lesson. Before statistical/ML models could touch text, they needed it cleaned into consistent units. This pipeline is foundational NLP — and you still use pieces of it for search, classical models, and data cleaning (B1).

preprocess.pytext = "The Runners were running quickly through the streets!"

# 1 · normalize — lowercase, strip punctuation
norm = "".join(c for c in text.lower() if c.isalnum() or c.isspace())

# 2 · tokenize — split into words (real tools handle edge cases)
tokens = norm.split()
# ['the','runners','were','running','quickly','through','the','streets']

# 3 · remove stop-words — drop low-signal filler
STOP = {"the", "were", "through", "a", "is"}
content = [t for t in tokens if t not in STOP]
# ['runners','running','quickly','streets']

# 4 · stem / lemmatize — reduce to a root form
# stemming (crude): 'runners'->'runner', 'running'->'run'  (chops suffixes)
# lemmatizing (smart): 'running'->'run', 'better'->'good'   (uses a dictionary)

Tokenization: then vs now intermediate

"Just split on spaces" is one of those ideas that sounds finished until you poke it once. Split "don't" on spaces and you get one token — but is that right, or is it hiding a "do" and a "not"? Split "New York" and you get two tokens for one place. Now try a URL, an emoji, a hashtag, or a sentence of Mandarin, which is written with no spaces at all — and the whole "split on spaces" plan collapses. This is the quiet trap of tokenization: it's the very first thing you do to text and the easiest to get subtly wrong, and getting it wrong poisons everything downstream. Every era of NLP had to answer "what is one unit of text?", and the answer kept changing.

Classic NLP answered it with rules — hand-written tokenizers stuffed with special cases for contractions, hyphens, abbreviations, and named entities. It worked, brittly, and it never fully solved the deepest problem: the unknown word. If your tokenizer's vocabulary was built from the training data and a user typed "supercalifragilistic" or a brand-new product name, the model had literally no slot for it — it became a useless "unknown" token, and meaning was lost. Think of it like a printing press with a fixed drawer of whole-word stamps: the day someone needs a word you don't own a stamp for, you're stuck.

Modern LLMs escape the trap with a genuinely clever move: build the units from data, and make them smaller than words. This is subword tokenization (BPE), and the analogy is switching from whole-word stamps to an alphabet of common fragments — token, ization, ing, ##s. With a fixed, modest set of fragments you can spell any word, including ones you've never seen, by gluing pieces together. No more "unknown" cliff. But this efficiency is why several LLM behaviours that puzzle beginners suddenly make sense: you're billed per token (fragment), not per word, so verbose text costs more; the model sometimes miscounts the letters in a word because it never saw the letters, only the chunks; and "token ≠ word" is why your 1,000-word prompt might be 1,300 tokens. The old tokenization problem didn't disappear — BPE just answered it in a way that scales. Splitting text into units sounds trivial and isn't — is "don't" one token or two? What about "New York", URLs, emoji, Chinese (no spaces)? Classic NLP used word-tokenizers with lots of rules. Modern LLMs use subword tokenization (BPE), which sidesteps most of it — and that shift explains real LLM behavior.

What still matters vs what LLMs replaced intermediate

This is the section that stops you from looking like someone who learned NLP from a blog post in 2015. When LLMs arrived, two opposite mistakes appeared, and both are expensive. The first is the museum-keeper who lovingly runs the full classic pipeline — lowercase, de-punctuate, strip stop-words, stem — on text before feeding it to an LLM, quietly destroying the exact signals the model was built to read. The second is the demolition crew who declares everything before transformers obsolete and reaches for a frontier model to check whether a string matches ABC-12345 — paying LLM prices and latency for a job a one-line regex does perfectly. This section is the map that keeps you out of both ditches.

The clean way to think about it: ask of each classic step, "does the LLM already do this better, or is this still my job?" Some steps the model absorbed — it handles morphology internally, so stemming is pointless; it uses learned subword tokens, so your hand-written tokenizer is dead weight. Those, you delete. But other steps live outside the model and are as important as ever: cleaning genuine garbage (mojibake, boilerplate, duplicated headers) still follows garbage-in-garbage-out, because a confused input confuses even a brilliant model; and understanding tokens and framing the task aren't preprocessing at all — they're how you control cost, fit the context window, chunk documents for RAG, and design your evals. The table below sorts every classic step into "the model handles this now" versus "still on you," and the distinction is worth memorising because it's a question you'll answer on the job constantly.

The analogy that sticks: the LLM is an expert reader, not a document scanner. You wouldn't hand an expert a photocopy with every third word blacked out and the capital letters removed and expect a good summary — yet that's what over-preprocessing does. You would hand them a clean, readable original. So your job shifts from "grind the text into uniform mush" to "hand over faithful, natural text and spend your energy on framing, tokens, and evals instead." Knowing exactly which old habit to drop and which to keep is the difference between fighting the model and working with it. LLMs made much of the classic pipeline optional — but not all of it. Knowing the difference saves you both from re-inventing what the model already does and from throwing away tools that are still the right choice.

When classic NLP still wins advanced

Here's a scenario that separates engineers from enthusiasts. You're asked to flag every support message containing an order number of the form ABC-12345. It's tempting to reach for the LLM — it's what you've been learning — but pause and do the arithmetic: at ten million messages a day, an API call each is a fortune, adds hundreds of milliseconds of latency per message, and gives a probabilistic answer to a question that has one exact right answer. A regex does the same job in microseconds, for free, deterministically, and you can prove to an auditor exactly why it fired. Using the LLM here isn't sophisticated — it's a rookie move that torches budget on a solved problem. This section is about the judgment that avoids it.

The principle is match the tool to the shape of the task, not to what's fashionable — the same discipline as choosing a model tier (C1) or deciding build-vs-adopt (I1). And the shape that tells you "classic wins" is recognizable: the task is a crisp, enumerable pattern (a fixed format, an exact keyword, a rule you could write down), or a hard non-functional constraint dominates — a brutal latency budget, a tiny per-item cost target at massive volume, or a legal need for full determinism and explainability. In all of those, a regex, a lookup table, or a small linear model isn't the compromise; it's the better engineering. The LLM's expensive superpower — flexible understanding of fuzzy, open-ended meaning — is wasted on a problem that was never fuzzy.

The mistake to guard against is the mirror image, too, so hold both. Flip the same order-number task to "flag frustrated customers" and the whole calculus inverts: frustration has no regex, no enumerable pattern, no fixed spelling — it's exactly the subtle, contextual judgment the LLM was built for, and now a rule-based approach is the naive choice. The skill isn't loyalty to either camp; it's reading which situation you're in. The rule of thumb worth carrying: if you can write the pattern down, write the pattern down; if you can only describe it by example and nuance, that's where the LLM earns its cost. The table below lines up the two columns so the tell-tale signs sit side by side. The LLM isn't always the answer. For some jobs, a classic technique is faster, cheaper, and perfectly adequate — the same "match the tool to the task" judgment as model tiering (C1) and build-vs-adopt (I1).

Common pitfalls advanced

Every mistake in this table has the same root cause, and naming it once saves you from all five. The root cause is applying a habit from the wrong era — treating an LLM as if it were a fragile 1990s counting model, or treating a regex-simple job as if it needed a frontier brain. Notice that these aren't exotic bugs; they're the comfortable, plausible-sounding defaults that a smart person reaches for precisely because they worked in a neighbouring context. That's what makes them pitfalls rather than obvious errors: each one feels reasonable in the moment.

Read the list as a quick pre-flight check you run whenever you touch text. Two of the rows — over-preprocessing and "the model reads letters/words" — come straight from mistaking the transformer for a bag-of-words model; the middle row comes from forgetting that classic tools still exist; and the last two come from underrating the token concept and the classic-NLP toolkit you just learned. If you internalised the earlier sections, none of these should surprise you — the value here is having them in one place so you can catch yourself before the mistake ships, not in the post-mortem. The fix column is the corrected instinct to build in its place.

Exercises advanced

import re

def simple_tokens(text):
    text = text.lower()
    raw = text.split()
    toks = [re.sub(r"^[^\w]+|[^\w]+$", "", w) for w in raw]  # strip edge punct
    return [t for t in toks if t]

print(simple_tokens("The QUICK, brown fox! Isn't it fast?"))
# ['the', 'quick', 'brown', 'fox', "isn't", 'it', 'fast']

import re
from collections import Counter

STOP = {"the", "a", "an", "is", "it", "in", "of", "to", "and"}

def tokens(text):
    return [re.sub(r"^[^\w]+|[^\w]+$", "", w) for w in text.lower().split()]

def content_freqs(text):
    toks = [t for t in tokens(text) if t and t not in STOP]
    return Counter(toks)

doc = "The cat sat on the mat. A cat is a cat, and the mat is warm."
print(content_freqs(doc).most_common(3))
# [('cat', 3), ('mat', 2), ('sat', 1)] -- stopwords gone, signal kept

def stem(word):
    for suf in ("ational", "ization", "ingly", "edly", "ing", "ers", "er", "ed", "es", "s"):
        if word.endswith(suf) and len(word) - len(suf) >= 3:
            base = word[:-len(suf)]
            if suf in ("ing", "ed") and base and base[-1] == base[-2:-1]:
                base = base[:-1]        # runn -> run (undo doubled consonant)
            return base
    return word

for w in ["running", "runs", "runner", "jumped", "jumping", "cats"]:
    print(f"{w:>8} -> {stem(w)}")
# running/runs/runner -> run(n); jumped/jumping -> jump; cats -> cat

def pick_method(latency_ms_budget, needs_explanation, volume_per_day, subtle_semantics):
    reasons = []
    if latency_ms_budget < 20:      reasons.append("hard latency budget")
    if needs_explanation:           reasons.append("must show why (auditable)")
    if volume_per_day > 5_000_000:  reasons.append("cost at scale")
    if subtle_semantics:
        return "LLM (subtle meaning / open-ended)", []
    if reasons:
        return "classic NLP (regex/TF-IDF/linear model)", reasons
    return "either -- prototype both", []

print(pick_method(10, True, 10_000_000, subtle_semantics=False))
print(pick_method(500, False, 1000, subtle_semantics=True))

SENSES = {
    "bank":  ["riverside", "financial institution"],
    "bat":   ["animal", "sports equipment"],
    "spring":["season", "coil", "water source"],
}

def ambiguities(sentence):
    out = {}
    for w in sentence.lower().split():
        w = w.strip(".,!?")
        if w in SENSES and len(SENSES[w]) > 1:
            out[w] = SENSES[w]
    return out

s = "I sat on the bank near the spring watching a bat."
for w, senses in ambiguities(s).items():
    print(f"'{w}' is ambiguous: {senses}")

import re
from collections import Counter

class TextPipeline:
    def __init__(self, lower=True, stopwords=None, stem=False):
        self.lower = lower
        self.stop = set(stopwords or [])
        self.stem = stem
    def _stem(self, w):
        for suf in ("ing", "ed", "es", "s"):
            if self.stem and w.endswith(suf) and len(w) - len(suf) >= 3:
                return w[:-len(suf)]
        return w
    def __call__(self, text):
        if self.lower: text = text.lower()
        toks = [re.sub(r"^[^\w]+|[^\w]+$", "", w) for w in text.split()]
        toks = [self._stem(t) for t in toks if t and t not in self.stop]
        return toks
    def vectorize(self, text):
        return Counter(self(text))

pipe = TextPipeline(stopwords={"the", "a", "is"}, stem=True)
print(pipe("The cats are running and the dogs are jumping"))
print(pipe.vectorize("cats cats dogs").most_common())

Knowledge check check yourself