Feature Engineering & Text Representation

Why text becomes vectors essential

Start with a frustration you've probably felt. You search a help centre for "my laptop won't turn on," and it returns nothing useful — even though there's a perfect article titled "troubleshooting a machine that won't boot." Same problem, different words, zero matches. That failure isn't the search engine being dumb; it's the search engine comparing spellings instead of meanings. Everything in this chapter is about closing that gap, and it all begins with one unavoidable fact: a computer has no idea what a word is.

Here's the mental model to hold onto. A model is a giant calculator. You can feed a calculator 3.7 and -0.2; you cannot feed it the word "diabetes." So before any learning, classifying, or retrieving can happen, something has to translate text into numbers — and not just any numbers, but numbers arranged so that math on them means something. That translation step is called representation, and it is the single most consequential design choice in an NLP system. Get it right and a simple model shines; get it wrong and no amount of model cleverness can recover the meaning you threw away.

The common beginner mistake is to treat representation as a boring preprocessing detail — "just tokenize and move on." In reality it sets the ceiling on everything downstream. Think of it like translating a poem into another language before anyone reads it: if the translation is flat, the most brilliant reader in the world still can't feel the poem. The representation is that translation, and this whole chapter is a ladder of ever-better translators — from crude word-counting (bag-of-words) to importance-weighting (TF-IDF) to the learned embeddings that finally capture meaning itself. Read the sections in order; each one exists to fix a specific flaw in the one before it, so the "why does this next thing exist?" answer is always "to repair what just broke."

Machine-learning models — from a logistic-regression classifier to a transformer — do math on numbers. Text is symbols. So the first job of any NLP system is representation: mapping text to a vector of numbers. The quality of that mapping caps everything downstream — a representation that loses meaning can't be rescued by a fancy model.

Lab K2.1 · Bag-of-words: counting essential

Imagine tipping a sentence into a paper bag and shaking it. The words are all still in there, but the order is gone — you can only pull them out and tally how many of each you got. That deliberately crude picture is bag-of-words, the oldest and simplest text-to-vector recipe, and it's worth taking seriously precisely because it's so blunt: everything cleverer in this chapter is a reaction to what this loses.

The recipe has two steps, and the honest way to understand it is to see why each step is there. First you fix a vocabulary — one master list of every word you'll pay attention to, in a fixed order. That order is the whole point: position 3 in the list will always mean the same word, so every document can be described against the same yardstick. Second, each document becomes a row of counts: "how many times does vocabulary-word-1 appear? word-2? word-3?" and so on. A sentence stops being letters and becomes a fixed-length list of numbers a model can finally do arithmetic on.

Why does this simple thing work at all? Because for many real tasks, which words show up is genuinely informative. A document stuffed with "refund," "charge," and "invoice" is probably about billing, no matter the word order; one full of "fever," "dose," and "symptom" is about health. Topic-level jobs — spam filtering, routing a ticket to the right team, coarse document sorting — lean on exactly this "what words are present" signal, and bag-of-words captures it cheaply with no training whatsoever. Don't let its age fool you into skipping it; it is still a perfectly good baseline you should reach for before anything fancier.

But feel the two things it throws away, because the rest of the chapter is built on that loss. It's order-blind: "dog bites man" and "man bites dog" collapse to the identical vector, so any meaning that lives in word order is simply gone. And it's meaning-blind: every word is its own separate slot, so "great" and "excellent" are treated as no more related than "great" and "toaster." On top of that the vectors are sparse (a row is almost all zeros, because any one document uses only a tiny slice of a big vocabulary) and huge (as wide as the entire vocabulary — tens of thousands of columns is normal). Keep those four words — order-blind, meaning-blind, sparse, huge — in mind as you read the diagram: TF-IDF will attack "huge and dominated by common words," and embeddings will finally attack "meaning-blind."

The simplest representation: build a vocabulary, then represent each document by how many times each word appears. Order is thrown away — hence "bag."

bow.pyfrom sklearn.feature_extraction.text import CountVectorizer

docs = ["the cat sat", "the dog ran", "the cat ran fast"]
vec = CountVectorizer()
X = vec.fit_transform(docs)     # sparse matrix: docs × vocabulary
print(vec.get_feature_names_out())   # ['cat' 'dog' 'fast' 'ran' 'sat' 'the']
print(X.toarray())              # each row = one doc's word counts

Lab K2.2 · TF-IDF: weighting by importance essential

Bag-of-words has a loud-neighbour problem. Count the words in almost any English document and the winners are "the," "of," "and," "a" — filler that appears everywhere and tells you nothing about what the document is about. Meanwhile the one word that actually pins down the topic — "diabetes," "refund," "kubernetes" — might appear just once or twice and gets drowned out. If you rank documents by raw counts, you're mostly ranking by how much grammatical glue they contain. TF-IDF is the fix, and it's one of those ideas that feels obvious the moment you see it.

The insight is that a word is informative when it's frequent here but rare everywhere else. Think of how you'd describe a friend in a crowd: you wouldn't say "has two eyes" (everyone does — useless) — you'd say "bright red jacket" (rare, therefore identifying). TF-IDF turns that instinct into a number by multiplying two factors. Term frequency (TF) asks "how much does this word show up in this document?" — the more, the more this document is about it. Inverse document frequency (IDF) asks "how rare is this word across the whole collection?" — and crucially it shrinks the weight of words that appear in many documents. Multiply them and a word only scores high when it's both prominent here and distinctive overall.

The most elegant consequence is worth pausing on: TF-IDF discovers stopwords by itself, from the data. A word like "the" appears in essentially every document, so its IDF is computed as roughly log(N/N) = log(1) = 0 — and anything times zero is zero. You never handed it a list of boring words to ignore; the math notices that ubiquitous words carry no distinguishing signal and mutes them automatically. That's why the same TF-IDF recipe works on medical notes, legal filings, or DevOps runbooks without you curating a stopword list for each — the "importance" is always defined relative to the collection in front of it.

A caution that trips people up: TF-IDF makes the counts smarter, but it does not make them understand. It's still, at heart, bag-of-words — one dimension per word, order discarded, and no clue that two different words can mean the same thing. It solves "common words dominate"; it does not solve "car and automobile are unrelated." That remaining wall is the whole subject of the next section, and it's the reason embeddings had to be invented. For now, notice the code below is nearly identical to the bag-of-words lab — same call, same matrix shape — because the only thing that changed is what each number means.

In bag-of-words, "the" (in every doc) counts as much as "diabetes" (in one). TF-IDF fixes that: weight each word by how often it appears in this doc (term frequency) times how rare it is across all docs (inverse document frequency). Common words get downweighted; distinctive words shine.

tfidf.pyfrom sklearn.feature_extraction.text import TfidfVectorizer

vec = TfidfVectorizer()
X = vec.fit_transform(docs)     # same shape as BoW, but weighted
# "the" (in every doc) → near-zero weight; a rare distinctive word → high weight

The gap TF-IDF can't cross essential

Here is the concrete failure that ended the counting era. A customer types "how do I recover my login" into your support search. The perfect answer article says "steps to reset your password." Between those two sentences there is not a single shared content word — recover ≠ reset, login ≠ password — so TF-IDF scores them at essentially zero similarity. The system confidently returns nothing, or worse, something irrelevant that happened to share the word "how." The intent is identical; the vocabulary is not; and TF-IDF can only see vocabulary.

This has a name — vocabulary mismatch — and it isn't a bug you can tune away, it's baked into the whole approach. Remember that in bag-of-words and TF-IDF, every word gets its own private dimension. "Car" is axis 1, "automobile" is axis 2, and to the geometry those two axes are perpendicular — as unrelated as "car" and "banana." The representation literally has no place to record that two words are near-synonyms, because that fact lives between words, and a one-slot-per-word scheme only ever stores facts about single words. No amount of clever weighting rescues you: you can weight "automobile" perfectly and it still shares no axis with "car."

The instinctive patches all disappoint, which is why this deserves its own section. You could hand-build a thesaurus and expand every query with synonyms — but language has millions of words, synonymy is fuzzy and context-dependent ("bank" the riverside vs "bank" the vault), and you'd be maintaining that list forever. You could stem or lemmatize — but that only collapses spelling variants ("running"→"run"), never true synonyms. The honest conclusion the field reached: matching on the surface form of words has a hard ceiling, and to break through you have to represent meaning directly, so that "car" and "automobile" end up close by construction rather than by a lookup table. That is exactly the leap the diagram below sets up and the embeddings section delivers — from matching spellings to matching meanings.

TF-IDF weights words better, but it's still word-matching: every word is its own dimension, and it has no idea that "car" and "automobile" mean the same thing. Search for "automobile" and a document full of "car" scores zero. This vocabulary mismatch is the wall classic representations hit — and the reason embeddings were a revolution.

Lab K2.3 · Embeddings: meaning as geometry intermediate

This is the idea that quietly powers everything you've built in this course. Every RAG system, every semantic search box, every "find similar" feature rests on one move: representing meaning as a location in space. It sounds abstract, so ground it with a map. Imagine a vast map where every word and sentence is a pin, and the rule is simply things that mean similar things get pinned close together. "Car" and "automobile" sit almost on top of each other; "car" and "banana" are on opposite sides of the map. Once you have that map, "are these two texts about the same thing?" becomes the delightfully concrete question "how far apart are their pins?" — pure geometry, which computers are superb at.

An embedding is exactly that: the coordinates of a text's pin, written as a dense vector of a few hundred to a few thousand real numbers. Contrast it with the sparse vectors you just met. TF-IDF gives you a 30,000-long vector that's almost all zeros, with one dimension nailed to each specific word. An embedding gives you maybe 384 or 1,536 numbers, all of them nonzero and meaningful, where no single number stands for a single word — instead meaning is smeared across the whole vector. That's the crucial architectural shift: giving up "one dimension = one word" is precisely what buys room to place synonyms nearby, because the same handful of dimensions can encode "vehicle-ness" for both "car" and "automobile."

The part that surprises everyone is where the coordinates come from: nobody hand-places the pins. The vectors are learned from data, and the training trick is beautifully simple in spirit — a model reads mountains of text and learns to predict a word from the company it keeps ("you shall know a word by the friends it keeps"). Words that show up in the same kinds of contexts — "doctor," "nurse," "hospital" — drift together on the map; words that never co-occur drift apart. Meaning was never programmed in; it fell out of the statistics of real usage. That is why the classic party trick works: the geometry ends up so structured that king − man + woman lands right next to queen. Directions on the map came to encode concepts like gender and royalty, purely from reading.

One honest caveat before the table, because it matters later: the earliest embeddings (word2vec, GloVe) gave each word one fixed pin regardless of context, so "bank" got a single blurry location caught between rivers and money. Modern embeddings — the ones your APIs return — are contextual: they read the surrounding sentence and place "bank" differently in "river bank" versus "savings bank." Same core idea (meaning as a location, compared by distance), just far sharper. Hold that thread; the "modern embeddings" section picks it up, and K5 reveals the transformer machinery that produces those context-aware coordinates.

An embedding maps each token/word/document to a dense vector (a few hundred to a few thousand real numbers) learned from data so that similar meanings land near each other. Instead of one dimension per word, meaning is spread across the whole vector — and relationships emerge as geometry.

Comparing by meaning: cosine similarity intermediate

You have a map of meaning — now you need a ruler. Embeddings turned every text into a point in space, but "these two points are near each other" is only useful if you can put a number on "near." The obvious ruler is straight-line distance, and it's exactly the wrong one for text. Here's why, and it's a mistake worth understanding rather than memorizing: a long document naturally has a bigger vector (more words push its coordinates further from the origin) than a short one on the identical topic. Straight-line distance would then rank the short focused answer as "less similar" purely for being short — penalizing brevity, which is absurd for search.

The fix is to stop measuring how far the pins are and start measuring which way they point. Picture each vector as an arrow from the origin. Cosine similarity looks only at the angle between two arrows and ignores their length entirely. Two texts about the same thing point the same direction — small angle, high similarity — whether one is a tweet and the other a 10-page report. That length-independence is the single property that makes cosine the default text-similarity metric across the entire industry; it's what lets a one-line FAQ answer fairly beat a rambling article for the same query.

The formula packs this into one line: divide the dot product of the two vectors by the product of their lengths (their norms). Don't let the notation intimidate — the dot product measures how much the arrows overlap, and dividing by both lengths is exactly the step that cancels magnitude so only direction survives. That division is not a technicality you can skip; forget it and you're back to being fooled by document length. The result lands on a clean, fixed scale you can reason about instantly: 1.0 means the arrows point the same way (same meaning), 0 means they're at right angles (unrelated), and −1 means they point opposite ways (opposite meaning). In practice with real embeddings you'll mostly see scores between roughly 0 and 1, and "rank by cosine, keep the top few" is semantic search — the precise operation the code below performs and the beating heart of every RAG retriever in this course.

Once texts are vectors, "how similar in meaning?" becomes "how close are the vectors?" The standard measure is cosine similarity — the angle between vectors, ignoring length. This is the exact operation under semantic search and RAG retrieval.

similarity.pyimport numpy as np

def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
# 1.0 = same direction (same meaning), 0 = unrelated, -1 = opposite

# in practice: embed a query and your docs, rank docs by cosine to the query
q = embed("how do I reset my password")
scores = [(d, cosine(q, embed(d))) for d in docs]
top = sorted(scores, key=lambda s: s[1], reverse=True)[:3]

From classic to modern embeddings intermediate

When you call an embedding API today, you're standing at the top of a twenty-year staircase — and it helps to know which step you're on. The reason this matters practically: developers who don't know the lineage assume all "embeddings" are the same thing, then get baffled when a word2vec vector and a modern API vector behave completely differently on their task. They're the same idea at wildly different power levels, and confusing the two leads to bad tool choices.

The staircase has three steps, and each one fixed a real limitation of the last. Static word embeddings (word2vec, GloVe, ~2013) were the breakthrough: one fixed vector per word, learned from co-occurrence, good enough to make "king − man + woman ≈ queen" work. Their fatal flaw was exactly the "bank" problem — a single word with two meanings got one averaged, muddy pin, because the model had no way to look at context. Contextual embeddings (BERT-style, from the transformers you'll meet in K5) fixed that by computing a word's vector fresh for each sentence: "bank" near "river" lands somewhere different than "bank" near "deposit." Meaning finally became situational, the way it actually is in language. Sentence and document embeddings (today's retrieval-tuned models and APIs) took the final step: instead of one vector per word, they hand you one vector for a whole passage, deliberately trained so that questions land near their answers — purpose-built for search and RAG rather than as a by-product.

So the honest summary is: the concept never changed — meaning as a location, compared by cosine — but each generation reads more context and packs more of it into the vector. Don't fall for "newer is always better," though. A tiny static embedding can be the right call when you need speed and have no GPU, just as TF-IDF still beats embeddings for exact-term matching. What the ladder buys you is options: knowing where each rung sits lets you pick the cheapest representation that clears your task's bar, which is the same match-the-tool-to-the-job discipline running through this whole course.

The idea is the same across eras; the quality climbed dramatically. Knowing the lineage tells you what today's embedding APIs actually give you.

Choosing a representation advanced

Common pitfalls advanced

Exercises advanced

from collections import Counter

def build_vocab(corpus):
    vocab = sorted({w for doc in corpus for w in doc.split()})
    return {w: i for i, w in enumerate(vocab)}

def bow(doc, vocab):
    v = [0] * len(vocab)
    for w, c in Counter(doc.split()).items():
        if w in vocab:
            v[vocab[w]] = c
    return v

corpus = ["cat sat mat", "dog sat log", "cat and dog"]
vocab = build_vocab(corpus)
print("vocab:", list(vocab))
for d in corpus:
    print(bow(d, vocab), d)

import math
from collections import Counter

def tfidf(corpus):
    docs = [doc.split() for doc in corpus]
    N = len(docs)
    df = Counter()
    for d in docs:
        for w in set(d):
            df[w] += 1
    out = []
    for d in docs:
        tf = Counter(d)
        n = len(d)
        out.append({w: (tf[w] / n) * math.log(N / df[w]) for w in tf})
    return out

corpus = ["the cat sat", "the dog sat", "the cat ran"]
for row in tfidf(corpus):
    print({w: round(s, 3) for w, s in sorted(row.items())})
# 'the' -> 0.0 everywhere (df==N, idf==0); distinctive words score higher

import math
from collections import Counter

def cosine(a, b):
    keys = set(a) | set(b)
    dot = sum(a.get(k, 0) * b.get(k, 0) for k in keys)
    na = math.sqrt(sum(v*v for v in a.values()))
    nb = math.sqrt(sum(v*v for v in b.values()))
    return dot / (na * nb) if na and nb else 0.0

def vec(text):
    return Counter(text.split())

query = vec("cat food")
docs = {"d1": vec("cat food cat treats"),
        "d2": vec("dog food"),
        "d3": vec("premium cat food for cats")}
for name, sim in sorted(((n, cosine(query, v)) for n, v in docs.items()),
                        key=lambda x: -x[1]):
    print(f"{name}: {sim:.3f}")

import math
from collections import Counter

def cosine(a, b):
    keys = set(a) | set(b)
    dot = sum(a.get(k,0)*b.get(k,0) for k in keys)
    na = math.sqrt(sum(v*v for v in a.values()))
    nb = math.sqrt(sum(v*v for v in b.values()))
    return dot/(na*nb) if na and nb else 0.0

print("TF-IDF/BoW cosine('car','automobile') =",
      cosine(Counter(["car"]), Counter(["automobile"])))   # 0.0 -- no shared token

# toy embedding: both map onto a 'vehicle' latent dimension
EMB = {"car":        [0.9, 0.1, 0.0],
       "automobile": [0.88, 0.12, 0.0],
       "banana":     [0.0, 0.1, 0.95]}
def cos_vec(a, b):
    dot = sum(x*y for x, y in zip(a, b))
    na = math.sqrt(sum(x*x for x in a)); nb = math.sqrt(sum(y*y for y in b))
    return dot/(na*nb)
print("embedding cosine('car','automobile') =", round(cos_vec(EMB["car"], EMB["automobile"]), 3))
print("embedding cosine('car','banana')     =", round(cos_vec(EMB["car"], EMB["banana"]), 3))

def choose_representation(need):
    rules = [
        ("exact keyword match",   "TF-IDF / BM25 (sparse, exact terms)"),
        ("interpretable features","TF-IDF (each dim is a word)"),
        ("tiny labeled dataset",  "TF-IDF + linear model (few params)"),
        ("synonyms / paraphrase", "dense embeddings (semantic)"),
        ("semantic search",       "dense embeddings + vector index"),
        ("cross-lingual",         "multilingual embeddings"),
    ]
    for key, rec in rules:
        if key in need:
            return rec
    return "prototype both and measure"

for n in ["exact keyword match", "semantic search", "synonyms / paraphrase"]:
    print(f"{n:>24} -> {choose_representation(n)}")

import math
from collections import Counter

class TfidfIndex:
    def __init__(self, corpus):
        self.docs = [d.split() for d in corpus]
        self.raw = corpus
        N = len(self.docs)
        df = Counter(w for d in self.docs for w in set(d))
        self.idf = {w: math.log((1 + N) / (1 + df[w])) + 1 for w in df}
        self.vecs = [self._vec(d) for d in self.docs]
    def _vec(self, toks):
        tf = Counter(toks); n = len(toks) or 1
        return {w: (tf[w]/n) * self.idf.get(w, 0.0) for w in tf}
    def _cos(self, a, b):
        keys = set(a) | set(b)
        dot = sum(a.get(k,0)*b.get(k,0) for k in keys)
        na = math.sqrt(sum(v*v for v in a.values()))
        nb = math.sqrt(sum(v*v for v in b.values()))
        return dot/(na*nb) if na and nb else 0.0
    def search(self, query, k=3):
        q = self._vec(query.split())
        scored = [(self._cos(q, v), self.raw[i]) for i, v in enumerate(self.vecs)]
        return sorted(scored, reverse=True)[:k]

idx = TfidfIndex(["cheap flights to paris", "best pizza in paris",
                  "cheap car rental", "luxury hotels paris"])
for score, doc in idx.search("cheap paris trip", k=3):
    print(f"{score:.3f}  {doc}")

Knowledge check check yourself