Feature Engineering & Text Representation
A model computes on numbers, not words — so the central NLP question is: how do you turn text into vectors? This chapter walks the ladder from counting words (bag-of-words, TF-IDF) to learned embeddings that capture meaning — the leap that makes semantic search and, ultimately, transformers possible.
Learning objectives
- Explain why text must become vectors, and what a good representation preserves.
- Build bag-of-words and TF-IDF representations and know their limits.
- Explain the leap to embeddings — dense vectors that encode meaning.
- Use cosine similarity to compare texts by meaning.
- Connect classic embeddings to the ones powering RAG and LLMs today.
Why text becomes vectors essential
Machine-learning models — from a logistic-regression classifier to a transformer — do math on numbers. Text is symbols. So the first job of any NLP system is representation: mapping text to a vector of numbers. The quality of that mapping caps everything downstream — a representation that loses meaning can't be rescued by a fancy model.
Lab K2.1 · Bag-of-words: counting essential
The simplest representation: build a vocabulary, then represent each document by how many times each word appears. Order is thrown away — hence "bag."
This picture shows the core trick of bag-of-words: turn a sentence into a row of numbers by counting words. A computer can't do math on the word "cat", but it can on the number 1.
- The top line is the vocabulary — a fixed, ordered list of every word we care about:
[cat, dog, sat, ran]. Position 1 always means "cat", position 2 always means "dog", and so on. - Each sentence becomes one row of counts, one number per vocabulary word.
"cat sat"→[1, 0, 1, 0]: one "cat", zero "dog", one "sat", zero "ran". - Read a row left-to-right against the vocabulary above it: a
1means the word is present (once), a0means absent. Green cells are the hits. - Notice the word order is gone — that's why it's called a "bag". "cat sat" and "sat cat" would give the exact same row of numbers.
In short: A document becomes a vector as long as the vocabulary, mostly full of zeros. That "mostly zeros" (sparse) and "one dimension per word" (meaning-blind) is exactly what the next steps — TF-IDF and embeddings — set out to improve.
bow.pyfrom sklearn.feature_extraction.text import CountVectorizer
docs = ["the cat sat", "the dog ran", "the cat ran fast"]
vec = CountVectorizer()
X = vec.fit_transform(docs) # sparse matrix: docs × vocabulary
print(vec.get_feature_names_out()) # ['cat' 'dog' 'fast' 'ran' 'sat' 'the']
print(X.toarray()) # each row = one doc's word counts
This is bag-of-words in real code. scikit-learn's CountVectorizer does the whole job for you: it reads your documents, builds the vocabulary, and fills in the word counts — the diagram above, automated.
docsis a plain list of three short strings. These are the "documents" we want to turn into number rows.vec = CountVectorizer()creates the counter.vec.fit_transform(docs)does two things at once: fit (look at all the docs and decide the vocabulary) and transform (count each word in each doc). The resultXis a matrix with one row per document and one column per vocabulary word.vec.get_feature_names_out()shows the vocabulary in column order — here['cat' 'dog' 'fast' 'ran' 'sat' 'the'], sorted alphabetically, not in the order you wrote them.X.toarray()turns the compact "sparse" matrix into a plain grid of numbers so you can read it. Each row lines up with the vocabulary above.
What the output means: You see the vocabulary list, then a grid of counts. For "the cat sat" the row reads 1 0 0 0 1 1 — one "cat", one "sat", one "the", zero of the rest.
Try this: Add a doc like "the dog sat" to docs and re-run. Watch a new row appear; the vocabulary only grows if you introduce a brand-new word.
Lab K2.2 · TF-IDF: weighting by importance essential
In bag-of-words, "the" (in every doc) counts as much as "diabetes" (in one). TF-IDF fixes that: weight each word by how often it appears in this doc (term frequency) times how rare it is across all docs (inverse document frequency). Common words get downweighted; distinctive words shine.
Requires: pip install scikit-learn
tfidf.pyfrom sklearn.feature_extraction.text import TfidfVectorizer
vec = TfidfVectorizer()
X = vec.fit_transform(docs) # same shape as BoW, but weighted
# "the" (in every doc) → near-zero weight; a rare distinctive word → high weight
Same idea as bag-of-words, but smarter about which words matter. TfidfVectorizer still produces one number per word per doc — but instead of a raw count, each number is a weight that rewards distinctive words and punishes filler like "the".
vec = TfidfVectorizer()andvec.fit_transform(docs)mirror the bag-of-words code exactly — same call, same shape of result. The only change is what the numbers mean.- Under the hood each number is TF × IDF: how often the word appears in this doc (term frequency) times how rare it is across all docs (inverse document frequency).
- Because "the" appears in every document, its IDF is tiny, so its weight collapses toward zero — it can't dominate anymore. A word that shows up in just one doc gets a high weight, marking it as characteristic of that doc.
What the output means: X has the same rows-and-columns shape as the bag-of-words matrix, but the cells are decimals (weights) instead of whole-number counts. Common words → near 0; distinctive words → larger values.
Try this: Print X.toarray() here too and compare it to the bag-of-words grid. Find the column for "the" and confirm its numbers are much smaller than a rare word's.
| Component | Meaning |
|---|---|
| TF (term frequency) | How often the word appears in this document — more = more relevant here |
| IDF (inverse doc frequency) | How rare the word is across all documents — rarer = more distinctive |
| TF × IDF | High for words frequent here but rare overall — the words that characterize this doc |
The gap TF-IDF can't cross essential
TF-IDF weights words better, but it's still word-matching: every word is its own dimension, and it has no idea that "car" and "automobile" mean the same thing. Search for "automobile" and a document full of "car" scores zero. This vocabulary mismatch is the wall classic representations hit — and the reason embeddings were a revolution.
This picture shows the one big weakness bag-of-words and TF-IDF cannot fix, and what embeddings do instead. Both panels are little maps where closeness means similar meaning.
- Left (red, TF-IDF): "car" and "automobile" sit on two separate axes at right angles. To this method they are unrelated dimensions — as far apart as two words that mean totally different things. It only matches identical words.
- Right (green, embeddings): the same two words are plotted as dots that are close together, because the model learned they mean nearly the same thing.
- The takeaway: TF-IDF matches spellings; embeddings match meanings. Search "car" over a document that only says "automobile" and TF-IDF scores it zero — the "vocabulary mismatch" wall.
In short: Distance on these maps = difference in meaning. Moving from the left picture to the right one — from separate word-axes to nearby meaning-points — is the whole reason embeddings exist.
Lab K2.3 · Embeddings: meaning as geometry intermediate
An embedding maps each token/word/document to a dense vector (a few hundred to a few thousand real numbers) learned from data so that similar meanings land near each other. Instead of one dimension per word, meaning is spread across the whole vector — and relationships emerge as geometry.
| Sparse (BoW / TF-IDF) | Dense (embeddings) | |
|---|---|---|
| Size | Vocabulary-sized (huge, mostly zeros) | Fixed, small (e.g. 384–3072), all meaningful |
| Made by | Counting | Learned from data (a model) |
| Captures | Which words appear | What the text means |
| Synonyms | Unrelated | Close together |
king − man + woman lands near queen. That was the proof that a learned geometry captures real semantic structure — analogy, gender, tense, category — not just word identity. Modern contextual embeddings (from transformers, K5) go further: the vector for "bank" differs in "river bank" vs "savings bank", because it's computed in context.Comparing by meaning: cosine similarity intermediate
Once texts are vectors, "how similar in meaning?" becomes "how close are the vectors?" The standard measure is cosine similarity — the angle between vectors, ignoring length. This is the exact operation under semantic search and RAG retrieval.
Requires: pip install numpy
similarity.pyimport numpy as np
def cosine(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
# 1.0 = same direction (same meaning), 0 = unrelated, -1 = opposite
# in practice: embed a query and your docs, rank docs by cosine to the query
q = embed("how do I reset my password")
scores = [(d, cosine(q, embed(d))) for d in docs]
top = sorted(scores, key=lambda s: s[1], reverse=True)[:3]
Once every text is a vector, "how similar are these two texts?" becomes "how close are their two vectors?" Cosine similarity answers that by measuring the angle between them — this is the exact math behind semantic search and RAG retrieval.
cosine(a, b)takes two vectors.np.dot(a, b)is the dot product; dividing bynp.linalg.norm(a) * np.linalg.norm(b)(their lengths) cancels out size so only direction matters. Two texts pointing the same way score high no matter how long they are.- The scale is fixed: 1.0 = same direction (same meaning), 0 = unrelated (at right angles), -1 = opposite.
embed(...)turns a piece of text into its vector. We embed the query"how do I reset my password"once, then embed each document.scores = [(d, cosine(q, embed(d))) for d in docs]scores every doc against the query, thensorted(..., reverse=True)[:3]keeps the three closest — the top search results.
What the output means: top is a list of the 3 documents most similar in meaning to the query, each paired with its cosine score (closest to 1.0 first). That ranked list is semantic search.
Try this: This is the retrieval core of Chapter 3's RAG. Swap the query string and watch which docs rise to the top — you're ranking by meaning, not by shared keywords.
From classic to modern embeddings intermediate
The idea is the same across eras; the quality climbed dramatically. Knowing the lineage tells you what today's embedding APIs actually give you.
| Generation | Example | Key property |
|---|---|---|
| Static word | word2vec, GloVe | One fixed vector per word (no context — "bank" is one point) |
| Contextual | BERT-style (from transformers, K5) | Vector depends on the sentence — "bank" shifts by context |
| Sentence / document | Modern embedding models & APIs | A whole passage → one vector, tuned for retrieval |
Choosing a representation advanced
| Use… | When… |
|---|---|
| Bag-of-words / TF-IDF | Keyword matching, small/interpretable models, no training budget, the sparse half of hybrid search |
| Embeddings | Semantic search, "find similar meaning", RAG, clustering by topic |
| Both (hybrid) | Production retrieval — keyword precision + semantic recall (A7) |
Common pitfalls advanced
| Pitfall | Fix |
|---|---|
| Expecting BoW/TF-IDF to grasp meaning | They match words; use embeddings for meaning |
| Dismissing TF-IDF as obsolete | Still great for keyword search & hybrid retrieval |
| Using dot product without normalizing | Cosine ignores length; normalize or use cosine directly |
| Assuming synonyms match in TF-IDF | They don't — vocabulary mismatch; that's what embeddings fix |
| Ignoring hybrid search in production | Combine sparse + dense for best retrieval (A7) |
Exercises advanced
Exercise K2.1 — BoW vs TF-IDF
Context: Seeing TF-IDF re-weight a common word versus a distinctive one, side by side with raw counts, is the fastest way to internalize what idf buys you.
Your task: Vectorize five short documents with both CountVectorizer and TfidfVectorizer, then compare the weights for a filler word like the against a distinctive one.
Requirements:
- Vectorize the same docs with counts and with TF-IDF
- Inspect the weight a common word gets under each scheme
- Inspect a distinctive word's weight under each scheme
- Confirm TF-IDF downweights the filler and highlights the signal
💡 Hint: Watch the ubiquitous term's weight collapse under TF-IDF while the rare term's weight holds up.
Exercise K2.2 — The synonym wall
Context: The synonym wall is the single clearest motivation for embeddings: intent-identical sentences with no shared words score low under keyword matching.
Your task: Compute TF-IDF cosine between "I need to reset my password" and "how do I recover my login", then embed both and compare cosine — explain the jump.
Requirements:
- Measure TF-IDF cosine between the two intent-identical sentences
- Observe the similarity is low despite matching intent
- Embed both sentences and recompute cosine
- Attribute the jump to the vocabulary-mismatch wall embeddings cross
💡 Hint: TF-IDF sees almost no shared tokens (reset/recover, password/login); embeddings place both near "regain account access".
Show what to look for
TF-IDF sees almost no shared words (reset/recover, password/login) → low similarity. Embeddings place both near "regain account access" → high similarity. That gap is the vocabulary-mismatch wall, and crossing it is why semantic search works.
Exercise K2.3 — Semantic search in ~20 lines
Context: Ranking embedded FAQ answers by cosine against an embedded question is, essentially, the retrieval core of a RAG system — built up from the representation layer.
Your task: Embed a small set of FAQ answers, embed a user question, rank the answers by cosine, and return the top match.
Requirements:
- Embed each FAQ answer once into a dense vector
- Embed the incoming question with the same model
- Rank the answers by cosine similarity to the question
- Return the single best-matching answer
💡 Hint: This is the same retrieve-by-cosine loop RAG uses; you have just built its core from scratch.
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: Bag-of-words is the simplest text→vector map: fix a vocabulary, then represent each document as counts over that vocabulary.
Your task: Build a vocabulary from a corpus, then vectorize each document as a count vector in vocabulary order.
Requirements:
- Derive the vocabulary as the sorted unique tokens across the corpus
- Represent each document as counts aligned to the vocabulary indices
- Words outside the vocabulary contribute nothing
- Print the vocabulary and each document's bag-of-words vector
💡 Hint: Every dimension is one vocabulary word and its value is that word's count — nothing more; the weighting problem is what TF-IDF fixes next.
Show solution
Vocabulary = sorted unique tokens; vector = counts in vocab order. Runnable:
from collections import Counter
def build_vocab(corpus):
vocab = sorted({w for doc in corpus for w in doc.split()})
return {w: i for i, w in enumerate(vocab)}
def bow(doc, vocab):
v = [0] * len(vocab)
for w, c in Counter(doc.split()).items():
if w in vocab:
v[vocab[w]] = c
return v
corpus = ["cat sat mat", "dog sat log", "cat and dog"]
vocab = build_vocab(corpus)
print("vocab:", list(vocab))
for d in corpus:
print(bow(d, vocab), d)BoW is the simplest text→vector map, but it treats every word as equally important — which TF-IDF fixes next.
Context: TF-IDF down-weights words that appear everywhere by multiplying term frequency by inverse document frequency, so filler words fade and distinctive terms stand out.
Your task: Implement tfidf(corpus) as term frequency × inverse document frequency and show that ubiquitous words receive near-zero weight.
Requirements:
- Compute document frequency per term across the corpus
- Use
idf = log(N / df)and multiply by term frequency - A word appearing in every document gets
idf = 0and vanishes - Distinctive words score higher than common ones
💡 Hint: TF-IDF discovers stopwords from the data itself: a term in all N docs has log(N/N)=0 idf and contributes nothing.
Show solution
idf = log(N / df); tf-idf = tf × idf. Pure stdlib. Runnable:
import math
from collections import Counter
def tfidf(corpus):
docs = [doc.split() for doc in corpus]
N = len(docs)
df = Counter()
for d in docs:
for w in set(d):
df[w] += 1
out = []
for d in docs:
tf = Counter(d)
n = len(d)
out.append({w: (tf[w] / n) * math.log(N / df[w]) for w in tf})
return out
corpus = ["the cat sat", "the dog sat", "the cat ran"]
for row in tfidf(corpus):
print({w: round(s, 3) for w, s in sorted(row.items())})
# 'the' -> 0.0 everywhere (df==N, idf==0); distinctive words score higherThe word "the" appears in every doc, so its idf is log(N/N)=0 and it contributes nothing — TF-IDF automatically discovers stopwords from the data.
Context: Cosine similarity compares vector direction, not magnitude, so a short focused document and a long one are compared fairly — the property that makes it the default text-similarity metric.
Your task: Implement cosine(a, b) in the standard library and use it to rank documents against a query vector.
Requirements:
- Compute cosine as the dot product over the product of the norms
- Handle a zero-length vector without dividing by zero
- Rank the documents best-first by similarity to the query
- Show that length independence lets a short doc outrank a longer one
💡 Hint: Because cosine normalizes out magnitude, document length stops distorting the match; only the shared-term direction matters.
Show solution
cos(a,b) = a·b / (|a||b|). Pure stdlib. Runnable:
import math
from collections import Counter
def cosine(a, b):
keys = set(a) | set(b)
dot = sum(a.get(k, 0) * b.get(k, 0) for k in keys)
na = math.sqrt(sum(v*v for v in a.values()))
nb = math.sqrt(sum(v*v for v in b.values()))
return dot / (na * nb) if na and nb else 0.0
def vec(text):
return Counter(text.split())
query = vec("cat food")
docs = {"d1": vec("cat food cat treats"),
"d2": vec("dog food"),
"d3": vec("premium cat food for cats")}
for name, sim in sorted(((n, cosine(query, v)) for n, v in docs.items()),
key=lambda x: -x[1]):
print(f"{name}: {sim:.3f}")Cosine ignores length, so a short focused doc can outrank a long one — the property that makes it the default text-similarity metric.
Context: TF-IDF has a hard ceiling: two synonyms with no shared token score zero similarity. Crossing that vocabulary-mismatch wall is exactly why semantic search moves to embeddings.
Your task: Demonstrate that cosine('car', 'automobile') is 0 under BoW/TF-IDF, then show a toy embedding where a shared latent dimension gives them high similarity.
Requirements:
- Show lexical cosine between the synonyms is exactly 0 (no shared token)
- Hand-build a small dense embedding where synonyms share a latent dimension
- Compute cosine on the dense vectors to show the synonyms now score high
- Confirm an unrelated word stays far apart in that space
💡 Hint: The leap is lexical → semantic: BoW sees no overlap, but a dense space places car and automobile near each other.
Show solution
First show TF-IDF's blind spot, then a hand-built embedding where synonyms share a dimension. Runnable:
import math
from collections import Counter
def cosine(a, b):
keys = set(a) | set(b)
dot = sum(a.get(k,0)*b.get(k,0) for k in keys)
na = math.sqrt(sum(v*v for v in a.values()))
nb = math.sqrt(sum(v*v for v in b.values()))
return dot/(na*nb) if na and nb else 0.0
print("TF-IDF/BoW cosine('car','automobile') =",
cosine(Counter(["car"]), Counter(["automobile"]))) # 0.0 -- no shared token
# toy embedding: both map onto a 'vehicle' latent dimension
EMB = {"car": [0.9, 0.1, 0.0],
"automobile": [0.88, 0.12, 0.0],
"banana": [0.0, 0.1, 0.95]}
def cos_vec(a, b):
dot = sum(x*y for x, y in zip(a, b))
na = math.sqrt(sum(x*x for x in a)); nb = math.sqrt(sum(y*y for y in b))
return dot/(na*nb)
print("embedding cosine('car','automobile') =", round(cos_vec(EMB["car"], EMB["automobile"]), 3))
print("embedding cosine('car','banana') =", round(cos_vec(EMB["car"], EMB["banana"]), 3))BoW/TF-IDF sees no overlap between synonyms; embeddings place them near each other in a dense space — the leap from lexical to semantic matching. Real embeddings come from a trained model (labeled needs-libs).
Context: Choosing a representation is a rule-of-thumb the lesson makes concrete: sparse TF-IDF for exact terms, small data, and interpretability; dense embeddings for meaning, synonyms, and transfer.
Your task: Encode choose_representation(need) that maps a stated requirement to the representation the lesson recommends.
Requirements:
- Route exact-keyword / interpretable / tiny-dataset needs to TF-IDF (sparse)
- Route synonym / paraphrase / semantic-search needs to dense embeddings
- Cover a cross-lingual need with multilingual embeddings
- Fall back to prototyping both when no rule matches
💡 Hint: Sparse for exact terms, small data, and interpretability; dense for meaning, synonyms, and cross-lingual transfer.
Show solution
Map the requirement to the representation the lesson recommends. Runnable:
def choose_representation(need):
rules = [
("exact keyword match", "TF-IDF / BM25 (sparse, exact terms)"),
("interpretable features","TF-IDF (each dim is a word)"),
("tiny labeled dataset", "TF-IDF + linear model (few params)"),
("synonyms / paraphrase", "dense embeddings (semantic)"),
("semantic search", "dense embeddings + vector index"),
("cross-lingual", "multilingual embeddings"),
]
for key, rec in rules:
if key in need:
return rec
return "prototype both and measure"
for n in ["exact keyword match", "semantic search", "synonyms / paraphrase"]:
print(f"{n:>24} -> {choose_representation(n)}")The rule of thumb: sparse (TF-IDF) for exact terms, small data, and interpretability; dense embeddings for meaning, synonyms, and transfer.
Context: A TF-IDF search index is a legitimate retrieval baseline — many production search systems start exactly here before adding dense embeddings for the semantic gap.
Your task: Ship a small TfidfIndex class: fit TF-IDF on a corpus, then rank documents for a free-text query by cosine similarity.
Requirements:
- Fit vocabulary and idf over the corpus at construction time
- Build a TF-IDF vector per document
- A
search(query, k)vectorizes the query and ranks by cosine - Return the top-
kdocuments best-first
💡 Hint: This is the classic sparse retrieval baseline (TF-IDF/BM25); embeddings get bolted on later only to close the synonym gap.
Show solution
Combine vocab, idf, tf-idf vectors, and cosine ranking into one class. Runnable:
import math
from collections import Counter
class TfidfIndex:
def __init__(self, corpus):
self.docs = [d.split() for d in corpus]
self.raw = corpus
N = len(self.docs)
df = Counter(w for d in self.docs for w in set(d))
self.idf = {w: math.log((1 + N) / (1 + df[w])) + 1 for w in df}
self.vecs = [self._vec(d) for d in self.docs]
def _vec(self, toks):
tf = Counter(toks); n = len(toks) or 1
return {w: (tf[w]/n) * self.idf.get(w, 0.0) for w in tf}
def _cos(self, a, b):
keys = set(a) | set(b)
dot = sum(a.get(k,0)*b.get(k,0) for k in keys)
na = math.sqrt(sum(v*v for v in a.values()))
nb = math.sqrt(sum(v*v for v in b.values()))
return dot/(na*nb) if na and nb else 0.0
def search(self, query, k=3):
q = self._vec(query.split())
scored = [(self._cos(q, v), self.raw[i]) for i, v in enumerate(self.vecs)]
return sorted(scored, reverse=True)[:k]
idx = TfidfIndex(["cheap flights to paris", "best pizza in paris",
"cheap car rental", "luxury hotels paris"])
for score, doc in idx.search("cheap paris trip", k=3):
print(f"{score:.3f} {doc}")This is a legitimate retrieval baseline — many production search systems start exactly here (TF-IDF/BM25) before adding dense embeddings for the semantic gap.
✓ Checkpoint — you can move on when you can…
- Explain why text must become vectors and what a good representation preserves.
- Build bag-of-words and TF-IDF and state their limits.
- Explain embeddings as dense, learned, meaning-capturing vectors.
- Use cosine similarity to rank texts by meaning.
- Choose sparse, dense, or hybrid for a retrieval task.
Knowledge check check yourself
What is the 'vocabulary mismatch' wall that TF-IDF cannot cross, and how do embeddings solve it?
Show answer
Cosine similarity is the standard way to compare embedded texts. What does it measure, why divide by the vectors' norms, and what do 1, 0, and -1 mean?