Retrieval that works

1 · Recap: the retrieve stage, and why top-k isn't enough

Here's the demo that goes wrong, and it's a story you'll live if you ship a RAG system. On your test set of a dozen questions, naive cosine top-k looks brilliant — you ask "how do I roll back a deploy?" and the right runbook comes back first. You ship it. Then a real user pastes an error code, "ERR_5041", and the system confidently returns three chunks about queues and caches that never mention the code at all. Nobody wrote a bug. The retriever did exactly what it was built to do — and that's the problem.

Think of dense embedding search as a librarian who's brilliant at gist but never reads the exact words. Ask her for "a book about a boy wizard" and she nails it; ask her for "the book with ISBN 978-0439708180" and she hands you something about ISBNs, or a different wizard book, because she matched the vibe of your request, not the literal string. That's dense retrieval: it maps meaning into geometry and ranks by nearness in that space, so paraphrases cluster together — which is exactly the superpower you want most of the time, and exactly the blind spot that sinks precise queries.

The mistake nearly everyone makes is assuming that because embeddings are newer and smarter, they must be a superset of old-fashioned keyword search — that dense search can do everything keyword search does, plus more. It can't. Meaning-matching and token-matching are different tools with different failure modes, and neither contains the other. The whole chapter hangs off internalising that one fact: retrieval quality isn't a single dial you turn up, it's a pipeline of three separable decisions.

In 3.1 the retrieve stage was one function: embed the question, score every chunk by cosine similarity, return the top k. That is the correct skeleton — but on real corpora it leaks. Dense cosine search is fuzzy by design: it ranks by overall meaning, which is exactly what you want for paraphrases and exactly what you don't want for precise tokens.

2 · Chunking strategies in depth

Before you retrieve anything, you make one quiet decision that caps everything downstream: where to cut the text. It feels like plumbing — split the document, move on — but it's the most consequential choice on this page. Every clever ranker, hybrid fuser, and re-ranker you build later can only ever return chunks you actually created. If the answer to a question got sliced across two chunks, no retriever on earth can hand you the whole answer, because the whole answer doesn't exist anywhere as a single unit.

Picture cutting a printed manual into index cards to file in a box. Cut carelessly — every 3 inches with scissors — and you'll slice a sentence in half, orphan a heading from its steps, or staple the end of one procedure to the start of the next. Later, when someone asks "how do I restart the service?", the card that surfaces might hold the heading "Restart" but only the first step, with steps two and three stranded on a card that got filed elsewhere. Cut along the natural seams instead — one procedure per card, heading included — and each card answers its own question. Chunking is choosing where those scissors land.

The classic mistake is fixed-size chunking on structured content because it's the easiest to code: just count 500 tokens and cut. It works fine on a wall of uniform prose, but on a runbook, an API doc, or a wiki it guillotines the exact structure that made the content answerable — separating a code example from the sentence explaining it, or a table row from its header. The fix isn't a fancier algorithm; it's cutting on the boundaries the author already gave you (headings, paragraphs, list items) and adding a little overlap so a fact straddling a seam still lands whole in at least one chunk.

3.1 introduced overlap; here we compare the three strategies you will actually choose between. A fact can only be retrieved if it sits inside a chunk that scores well — so the boundary you cut on decides what is findable.

Two things ride along with every chunk. Overlap repeats a little text across boundaries so a straddling fact lands whole in at least one chunk. Metadata — source, section, date, doc type — travels with the chunk so you can filter and cite it later. Here is a structural chunker that keeps headings attached and carries metadata:

chunk_structural.pyimport re

def chunk_structural(doc, source, doc_type):
    """Split on markdown-ish headings; keep each heading with its body.
    Returns a list of {text, source, section, type} chunks (metadata rides along)."""
    chunks, section, buf = [], 'intro', []
    def flush():
        if buf:
            chunks.append({
                'text': f'{section}: ' + ' '.join(buf).strip(),
                'source': source, 'section': section, 'type': doc_type,
            })
    for line in doc.strip().splitlines():
        line = line.strip()
        if not line:
            continue
        m = re.match(r'#+\s*(.+)', line)   # a heading starts a new section
        if m:
            flush(); buf = []; section = m.group(1)
        else:
            buf.append(line)
    flush()
    return chunks

doc = '''# Restart
Drain the queue then scale the consumer group.
# Rollback
Redeploy the previous image tag and clear the cache.'''

for c in chunk_structural(doc, source='runbook.md', doc_type='runbook'):
    print(c)

{'text': 'Restart: Drain the queue then scale the consumer group.', 'source': 'runbook.md', 'section': 'Restart', 'type': 'runbook'}
{'text': 'Rollback: Redeploy the previous image tag and clear the cache.', 'source': 'runbook.md', 'section': 'Rollback', 'type': 'runbook'}

Each chunk is one self-contained section, its heading is folded into the text (so “Restart” is embedded with its steps), and it carries the metadata we will filter and cite on later. That metadata is not decoration — section 5 uses it to scope search.

3 · Vector search: what an index actually does

Your 3.1 retriever had a scaling bomb quietly ticking inside it, and it won't go off in testing. Scoring the query against every stored vector is fine at a hundred chunks and invisible at a thousand. But retrieval cost grows linearly with the corpus — double the documents, double the work per query — so the same code that answers in 5 milliseconds on your laptop demo takes half a second when the knowledge base grows to a million chunks, and that's before any concurrent users. The word for that pattern is the O(n) scan you met in the DSA track, and it's why "it was fast when we launched" is one of the most common laments in production RAG.

Here's the intuition for the fix. Imagine finding the nearest coffee shop to you. The brute-force way is to measure the straight-line distance to every coffee shop in the country and take the minimum — correct, but absurd. What you actually do is glance at a map: the map has already organised places by region, so you ignore everything outside your city and only compare the handful nearby. A vector index is that map for embeddings — a data structure that pre-organises the vectors so a query can jump to the promising neighbourhood and skip the 99.9% that couldn't possibly be closest.

The mistake to avoid is reaching for the fancy index too early. Approximate indexes (HNSW, IVF) are a genuine trade — they buy speed by occasionally missing a true nearest neighbour — and that trade only pays off at scale. For a few thousand chunks, brute-force is exact, dead simple, and plenty fast; adding an ANN index there just buys you tuning knobs and a recall risk for a speed-up you don't need. Learn brute-force first because it's the honest baseline every index is measured against; switch when your n — not your excitement about the technique — demands it.

Scoring the query against every stored vector is called brute-force (or “flat”) search. It is exact — it truly finds the nearest neighbours — and for a few thousand chunks it is perfectly fast. It is also what you built in 3.1. Here it is, made explicit as a tiny index:

brute_force.pyimport math, re
from collections import Counter

VOCAB = {}
def embed(text):                      # the SAME fake embedder as 3.1
    counts = Counter(re.findall(r'[a-z]+', text.lower()))
    for w in counts: VOCAB.setdefault(w, len(VOCAB))
    vec = [0.0] * len(VOCAB)
    for w, c in counts.items(): vec[VOCAB[w]] = float(c)
    return vec

def cosine(a, b):
    n = max(len(a), len(b)); a = a + [0.0]*(n-len(a)); b = b + [0.0]*(n-len(b))
    dot = sum(x*y for x, y in zip(a, b))
    na = math.sqrt(sum(x*x for x in a)); nb = math.sqrt(sum(x*x for x in b))
    return dot / (na*nb + 1e-9)

docs = ['drain the queue then scale the consumer group',
        'redeploy the previous image tag and clear the cache',
        'invoices are stored in postgres by the billing service']
index = [(d, embed(d)) for d in docs]          # embed once, keep the vectors

def brute_force_topk(query, k=2):
    q = embed(query)
    scored = sorted(index, key=lambda di: cosine(q, di[1]), reverse=True)
    return [(round(cosine(q, v), 3), d) for d, v in scored[:k]]

for score, d in brute_force_topk('how do I clear the image cache?'):
    print(score, d)

0.57 redeploy the previous image tag and clear the cache
0.239 drain the queue then scale the consumer group

Brute-force compares against N vectors, so cost grows with the corpus. At millions of vectors that is too slow, so production stores use an Approximate Nearest Neighbour (ANN) index — HNSW (a navigable graph of vectors) or IVF (cluster the space, search only the near clusters). These trade a little recall for a huge speed-up. The index internals and when to tune them are 3.4's job — here just hold the idea: an index is a data structure that finds near vectors without scanning them all.

4 · Hybrid search — keyword + dense, fused with RRF

This is the section that turns a demo retriever into one you'd actually trust, so slow down here. Everything so far has been setup; hybrid search is the single change that fixes the most real-world misses for the least effort. If you take one technique from this whole chapter into a production system, make it this one — it's the difference between a retriever that shrugs at error codes and one that handles both "vehicle won't crank" and "ERR_5041" without you knowing in advance which kind of query is coming.

The core idea is embarrassingly simple once you see it: you have two specialists, so ask both and combine their answers. Dense search is your meaning expert; BM25 is your exact-token expert. A real query — "ERR_5041 on checkout" — has a literal part (the code) that keyword search nails and a conceptual part (what to do about it) that dense search nails. Running only one throws away half the query. Think of a hiring panel: you don't want a room full of people who all interview for the same trait; you want one who probes technical depth and one who probes communication, then you merge their rankings. Hybrid search is that panel for retrieval.

The trap that eats days of debugging is trying to merge by adding the scores. It feels obvious — BM25 gives a chunk 2.5, cosine gives it 0.4, add them, sort. But those numbers live on incompatible scales that have no shared meaning: BM25 scores are unbounded and corpus-dependent, cosine sits in a narrow band near zero-to-one, and "normalising" them into agreement is a per-corpus tuning nightmare that breaks the moment your data shifts. The escape hatch is to throw away the raw scores and keep only rank position — first, second, third — because a "#1 result" means the same thing coming from either ranker. That's exactly what Reciprocal Rank Fusion does, and it's why it's the near-universal default for hybrid merge.

Dense search finds meaning; keyword search finds exact tokens. Real queries need both — a user asking about “ERR_5041 on checkout” wants the literal error code (keyword) and the conceptually related recovery steps (dense). Hybrid search runs both retrievers and merges their ranked lists.

The keyword side uses BM25, the standard term-frequency ranker behind classic search engines: a chunk scores higher when the query's terms appear often in it but are rare across the corpus, with diminishing returns as a term repeats. Below is a simplified BM25 (term-frequency scoring; the real formula adds document-length normalisation, omitted here for clarity):

bm25.pyimport math, re
from collections import Counter

def toks(t): return re.findall(r'[a-z0-9]+', t.lower())

def build_bm25(corpus):
    docs = [Counter(toks(d)) for d in corpus]
    N = len(docs)
    df = Counter()                          # in how many docs each term appears
    for d in docs:
        for term in d: df[term] += 1
    idf = {t: math.log(1 + (N - n + 0.5) / (n + 0.5)) for t, n in df.items()}
    return docs, idf

def bm25_rank(query, corpus, k1=1.5):
    docs, idf = build_bm25(corpus)
    scores = []
    for i, d in enumerate(docs):
        s = 0.0
        for term in toks(query):
            if term in d:
                tf = d[term]
                s += idf.get(term, 0.0) * (tf * (k1 + 1)) / (tf + k1)   # TF saturation
        scores.append((s, i))
    return sorted(scores, reverse=True)     # (score, doc_index), best first

corpus = ['checkout error err_5041 means the queue is full',
          'drain the queue then scale the consumer group to recover',
          'billing stores invoices in postgres']
for s, i in bm25_rank('err_5041 checkout', corpus):
    print(round(s, 3), corpus[i])

2.942 checkout error err_5041 means the queue is full
0.0 billing stores invoices in postgres
0.0 drain the queue then scale the consumer group to recover

BM25 nails the exact-token query that dense search fuzzes. Now the fusion. You could try to add the two scores, but BM25 and cosine live on different scales — that comparison is meaningless. Reciprocal Rank Fusion (RRF) sidesteps it entirely: it merges by rank position, not score, giving each chunk 1/(k + rank) from each list and summing. Rank is comparable across any two rankers, which is why RRF is the standard hybrid-merge trick:

rrf.pydef rrf(ranked_lists, k=60):
    """Fuse several ranked lists of ids into one. ranked_lists: list of
    [id, id, ...] each ordered best-first. Returns [(id, fused_score)] best-first.
    k=60 is the common default; it damps the weight of very top ranks a little."""
    fused = {}
    for ranking in ranked_lists:
        for rank, doc_id in enumerate(ranking):     # rank 0 = best
            fused[doc_id] = fused.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
    return sorted(fused.items(), key=lambda kv: kv[1], reverse=True)

# id lists from two retrievers over the SAME chunk ids (0,1,2,3):
bm25_ids  = [0, 2, 1, 3]     # keyword liked chunk 0 best
dense_ids = [2, 0, 3, 1]     # dense liked chunk 2 best

for doc_id, score in rrf([bm25_ids, dense_ids]):
    print(f'chunk {doc_id}: {round(score, 5)}')

chunk 0: 0.03252
chunk 2: 0.03252
chunk 1: 0.0315
chunk 3: 0.0315

Chunks 0 and 2 tie at the top because each was ranked #1 by one retriever and #2 by the other — RRF rewards agreement across the two lists without ever comparing their raw scores. In a full hybrid retriever you run bm25_rank and your dense brute_force_topk, pass both id-lists to rrf, and return the fused top k. That single merge typically beats either retriever alone.

5 · Metadata filtering — scoping the search

A retriever can be perfectly relevant and still perfectly wrong. Picture a user in a regulated company asking "what's our current refund policy?" The best-matching chunk by pure meaning might be a beautifully worded policy — from 2019, since superseded, and legally dangerous to quote. Or worse: in a multi-tenant product, the most semantically similar chunk belongs to another customer's data. Similarity answered "what's most related?"; nobody asked "what's this user allowed and expected to see?" Those are different questions, and conflating them is how RAG systems leak stale answers and, occasionally, private data.

The mental model is a search with a bouncer at the door. Metadata — the source, section, date, and type tags that rode along with each chunk back in section 2 — is the guest list. Before (or after) you rank by relevance, you check each candidate against the list: wrong year, wrong product, wrong tenant, not admitted. Relevance ranking decides who's interesting; metadata filtering decides who's eligible, and eligibility is not negotiable when the constraint is security or compliance rather than mere preference.

The subtle mistake is doing the filter in the wrong order — specifically, ranking first and filtering the survivors (post-filter) when you should have filtered first (pre-filter). It sounds harmless until you picture it: you take the top 5 by similarity, then drop everything not from 2026, and discover 4 of your 5 were old — you're left with one result, or none, even though plenty of fresh, relevant chunks existed just below the cut. Pre-filtering searches only the eligible pool, so it always fills your k when eligible matches exist, and for hard constraints like tenant isolation it's the only correct choice. Post-filter only when the constraint is loose and your index can't filter natively.

Semantic relevance is not the same as being allowed or in scope. A user asking “what changed this quarter?” does not want a semantically perfect chunk from 2019. Because our chunks carry metadata (section 2), we can constrain by date, source, or doc type. There are two places to do it:

metadata_filter.pychunks = [
    {'text': 'drain the queue then scale', 'type': 'runbook', 'year': 2026},
    {'text': 'legacy queue restart steps', 'type': 'runbook', 'year': 2019},
    {'text': 'quarterly billing summary',  'type': 'report',  'year': 2026},
]

def retrieve(query, chunks, where=None, k=2):
    # 1) PRE-FILTER on metadata before any scoring
    pool = [c for c in chunks if all(c.get(f) == v for f, v in (where or {}).items())]
    # 2) rank the survivors (toy keyword overlap stands in for a real scorer)
    qset = set(query.lower().split())
    scored = sorted(pool, key=lambda c: len(qset & set(c['text'].lower().split())), reverse=True)
    return scored[:k]

q = 'how do I restart the queue'
print('no filter :', [c['text'] for c in retrieve(q, chunks)])
print('2026 only :', [c['text'] for c in retrieve(q, chunks, where={'year': 2026})])
print('runbooks  :', [c['text'] for c in retrieve(q, chunks, where={'type': 'runbook'})])

no filter : ['drain the queue then scale', 'legacy queue restart steps']
2026 only : ['drain the queue then scale', 'quarterly billing summary']
runbooks  : ['drain the queue then scale', 'legacy queue restart steps']

The where clause scopes the search: asking for 2026 drops the 2019 runbook before scoring, and asking for runbooks drops the report. Real vector databases expose exactly this as a filter argument on the query; the mechanism above is what they do under the hood.

6 · Query preprocessing — small, boring, high-leverage

The highest return-on-effort fix in retrieval is also the one nobody wants to write, because it's dull. There are no embeddings to admire and no clever fusion to diagram — just quietly making sure the query and the index agree on the basics. Yet a whole category of "why did it miss the obvious chunk?" bugs trace back to a mismatch this boring: the query said ERR_5041 and the index stored err_5041, and because they're different strings, the exact-match ranker that should have scored 10 scored 0.

The rule of thumb is a single sentence: whatever you do to the text you index, do the identical thing to the query — no more, no less. Think of it like a key and a lock cut from the same jig. If you lowercase your documents when indexing, you must lowercase the query, or the "key" won't turn. If you don't touch the query but silently lowercased the index (or vice versa), you've filed the key and the lock differently and they'll never mesh. Symmetry is the whole game; asymmetry is the bug.

The trap here is over-processing — being too clever. Beginners reach for aggressive stemming and stopword stripping because a tutorial said classic search engines do it, and end up mangling the exact tokens that keyword search exists to protect: a version string "v2.1" gets stemmed to "v2", a SKU or error code gets its digits shaved off, and now the literal-match power you carefully added with BM25 is gone. The lesson from the whole chapter reappears here in miniature: normalise lightly and symmetrically, and leave codes, IDs, and versions exactly as they are — the exact token is the signal, not noise to be scrubbed.

Before you search, normalise the query so it lines up with how chunks were indexed. This is unglamorous and cheap, and it prevents whole classes of misses:

The larger lesson: exact match still matters. Dense retrieval is not a superset of keyword retrieval — it is a different, complementary tool. That is the entire justification for hybrid search: keep the literal-token power of BM25 and the meaning-power of embeddings, and let RRF decide.

7 · When each approach wins

Here's the moment the toolkit becomes judgement. You now hold five tools — dense search, BM25, hybrid, metadata filtering, ANN indexing — and the beginner instinct is to grab the newest or most impressive one for everything. That's exactly backwards. A senior engineer isn't the person who knows the most techniques; it's the person who can look at a query and a corpus and say "this one wants keyword, that one wants dense, and at your scale you'll also need an index." Matching the tool to the situation is the skill this whole chapter was building toward.

The analogy is a mechanic's toolbox. A torque wrench is a wonderful instrument, but you don't use it to hammer a nail — not because it's a bad tool, but because it's the wrong tool for that job. Dense embeddings are your torque wrench: precise and modern and genuinely better for the paraphrase queries they're built for, and genuinely worse than a plain BM25 "hammer" when the job is driving an exact error code home. The table below is your at-a-glance guide to which tool each job calls for — read it as decisions, not trivia.

The one summary worth carrying out of here: when in doubt, reach for hybrid. The honest truth of production retrieval is that you rarely know in advance whether the next query is a paraphrase or an exact token — users mix both, often in the same sentence. Hybrid plus RRF covers both cases safely with almost no tuning, which is why it's the sane default and the pure-dense or pure-keyword rows below are the specialisations you drop to only when you know the query shape.