Retrieval that works

1 · Recap: the retrieve stage, and why top-k isn't enough

In 3.1 the retrieve stage was one function: embed the question, score every chunk by cosine similarity, return the top k. That is the correct skeleton — but on real corpora it leaks. Dense cosine search is fuzzy by design: it ranks by overall meaning, which is exactly what you want for paraphrases and exactly what you don't want for precise tokens.

2 · Chunking strategies in depth

3.1 introduced overlap; here we compare the three strategies you will actually choose between. A fact can only be retrieved if it sits inside a chunk that scores well — so the boundary you cut on decides what is findable.

Two things ride along with every chunk. Overlap repeats a little text across boundaries so a straddling fact lands whole in at least one chunk. Metadata — source, section, date, doc type — travels with the chunk so you can filter and cite it later. Here is a structural chunker that keeps headings attached and carries metadata:

chunk_structural.pyimport re

def chunk_structural(doc, source, doc_type):
    """Split on markdown-ish headings; keep each heading with its body.
    Returns a list of {text, source, section, type} chunks (metadata rides along)."""
    chunks, section, buf = [], 'intro', []
    def flush():
        if buf:
            chunks.append({
                'text': f'{section}: ' + ' '.join(buf).strip(),
                'source': source, 'section': section, 'type': doc_type,
            })
    for line in doc.strip().splitlines():
        line = line.strip()
        if not line:
            continue
        m = re.match(r'#+\s*(.+)', line)   # a heading starts a new section
        if m:
            flush(); buf = []; section = m.group(1)
        else:
            buf.append(line)
    flush()
    return chunks

doc = '''# Restart
Drain the queue then scale the consumer group.
# Rollback
Redeploy the previous image tag and clear the cache.'''

for c in chunk_structural(doc, source='runbook.md', doc_type='runbook'):
    print(c)

{'text': 'Restart: Drain the queue then scale the consumer group.', 'source': 'runbook.md', 'section': 'Restart', 'type': 'runbook'}
{'text': 'Rollback: Redeploy the previous image tag and clear the cache.', 'source': 'runbook.md', 'section': 'Rollback', 'type': 'runbook'}

Each chunk is one self-contained section, its heading is folded into the text (so “Restart” is embedded with its steps), and it carries the metadata we will filter and cite on later. That metadata is not decoration — section 5 uses it to scope search.

3 · Vector search: what an index actually does

Scoring the query against every stored vector is called brute-force (or “flat”) search. It is exact — it truly finds the nearest neighbours — and for a few thousand chunks it is perfectly fast. It is also what you built in 3.1. Here it is, made explicit as a tiny index:

brute_force.pyimport math, re
from collections import Counter

VOCAB = {}
def embed(text):                      # the SAME fake embedder as 3.1
    counts = Counter(re.findall(r'[a-z]+', text.lower()))
    for w in counts: VOCAB.setdefault(w, len(VOCAB))
    vec = [0.0] * len(VOCAB)
    for w, c in counts.items(): vec[VOCAB[w]] = float(c)
    return vec

def cosine(a, b):
    n = max(len(a), len(b)); a = a + [0.0]*(n-len(a)); b = b + [0.0]*(n-len(b))
    dot = sum(x*y for x, y in zip(a, b))
    na = math.sqrt(sum(x*x for x in a)); nb = math.sqrt(sum(x*x for x in b))
    return dot / (na*nb + 1e-9)

docs = ['drain the queue then scale the consumer group',
        'redeploy the previous image tag and clear the cache',
        'invoices are stored in postgres by the billing service']
index = [(d, embed(d)) for d in docs]          # embed once, keep the vectors

def brute_force_topk(query, k=2):
    q = embed(query)
    scored = sorted(index, key=lambda di: cosine(q, di[1]), reverse=True)
    return [(round(cosine(q, v), 3), d) for d, v in scored[:k]]

for score, d in brute_force_topk('how do I clear the image cache?'):
    print(score, d)

0.57 redeploy the previous image tag and clear the cache
0.239 drain the queue then scale the consumer group

Brute-force compares against N vectors, so cost grows with the corpus. At millions of vectors that is too slow, so production stores use an Approximate Nearest Neighbour (ANN) index — HNSW (a navigable graph of vectors) or IVF (cluster the space, search only the near clusters). These trade a little recall for a huge speed-up. The index internals and when to tune them are 3.4's job — here just hold the idea: an index is a data structure that finds near vectors without scanning them all.

4 · Hybrid search — keyword + dense, fused with RRF

Dense search finds meaning; keyword search finds exact tokens. Real queries need both — a user asking about “ERR_5041 on checkout” wants the literal error code (keyword) and the conceptually related recovery steps (dense). Hybrid search runs both retrievers and merges their ranked lists.

The keyword side uses BM25, the standard term-frequency ranker behind classic search engines: a chunk scores higher when the query's terms appear often in it but are rare across the corpus, with diminishing returns as a term repeats. Below is a simplified BM25 (term-frequency scoring; the real formula adds document-length normalisation, omitted here for clarity):

bm25.pyimport math, re
from collections import Counter

def toks(t): return re.findall(r'[a-z0-9]+', t.lower())

def build_bm25(corpus):
    docs = [Counter(toks(d)) for d in corpus]
    N = len(docs)
    df = Counter()                          # in how many docs each term appears
    for d in docs:
        for term in d: df[term] += 1
    idf = {t: math.log(1 + (N - n + 0.5) / (n + 0.5)) for t, n in df.items()}
    return docs, idf

def bm25_rank(query, corpus, k1=1.5):
    docs, idf = build_bm25(corpus)
    scores = []
    for i, d in enumerate(docs):
        s = 0.0
        for term in toks(query):
            if term in d:
                tf = d[term]
                s += idf.get(term, 0.0) * (tf * (k1 + 1)) / (tf + k1)   # TF saturation
        scores.append((s, i))
    return sorted(scores, reverse=True)     # (score, doc_index), best first

corpus = ['checkout error err_5041 means the queue is full',
          'drain the queue then scale the consumer group to recover',
          'billing stores invoices in postgres']
for s, i in bm25_rank('err_5041 checkout', corpus):
    print(round(s, 3), corpus[i])

2.942 checkout error err_5041 means the queue is full
0.0 billing stores invoices in postgres
0.0 drain the queue then scale the consumer group to recover

BM25 nails the exact-token query that dense search fuzzes. Now the fusion. You could try to add the two scores, but BM25 and cosine live on different scales — that comparison is meaningless. Reciprocal Rank Fusion (RRF) sidesteps it entirely: it merges by rank position, not score, giving each chunk 1/(k + rank) from each list and summing. Rank is comparable across any two rankers, which is why RRF is the standard hybrid-merge trick:

rrf.pydef rrf(ranked_lists, k=60):
    """Fuse several ranked lists of ids into one. ranked_lists: list of
    [id, id, ...] each ordered best-first. Returns [(id, fused_score)] best-first.
    k=60 is the common default; it damps the weight of very top ranks a little."""
    fused = {}
    for ranking in ranked_lists:
        for rank, doc_id in enumerate(ranking):     # rank 0 = best
            fused[doc_id] = fused.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
    return sorted(fused.items(), key=lambda kv: kv[1], reverse=True)

# id lists from two retrievers over the SAME chunk ids (0,1,2,3):
bm25_ids  = [0, 2, 1, 3]     # keyword liked chunk 0 best
dense_ids = [2, 0, 3, 1]     # dense liked chunk 2 best

for doc_id, score in rrf([bm25_ids, dense_ids]):
    print(f'chunk {doc_id}: {round(score, 5)}')

chunk 0: 0.03252
chunk 2: 0.03252
chunk 1: 0.0315
chunk 3: 0.0315

Chunks 0 and 2 tie at the top because each was ranked #1 by one retriever and #2 by the other — RRF rewards agreement across the two lists without ever comparing their raw scores. In a full hybrid retriever you run bm25_rank and your dense brute_force_topk, pass both id-lists to rrf, and return the fused top k. That single merge typically beats either retriever alone.

5 · Metadata filtering — scoping the search

Semantic relevance is not the same as being allowed or in scope. A user asking “what changed this quarter?” does not want a semantically perfect chunk from 2019. Because our chunks carry metadata (section 2), we can constrain by date, source, or doc type. There are two places to do it:

metadata_filter.pychunks = [
    {'text': 'drain the queue then scale', 'type': 'runbook', 'year': 2026},
    {'text': 'legacy queue restart steps', 'type': 'runbook', 'year': 2019},
    {'text': 'quarterly billing summary',  'type': 'report',  'year': 2026},
]

def retrieve(query, chunks, where=None, k=2):
    # 1) PRE-FILTER on metadata before any scoring
    pool = [c for c in chunks if all(c.get(f) == v for f, v in (where or {}).items())]
    # 2) rank the survivors (toy keyword overlap stands in for a real scorer)
    qset = set(query.lower().split())
    scored = sorted(pool, key=lambda c: len(qset & set(c['text'].lower().split())), reverse=True)
    return scored[:k]

q = 'how do I restart the queue'
print('no filter :', [c['text'] for c in retrieve(q, chunks)])
print('2026 only :', [c['text'] for c in retrieve(q, chunks, where={'year': 2026})])
print('runbooks  :', [c['text'] for c in retrieve(q, chunks, where={'type': 'runbook'})])

no filter : ['drain the queue then scale', 'legacy queue restart steps']
2026 only : ['drain the queue then scale', 'quarterly billing summary']
runbooks  : ['drain the queue then scale', 'legacy queue restart steps']

The where clause scopes the search: asking for 2026 drops the 2019 runbook before scoring, and asking for runbooks drops the report. Real vector databases expose exactly this as a filter argument on the query; the mechanism above is what they do under the hood.

6 · Query preprocessing — small, boring, high-leverage

Before you search, normalise the query so it lines up with how chunks were indexed. This is unglamorous and cheap, and it prevents whole classes of misses:

The larger lesson: exact match still matters. Dense retrieval is not a superset of keyword retrieval — it is a different, complementary tool. That is the entire justification for hybrid search: keep the literal-token power of BM25 and the meaning-power of embeddings, and let RRF decide.

7 · When each approach wins