Advanced retrieval

1 · Why naive top-k retrieval fails essential

Basic RAG splits documents into fixed-size chunks, embeds each one, and at query time returns the top-k chunks whose embedding is closest to the query embedding. That works for easy questions and quietly fails for hard ones. Three failure modes dominate:

The fix is not one trick but a pipeline: situate each chunk before you embed it, retrieve with two methods that fail differently, fuse their rankings, then rerank the survivors with a model that actually reads query and chunk together. Here is the whole shape:

And the ingestion path that feeds it — the key move is contextual chunking, where each raw chunk gets a one-line situating summary prepended before it is embedded and indexed:

2 · Contextual retrieval — situate the chunk essential

A chunk ripped out of a 40-page document often can't stand on its own. Contextual retrieval (the Anthropic technique) fixes this at ingestion time: for each chunk, generate a short sentence that says where this chunk sits in the document, and prepend it before embedding. The stored text becomes "context line + original chunk", so both the embedding and the BM25 index capture what the chunk is actually about. Below we model the transform with plain string work and show how it changes token overlap with the query.

contextual_chunk.pydef contextualize(doc_title, section, chunk):
    """Prepend a one-line situating summary before the chunk (Anthropic technique).
    In production the context line is written by a cheap LLM; here we template it."""
    context = f"This chunk is from '{doc_title}', section '{section}'."
    return context + " " + chunk

def overlap(query, text):
    """Fraction of query words that appear in the text (a crude retrieval proxy)."""
    q = set(query.lower().split())
    t = set(text.lower().replace("'", "").replace(".", "").split())
    return round(len(q & t) / len(q), 2)

raw = "It grew 3% quarter over quarter."
ctx = contextualize("Acme FY24 Report", "EMEA revenue", raw)
query = "acme emea revenue growth"

print("raw chunk   :", raw)
print("ctx chunk   :", ctx)
print("overlap raw :", overlap(query, raw))
print("overlap ctx :", overlap(query, ctx))

raw chunk   : It grew 3% quarter over quarter.
ctx chunk   : This chunk is from 'Acme FY24 Report', section 'EMEA revenue'. It grew 3% quarter over quarter.
overlap raw : 0.0
overlap ctx : 0.75

3 · Hybrid search — dense + sparse, fused with RRF intermediate

Dense (embedding) search captures meaning but misses exact tokens; sparse search — classically BM25, a keyword-frequency score — nails exact terms but misses paraphrase. They fail on different queries, so run both and combine. The standard combiner is Reciprocal Rank Fusion (RRF): score each document by the sum of 1/(k + rank) across the two ranked lists. It needs no score calibration — only ranks — which is why it is the default fusion in most hybrid stacks.

hybrid_rrf.pydef rrf(rankings, k=60):
    """rankings: list of ranked lists (each is doc-ids best-first). Returns fused
    scores. RRF adds 1/(k + rank) for each list a doc appears in; rank is 0-based."""
    scores = {}
    for ranked in rankings:
        for rank, doc in enumerate(ranked):
            scores[doc] = scores.get(doc, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=lambda d: scores[d], reverse=True)

# A "dense-ish" ranking (by meaning) and a keyword (BM25-ish) ranking disagree.
dense  = ["d_reimburse", "d_shipping", "d_refund", "d_hours"]   # got paraphrase
sparse = ["d_refund", "d_err5021", "d_reimburse", "d_shipping"]  # got exact terms

fused = rrf([dense, sparse])
print("dense top1 :", dense[0])
print("sparse top1:", sparse[0])
print("fused order:", fused[:3])

dense top1 : d_reimburse
sparse top1: d_refund
fused order: ['d_reimburse', 'd_refund', 'd_shipping']

Notice d_reimburse (only dense found it) and d_refund (both found it, ranked high by sparse) both surface at the top — RRF rewards documents that either rank rank highly in one list or appear in both. No embedding-vs-BM25 score normalization was needed.

4 · Reranking — a cross-encoder narrows a wide recall set advanced

Retrieval optimizes for recall: cast a wide net (fetch, say, top-50) so the answer is somewhere in the set. But the LLM only wants the best few. A reranker — typically a cross-encoder that reads the query and each candidate together in one forward pass — scores relevance far more accurately than the bi-encoder embeddings used for first-stage recall. The pattern is retrieve wide, rerank to narrow: cheap recall over the whole corpus, expensive precision over the shortlist only.

rerank_stub.pydef first_stage_recall(query, corpus):
    """Cheap keyword-frequency proxy: count query-word occurrences. Repetition
    inflates the score (fast, wide, imprecise) — like a first-stage BM25 net."""
    q = query.lower().split()
    scored = [(doc, sum(text.lower().split().count(w) for w in q)) for doc, text in corpus]
    return [doc for doc, _ in sorted(scored, key=lambda x: x[1], reverse=True)]

def cross_encoder_score(query, text):
    """Stub for a cross-encoder: richer than word-frequency. Rewards the exact
    phrase and penalizes length (a real model reads query+doc jointly)."""
    q = query.lower()
    score = 2.0 if q in text.lower() else len(set(q.split()) & set(text.lower().split()))
    return score - 0.01 * len(text.split())   # slight brevity preference

def rerank(query, candidates, corpus_text):
    return sorted(candidates, key=lambda d: cross_encoder_score(query, corpus_text[d]),
                  reverse=True)

corpus = [
    ("d1", "refund refund refund refund policy and refund window notes"),
    ("d2", "shipping times and delivery windows for your orders"),
    ("d3", "how to request a refund from the orders page"),
]
text = dict(corpus)
query = "request a refund"

recall = first_stage_recall(query, corpus)   # wide, keyword-frequency biased
final  = rerank(query, recall, text)         # precise, phrase-aware
print("first-stage:", recall)
print("reranked   :", final)

first-stage: ['d1', 'd3', 'd2']
reranked   : ['d3', 'd1', 'd2']

First-stage recall put d1 on top because it repeats the word "refund" five times — keyword frequency, not relevance. The reranker reads the whole query against each doc, spots the exact phrase "request a refund" in d3, and promotes it. That is the recall-vs-precision division of labor: the first stage must not lose the answer; the reranker decides what the LLM actually sees.

5 · Embedding fine-tuning — adapt to your domain advanced

Off-the-shelf embeddings are trained on general web text. If your domain has its own vocabulary (legal, medical, internal product codes), a general embedder may place "related" documents far apart in vector space. Fine-tuning the embedding model on your own (query, relevant-doc) pairs pulls domain-related items closer together. It is powerful but the last lever to pull — contextual chunking, hybrid search, and reranking are cheaper and usually get you most of the way.

finetune_decision.pydef should_finetune_embeddings(domain_vocab_gap, have_labeled_pairs,
                              tried_hybrid_and_rerank, volume_high):
    if not tried_hybrid_and_rerank:
        return "NO — exhaust contextual chunking + hybrid + rerank first (cheaper)"
    if not have_labeled_pairs:
        return "NO — no (query, relevant-doc) pairs to train/eval on"
    if domain_vocab_gap and volume_high:
        return "YES — off-distribution domain + volume justifies the retrain cost"
    return "MAYBE — measure the recall gain vs the retrain/maintenance cost first"

print(should_finetune_embeddings(True, True, False, True))    # skipped the basics
print(should_finetune_embeddings(True, True, True, True))      # the real case
print(should_finetune_embeddings(True, False, True, True))     # no data

NO — exhaust contextual chunking + hybrid + rerank first (cheaper)
YES — off-distribution domain + volume justifies the retrain cost
NO — no (query, relevant-doc) pairs to train/eval on

finetune_embeddings.py# Illustrative — requires sentence-transformers + a GPU + labeled pairs. Does NOT run offline.
from sentence_transformers import SentenceTransformer, InputExample, losses
from torch.utils.data import DataLoader

model = SentenceTransformer("all-MiniLM-L6-v2")          # base embedder
train = [InputExample(texts=[q, pos]) for q, pos in your_query_doc_pairs]
loader = DataLoader(train, batch_size=32, shuffle=True)
loss = losses.MultipleNegativesRankingLoss(model)        # pulls (query, relevant) closer
model.fit(train_objectives=[(loader, loss)], epochs=1)
model.save("domain-embedder")                            # then re-embed your corpus

6 · Smarter chunking — beyond fixed size professional

Fixed-size chunking ("every 500 tokens") is blind: it slices mid-sentence, splits a table from its header, and separates a claim from its evidence — manufacturing the ambiguous chunks from section 1. Structure-aware chunking respects document boundaries (headings, paragraphs, list items, code blocks); semantic chunking starts a new chunk when the topic shifts. Below we model a structure-aware splitter that never breaks a paragraph and merges tiny fragments up to a target size.

chunking.pydef fixed_size(text, size=40):
    """Naive: cut every `size` characters, ignoring meaning."""
    return [text[i:i+size] for i in range(0, len(text), size)]

def structure_aware(text, target=60):
    """Split on paragraph boundaries; merge small paras up to ~target chars,
    but never cut a paragraph in half."""
    paras = [para.strip() for para in text.split("\n\n") if para.strip()]
    chunks, buf = [], ""
    for para in paras:
        if buf and len(buf) + len(para) + 1 > target:
            chunks.append(buf); buf = para
        else:
            buf = (buf + " " + para).strip()
    if buf:
        chunks.append(buf)
    return chunks

doc = ("Refund policy overview.\n\n"
       "Refunds are issued within 30 days.\n\n"
       "Contact support for exceptions.")

fx = fixed_size(doc)
sa = structure_aware(doc)
print("fixed-size chunks :", len(fx), "| first:", repr(fx[0]))
print("structure chunks  :", len(sa))
for c in sa:
    print("  -", c)

fixed-size chunks : 3 | first: 'Refund policy overview.\n\nRefunds are iss'
structure chunks  : 2
  - Refund policy overview. Refunds are issued within 30 days.
  - Contact support for exceptions.

The fixed-size cut slices straight through the middle of "issued" (…Refunds are iss) — the exact ambiguous-chunk problem, mid-word and mid-thought. The structure-aware splitter keeps whole paragraphs together and merges the tiny intro line up into a coherent chunk. Better boundaries mean less need for the context line to rescue meaning later.

7 · Measure it — recall@k before and after professional

Every technique here is a hypothesis; prove it with an eval. The workhorse metric for retrieval is recall@k: of the queries whose known-relevant document was retrieved in the top-k, what fraction? (Covered as an eval discipline in FA5 · RAG evaluation.) Compare the naive pipeline against the advanced one on the same query set — if recall@k doesn't move, the added complexity isn't earning its keep.

recall_at_k.pydef recall_at_k(results, relevant, k):
    """results: {query: ranked doc-id list}. relevant: {query: gold doc-id}.
    Returns fraction of queries whose gold doc is in the top-k."""
    hits = sum(1 for q, gold in relevant.items() if gold in results[q][:k])
    return round(hits / len(relevant), 2)

relevant = {"q1": "d_reimburse", "q2": "d_err5021", "q3": "d_refund"}

# Naive dense-only top-k misses the paraphrase and the exact error code.
naive = {"q1": ["d_shipping", "d_hours", "d_reimburse"],
         "q2": ["d_refund", "d_hours", "d_shipping"],
         "q3": ["d_refund", "d_reimburse", "d_hours"]}

# Advanced (contextual + hybrid + rerank) surfaces the right doc first.
advanced = {"q1": ["d_reimburse", "d_shipping", "d_hours"],
            "q2": ["d_err5021", "d_refund", "d_hours"],
            "q3": ["d_refund", "d_reimburse", "d_hours"]}

print("recall@1 naive    :", recall_at_k(naive, relevant, k=1))
print("recall@1 advanced :", recall_at_k(advanced, relevant, k=1))
print("recall@3 naive    :", recall_at_k(naive, relevant, k=3))
print("recall@3 advanced :", recall_at_k(advanced, relevant, k=3))

recall@1 naive    : 0.33
recall@1 advanced : 1.0
recall@3 naive    : 0.67
recall@3 advanced : 1.0

At k=1 the naive pipeline finds the right doc for only 1 of 3 queries; the advanced pipeline gets all three. The gap is largest at small k — exactly where it matters, because the LLM only reads the top few. Without this measurement you would be guessing whether the extra stages help.

8 · Tech-lead — designing the retrieval pipeline tech-lead

A lead owns the whole pipeline as a system, not a bag of tricks. The disciplined build order is: get chunking right (structure-aware), add a context line per chunk at ingestion, index for both dense and BM25, fuse with RRF, rerank the shortlist with a cross-encoder, and gate every change on recall@k over a fixed eval set. Fine-tune embeddings only if a measured domain gap survives all of that. Choose the vector store and framework that make this composition cheap to run and evolve (AE7 · Vector DB frameworks).

The build order a lead enforces

1. Chunk by structure — never cut mid-thought (chunking.py).
2. Contextualize — prepend a situating line before embedding (contextual_chunk.py).
3. Dual-index — embed for dense search and build a BM25 index over the same contextualized text.
4. Fuse — retrieve wide from both, combine ranks with RRF (hybrid_rrf.py).
5. Rerank — cross-encoder narrows the wide set to a precise top-k (rerank_stub.py).
6. Measure — recall@k before/after on a fixed query set (recall_at_k.py); only keep stages that move it.
7. Fine-tune embeddings — last resort, only for a proven off-distribution domain gap.

def contextualize(doc_title, section, chunk):
    context = f"This chunk is from '{doc_title}', section '{section}'."
    return context + " " + chunk

raw = "It grew 3% quarter over quarter."
print(contextualize("FY24 Report", "Revenue", raw))
# This chunk is from 'FY24 Report', section 'Revenue'. It grew 3% quarter over quarter.

def rrf(rankings, k=60):
    scores = {}
    for ranked in rankings:
        for rank, doc in enumerate(ranked):
            scores[doc] = scores.get(doc, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=lambda d: scores[d], reverse=True)

dense  = ["d2", "d1", "d5"]   # by meaning
sparse = ["d5", "d3", "d2"]   # by keyword
print(rrf([dense, sparse]))
# d2 first: ranks high in one list AND appears in both -> floats to the top

def first_stage(query, corpus):                 # cheap: raw keyword frequency
    q = query.lower().split()
    scored = [(d, sum(t.lower().split().count(w) for w in q)) for d,t in corpus]
    return [d for d,_ in sorted(scored, key=lambda x:x[1], reverse=True)]

def cross_encoder(query, text):                  # richer: fraction of query terms covered
    q, t = set(query.lower().split()), set(text.lower().split())
    return len(q & t) / len(q)                    # stand-in for a real cross-encoder

corpus = [
    ("d1", "refund refund refund refund process"),        # spams 'refund', misses 'window'
    ("d2", "how to enable two factor authentication"),
    ("d3", "the refund window is five business days"),     # actually answers the query
]
wide = first_stage("refund window", corpus)      # wide, imprecise
texts = dict(corpus)
reranked = sorted(wide, key=lambda d: cross_encoder("refund window", texts[d]),
                  reverse=True)
print("first-stage:", wide)      # ['d1', 'd3', 'd2'] — repetition inflates d1
print("reranked   :", reranked)  # ['d3', 'd1', 'd2'] — d3 promoted (covers both terms)

def recall_at_k(retrieved, relevant, k):
    return len(set(retrieved[:k]) & set(relevant)) / len(set(relevant))

evalset = [
    # query id: (naive top-k ids, hybrid+rerank top-k ids, relevant ids)
    ("q1", ["d5","d2","d9"], ["d1","d5","d2"], ["d1"]),
    ("q2", ["d3","d7","d8"], ["d3","d4","d7"], ["d4"]),
]
def mean_recall(idx, k=3):
    return sum(recall_at_k(row[idx], row[3], k) for row in evalset) / len(evalset)

print(f"naive  recall@3: {mean_recall(1):.0%}")   # 0%  (missed both)
print(f"hybrid recall@3: {mean_recall(2):.0%}")   # 100% (both in top-3)

def should_finetune_embeddings(domain_vocab_gap, have_labeled_pairs,
                               tried_hybrid_and_rerank, volume_high):
    if not tried_hybrid_and_rerank:
        return "NO — exhaust contextual chunking + hybrid + rerank first (cheaper)"
    if not have_labeled_pairs:
        return "NO — no (query, relevant-doc) pairs to train/eval on"
    if domain_vocab_gap and volume_high:
        return "YES — off-distribution domain + volume justifies the retrain cost"
    return "MAYBE — measure recall@k gain vs cost before committing"

print(should_finetune_embeddings(True, False, True, True))   # NO — no pairs
print(should_finetune_embeddings(True, True, True, True))    # YES

def design_pipeline(ambiguous_chunks, vocab_mismatch, exact_term_misses):
    stages = ["dense retrieval (baseline)"]
    if ambiguous_chunks:
        stages.insert(0, "contextual chunking (situate before embed)")
    if exact_term_misses:
        stages.append("sparse/BM25 + RRF fusion (catch exact tokens)")
    elif vocab_mismatch:
        stages.append("hybrid dense+sparse + RRF (bridge vocabulary)")
    stages.append("cross-encoder rerank -> precise top-k")
    stages.append("measure recall@k before/after")
    return stages

for s in design_pipeline(ambiguous_chunks=True, vocab_mismatch=True,
                         exact_term_misses=True):
    print("-", s)

Knowledge check check yourself