Advanced retrieval

1 · Why naive top-k retrieval fails essential

You've probably shipped this exact bug. You build a RAG demo, drop in a hundred documents, ask it a handful of obvious questions, and it answers beautifully. You ship it. Then a real user types "why was my ERR_5021 charge not reimbursed?" — and the assistant confidently answers about shipping windows, because the one chunk that held the answer never made it into the top-k. Nothing crashed. There's no stack trace. The retriever just quietly handed the LLM the wrong paragraphs, and a fluent model wrote a fluent wrong answer on top of them. That silent failure is the most expensive bug in RAG, because it looks like a model problem when it's really a retrieval problem.

Here's the mental model to hold onto: an LLM in a RAG system can only be as good as the chunks it's fed. Generation quality is capped by retrieval quality — no prompt-engineering trick rescues an answer that was built from the wrong evidence. So when RAG "hallucinates," the first place to look is almost never the prompt; it's whether the right chunk was even in the context window. Think of the retriever as the researcher and the LLM as the writer: a brilliant writer handed the wrong files still produces a confident, wrong report.

Naive top-k retrieval fails for a reason worth naming precisely. It embeds each chunk as a single point in vector space and, at query time, returns whichever chunks sit nearest the query point. That geometry rewards overall topical similarity and is blind to three things that matter constantly in real corpora: a chunk that lost its context when it was sliced off (so "it grew 3%" no longer says what grew), a query that uses different words than the document for the same idea (dense vectors drift apart when "refund" meets "reimbursement"), and rare exact tokens like IDs or error codes that carry almost no semantic weight yet are the whole point of the query. The common mistake is to treat these as edge cases you can prompt around. They aren't — they're structural, and the rest of this lesson is a pipeline built specifically to defeat each one. Read the table below as the three enemies, then read the two diagrams as the two-front war (ingestion and query time) you wage against them.

Basic RAG splits documents into fixed-size chunks, embeds each one, and at query time returns the top-k chunks whose embedding is closest to the query embedding. That works for easy questions and quietly fails for hard ones. Three failure modes dominate:

The fix is not one trick but a pipeline: situate each chunk before you embed it, retrieve with two methods that fail differently, fuse their rankings, then rerank the survivors with a model that actually reads query and chunk together. Here is the whole shape:

And the ingestion path that feeds it — the key move is contextual chunking, where each raw chunk gets a one-line situating summary prepended before it is embedded and indexed:

2 · Contextual retrieval — situate the chunk essential

Imagine tearing a single sentence out of a 40-page annual report and handing it to a stranger. The sentence reads: "It grew 3% quarter over quarter." Grew what? Which region, which year, which quarter? You know, because you saw the heading two pages up — but the chunk doesn't carry that heading. When the retriever embeds this orphaned sentence, it embeds a fact with no subject, and a query like "Acme EMEA revenue growth" sails right past it because they share literally zero words. This is the ambiguous-chunk failure from section 1, and it's not rare: it happens to most chunks in most long documents, because meaning in prose is spread across paragraphs, not packed into each one.

The fix is almost embarrassingly simple, and it's worth understanding why it works before you see the code. Think of it like adding a return address to an envelope. The letter inside (the raw chunk) is unchanged, but now anyone sorting the mail can see where it came from. Contextual retrieval — a technique published by Anthropic — does exactly this at ingestion time: for every chunk, it generates one short sentence saying where the chunk sits ("This is from the FY24 report, EMEA revenue section"), and prepends that line to the chunk before embedding it. The stored, embedded text becomes "context line + original chunk". Both the dense embedding and the keyword index now see the document's real vocabulary — "Acme", "EMEA", "revenue" — glued onto the sentence that was previously anonymous.

Why does prepending words change retrieval so much? Because a retriever can only match on signal that's present in the chunk. An embedding is a summary of the tokens it was shown; if the situating terms were never in the chunk, no amount of clever querying recovers them. By pulling those terms into the chunk itself, contextual retrieval raises the chunk's overlap with the very queries it should answer — which is precisely what the lab below measures, going from 0% word overlap to 75% on the same query. In production the context line is written by a cheap LLM pass over each chunk (Anthropic pairs this with prompt caching so it stays affordable); the lab templates it so it runs offline. The common mistake here is skipping this step because it feels like "just adding boilerplate" — but it is often the single highest-return change in the whole pipeline, and Anthropic reported it cut retrieval failures substantially before any reranking was added.

A chunk ripped out of a 40-page document often can't stand on its own. Contextual retrieval (the Anthropic technique) fixes this at ingestion time: for each chunk, generate a short sentence that says where this chunk sits in the document, and prepend it before embedding. The stored text becomes "context line + original chunk", so both the embedding and the BM25 index capture what the chunk is actually about. Below we model the transform with plain string work and show how it changes token overlap with the query.

contextual_chunk.pydef contextualize(doc_title, section, chunk):
    """Prepend a one-line situating summary before the chunk (Anthropic technique).
    In production the context line is written by a cheap LLM; here we template it."""
    context = f"This chunk is from '{doc_title}', section '{section}'."
    return context + " " + chunk

def overlap(query, text):
    """Fraction of query words that appear in the text (a crude retrieval proxy)."""
    q = set(query.lower().split())
    t = set(text.lower().replace("'", "").replace(".", "").split())
    return round(len(q & t) / len(q), 2)

raw = "It grew 3% quarter over quarter."
ctx = contextualize("Acme FY24 Report", "EMEA revenue", raw)
query = "acme emea revenue growth"

print("raw chunk   :", raw)
print("ctx chunk   :", ctx)
print("overlap raw :", overlap(query, raw))
print("overlap ctx :", overlap(query, ctx))

raw chunk   : It grew 3% quarter over quarter.
ctx chunk   : This chunk is from 'Acme FY24 Report', section 'EMEA revenue'. It grew 3% quarter over quarter.
overlap raw : 0.0
overlap ctx : 0.75

3 · Hybrid search — dense + sparse, fused with RRF intermediate

Here's a failure that makes engineers distrust embeddings entirely. You switch your search from old-fashioned keyword matching to shiny vector search, and paraphrase queries start working — "reimbursement" finds the "refund" doc, brilliant. Then a user pastes an error code, "ERR_5021", and vector search returns three vaguely-related help articles and completely misses the one page that literally contains "ERR_5021". You've traded one blind spot for another. The instinct is to pick a winner — dense or sparse — but that's the wrong frame. The right realization is that these two methods fail on different queries, which means you should stop choosing and run both.

Understand why they're complementary and the whole design falls out. Dense (embedding) search maps text to a point in meaning-space, so it shines on paraphrase and concept — but a rare literal token like an error code barely moves that point, so exact-match queries slip through. Sparse search — classically BM25, a battle-tested score built on how often query words appear in a document, weighted so rare words count more and long documents don't win by sheer size — is the mirror image: it nails exact tokens and IDs and misses paraphrase entirely, because to BM25 "refund" and "reimbursement" are unrelated strings. One is strong exactly where the other is weak. Running both and merging their results is hybrid search, and it's the default in serious retrieval stacks for this reason.

That leaves one real problem: how do you merge two ranked lists? You can't just add their scores — a cosine similarity of 0.82 and a BM25 score of 14.3 live on incompatible scales, and normalizing them is fiddly and fragile. The elegant escape is to throw the scores away and keep only the ranks. Reciprocal Rank Fusion (RRF) scores each document by summing 1/(k + rank) across every list it appears in (rank counted from 0, k a small constant like 60). A document that ranks near the top of either list scores well; a document that both retrievers agree on gets two contributions and floats even higher. Because it only ever looks at position, RRF needs no calibration between the two scoring systems — which is exactly why it's the go-to fusion method. The common mistake it prevents: hand-tuning a weighted blend of raw dense and sparse scores, then re-tuning it every time you swap an embedding model. RRF sidesteps that maintenance trap entirely.

Dense (embedding) search captures meaning but misses exact tokens; sparse search — classically BM25, a keyword-frequency score — nails exact terms but misses paraphrase. They fail on different queries, so run both and combine. The standard combiner is Reciprocal Rank Fusion (RRF): score each document by the sum of 1/(k + rank) across the two ranked lists. It needs no score calibration — only ranks — which is why it is the default fusion in most hybrid stacks.

hybrid_rrf.pydef rrf(rankings, k=60):
    """rankings: list of ranked lists (each is doc-ids best-first). Returns fused
    scores. RRF adds 1/(k + rank) for each list a doc appears in; rank is 0-based."""
    scores = {}
    for ranked in rankings:
        for rank, doc in enumerate(ranked):
            scores[doc] = scores.get(doc, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=lambda d: scores[d], reverse=True)

# A "dense-ish" ranking (by meaning) and a keyword (BM25-ish) ranking disagree.
dense  = ["d_reimburse", "d_shipping", "d_refund", "d_hours"]   # got paraphrase
sparse = ["d_refund", "d_err5021", "d_reimburse", "d_shipping"]  # got exact terms

fused = rrf([dense, sparse])
print("dense top1 :", dense[0])
print("sparse top1:", sparse[0])
print("fused order:", fused[:3])

dense top1 : d_reimburse
sparse top1: d_refund
fused order: ['d_reimburse', 'd_refund', 'd_shipping']

Notice d_reimburse (only dense found it) and d_refund (both found it, ranked high by sparse) both surface at the top — RRF rewards documents that either rank rank highly in one list or appear in both. No embedding-vs-BM25 score normalization was needed.

4 · Reranking — a cross-encoder narrows a wide recall set advanced

You've now got wide recall — and a new problem. Hybrid search reliably drags the right chunk into the candidate set, but often at rank 8 or rank 20, buried under near-misses. You can't hand the LLM twenty chunks: you'll blow the context window, spike the cost, and dilute the good evidence with noise. So you're stuck between two bad options — trust the first-stage ordering (which was optimized to not lose the answer, not to rank it first) or crank up top-k and pay for it. Reranking is the way out, and it rests on a distinction worth burning in: recall vs precision. First-stage retrieval optimizes recall (is the answer somewhere in the net?); reranking optimizes precision (are the top few actually the best?).

The reason a reranker can do what first-stage search can't comes down to architecture, and it's the key idea of this section. Your embedding retriever is a bi-encoder: it encodes the query and each document separately, into two independent vectors, and compares them with a cheap distance. That separation is what makes it fast enough to scan millions of docs — but it also means the model never actually reads the query and the document side by side; it only compares two summaries made in isolation. A cross-encoder does the opposite: it feeds the query and one candidate into the model together, in a single forward pass, so every word of the query can attend to every word of the document. It sees phrase matches, negations, and "this answers that" relationships a bi-encoder structurally cannot. The catch is symmetric: because it must run once per (query, document) pair, it's far too slow to run over the whole corpus.

Which is why the pattern is always retrieve wide, rerank narrow: use the cheap bi-encoder (plus BM25) to pull a shortlist of maybe 50 candidates from millions, then spend the expensive cross-encoder only on those 50 to pick the true top few. Think of it as a hiring funnel — a cheap résumé keyword filter to get a longlist, then expensive in-depth interviews only for the shortlist. You'd never interview every applicant, and you'd never hire off keywords alone. The common mistakes are the two ends of that dial: rerank a shortlist that's too small and the answer was already dropped before the good model ever saw it (you're just re-ranking garbage); rerank a set that's too large and latency and cost balloon. The width of the recall set is the knob you tune against your latency budget — and the lab below shows exactly how a reranker rescues a doc that keyword frequency had unfairly buried.

Retrieval optimizes for recall: cast a wide net (fetch, say, top-50) so the answer is somewhere in the set. But the LLM only wants the best few. A reranker — typically a cross-encoder that reads the query and each candidate together in one forward pass — scores relevance far more accurately than the bi-encoder embeddings used for first-stage recall. The pattern is retrieve wide, rerank to narrow: cheap recall over the whole corpus, expensive precision over the shortlist only.

rerank_stub.pydef first_stage_recall(query, corpus):
    """Cheap keyword-frequency proxy: count query-word occurrences. Repetition
    inflates the score (fast, wide, imprecise) — like a first-stage BM25 net."""
    q = query.lower().split()
    scored = [(doc, sum(text.lower().split().count(w) for w in q)) for doc, text in corpus]
    return [doc for doc, _ in sorted(scored, key=lambda x: x[1], reverse=True)]

def cross_encoder_score(query, text):
    """Stub for a cross-encoder: richer than word-frequency. Rewards the exact
    phrase and penalizes length (a real model reads query+doc jointly)."""
    q = query.lower()
    score = 2.0 if q in text.lower() else len(set(q.split()) & set(text.lower().split()))
    return score - 0.01 * len(text.split())   # slight brevity preference

def rerank(query, candidates, corpus_text):
    return sorted(candidates, key=lambda d: cross_encoder_score(query, corpus_text[d]),
                  reverse=True)

corpus = [
    ("d1", "refund refund refund refund policy and refund window notes"),
    ("d2", "shipping times and delivery windows for your orders"),
    ("d3", "how to request a refund from the orders page"),
]
text = dict(corpus)
query = "request a refund"

recall = first_stage_recall(query, corpus)   # wide, keyword-frequency biased
final  = rerank(query, recall, text)         # precise, phrase-aware
print("first-stage:", recall)
print("reranked   :", final)

first-stage: ['d1', 'd3', 'd2']
reranked   : ['d3', 'd1', 'd2']

First-stage recall put d1 on top because it repeats the word "refund" five times — keyword frequency, not relevance. The reranker reads the whole query against each doc, spots the exact phrase "request a refund" in d3, and promotes it. That is the recall-vs-precision division of labor: the first stage must not lose the answer; the reranker decides what the LLM actually sees.

5 · Embedding fine-tuning — adapt to your domain advanced

This is the section where ambitious teams burn a month for nothing. Retrieval is still a little soft, someone says "our domain is special, let's fine-tune the embedding model," and off they go collecting data, standing up GPUs, and retraining — only to discover afterward that structure-aware chunking and a reranker would have closed the same gap in an afternoon. Fine-tuning embeddings is a real and powerful tool, but it is the last lever you pull, not the first, and knowing when it's justified is worth more than knowing how to do it.

Start with why it can help at all. An off-the-shelf embedder learned its geometry from general web text, so it places words where the internet, on average, thinks they belong. If your world has its own private vocabulary — legal terms of art, drug names, internal product codes, ticket taxonomies — the general model may scatter items that you consider closely related far apart in vector space, because it never saw them used together. Fine-tuning on your own (query, relevant-document) pairs nudges the geometry: it pulls the things your users treat as matches closer together and pushes non-matches apart. Picture a librarian trained on the whole internet reshelving your specialist library after being shown which books your patrons actually request together — same books, better neighbourhood.

The reason it's the last resort is a plain cost-benefit truth. Fine-tuning demands labeled pairs (which you usually don't have and must build), an eval set to prove it helped, a training run, and — the part people forget — ongoing maintenance: every time your corpus or model shifts, you may need to retrain. Contextual chunking, hybrid search, and reranking are cheaper, faster to ship, and usually get you most of the way there. So the honest question isn't "should we fine-tune?" but "have we exhausted the cheap wins, do we have training data, and is our domain gap real and measured?" The lab below is literally a decision gate that asks exactly those questions in order — and its most common honest answer is "not yet." The mistake to avoid is treating fine-tuning as the prestige move that proves you're serious; the senior move is refusing to pay for it until the measurement demands it.

Off-the-shelf embeddings are trained on general web text. If your domain has its own vocabulary (legal, medical, internal product codes), a general embedder may place "related" documents far apart in vector space. Fine-tuning the embedding model on your own (query, relevant-doc) pairs pulls domain-related items closer together. It is powerful but the last lever to pull — contextual chunking, hybrid search, and reranking are cheaper and usually get you most of the way.

finetune_decision.pydef should_finetune_embeddings(domain_vocab_gap, have_labeled_pairs,
                              tried_hybrid_and_rerank, volume_high):
    if not tried_hybrid_and_rerank:
        return "NO — exhaust contextual chunking + hybrid + rerank first (cheaper)"
    if not have_labeled_pairs:
        return "NO — no (query, relevant-doc) pairs to train/eval on"
    if domain_vocab_gap and volume_high:
        return "YES — off-distribution domain + volume justifies the retrain cost"
    return "MAYBE — measure the recall gain vs the retrain/maintenance cost first"

print(should_finetune_embeddings(True, True, False, True))    # skipped the basics
print(should_finetune_embeddings(True, True, True, True))      # the real case
print(should_finetune_embeddings(True, False, True, True))     # no data

NO — exhaust contextual chunking + hybrid + rerank first (cheaper)
YES — off-distribution domain + volume justifies the retrain cost
NO — no (query, relevant-doc) pairs to train/eval on

finetune_embeddings.py# Illustrative — requires sentence-transformers + a GPU + labeled pairs. Does NOT run offline.
from sentence_transformers import SentenceTransformer, InputExample, losses
from torch.utils.data import DataLoader

model = SentenceTransformer("all-MiniLM-L6-v2")          # base embedder
train = [InputExample(texts=[q, pos]) for q, pos in your_query_doc_pairs]
loader = DataLoader(train, batch_size=32, shuffle=True)
loss = losses.MultipleNegativesRankingLoss(model)        # pulls (query, relevant) closer
model.fit(train_objectives=[(loader, loss)], epochs=1)
model.save("domain-embedder")                            # then re-embed your corpus

6 · Smarter chunking — beyond fixed size professional

Notice we saved the most upstream step for late — that's deliberate. Every failure in section 1 has a root cause, and more often than not it's a decision you made at the very first step of ingestion: where do I cut the document into pieces? The default answer nearly everyone starts with is "every 500 tokens," and it's blind by construction. A fixed-size cutter can't see meaning, so it slices straight through the middle of a sentence, guillotines a table away from the header that explains its columns, and strands a claim in one chunk while its supporting evidence lands in the next. In other words, fixed-size chunking is a factory for the ambiguous chunks you spent section 2 rescuing with context lines. Fix the boundaries and you manufacture fewer broken chunks in the first place — cheaper than patching them downstream.

The better idea is to cut where the document already tells you it's safe to cut. Structure-aware chunking respects the boundaries the author put there — headings, paragraphs, list items, code blocks, table rows — so a chunk is a whole thought rather than an arbitrary 500-token window. Because tiny fragments (a one-line heading, a short intro sentence) make weak standalone chunks, a good structure-aware splitter also merges small neighbours up to a target size, but it will never split a paragraph in half to hit that target. Think of it as cutting along the perforations instead of taking scissors to the middle of the page: the pieces come apart cleanly, each one intact.

A step further is semantic chunking, which starts a new chunk when the topic shifts — detected by watching how the embedding of successive sentences drifts — rather than relying on layout markers. It's more powerful for prose that lacks clear structure, but also more expensive and more failure-prone, so structure-aware is the sensible default and semantic is the reach-for-it-when-layout-fails upgrade. The common mistake with all of this is chasing one universal chunk size: the right boundary depends on your documents and your queries, and the target size is a genuine trade-off — bigger chunks keep more context together but retrieve less precisely, smaller chunks are precise but lose surrounding meaning. The lab below makes the failure vivid: watch a fixed-size cut land mid-word in "issued", then watch the structure-aware splitter keep whole paragraphs together.

Fixed-size chunking ("every 500 tokens") is blind: it slices mid-sentence, splits a table from its header, and separates a claim from its evidence — manufacturing the ambiguous chunks from section 1. Structure-aware chunking respects document boundaries (headings, paragraphs, list items, code blocks); semantic chunking starts a new chunk when the topic shifts. Below we model a structure-aware splitter that never breaks a paragraph and merges tiny fragments up to a target size.

chunking.pydef fixed_size(text, size=40):
    """Naive: cut every `size` characters, ignoring meaning."""
    return [text[i:i+size] for i in range(0, len(text), size)]

def structure_aware(text, target=60):
    """Split on paragraph boundaries; merge small paras up to ~target chars,
    but never cut a paragraph in half."""
    paras = [para.strip() for para in text.split("\n\n") if para.strip()]
    chunks, buf = [], ""
    for para in paras:
        if buf and len(buf) + len(para) + 1 > target:
            chunks.append(buf); buf = para
        else:
            buf = (buf + " " + para).strip()
    if buf:
        chunks.append(buf)
    return chunks

doc = ("Refund policy overview.\n\n"
       "Refunds are issued within 30 days.\n\n"
       "Contact support for exceptions.")

fx = fixed_size(doc)
sa = structure_aware(doc)
print("fixed-size chunks :", len(fx), "| first:", repr(fx[0]))
print("structure chunks  :", len(sa))
for c in sa:
    print("  -", c)

fixed-size chunks : 3 | first: 'Refund policy overview.\n\nRefunds are iss'
structure chunks  : 2
  - Refund policy overview. Refunds are issued within 30 days.
  - Contact support for exceptions.

The fixed-size cut slices straight through the middle of "issued" (…Refunds are iss) — the exact ambiguous-chunk problem, mid-word and mid-thought. The structure-aware splitter keeps whole paragraphs together and merges the tiny intro line up into a coherent chunk. Better boundaries mean less need for the context line to rescue meaning later.

7 · Measure it — recall@k before and after professional

Everything so far has been a claim: "this stage helps." Contextual chunking helps. Hybrid search helps. Reranking helps. But helps is a feeling, and feelings ship regressions. The engineers who lose the most time in retrieval are the ones who bolt on technique after technique — RRF, a reranker, a fancier chunker — because a blog post said to, never measuring, until the pipeline is a slow, expensive stack of cargo-culted parts and nobody can say which one is actually earning its keep. The discipline that separates that from real engineering is embarrassingly simple: measure the same query set before and after every change.

The workhorse metric is recall@k, and its definition is exactly as blunt as it sounds. Take a fixed set of evaluation queries, each with a known "gold" document that should be retrieved. Run them through your pipeline, look at the top-k results, and ask: for what fraction of queries did the gold doc land in the top-k? Recall@1 = 0.33 means only a third of your queries got the right doc into the very top slot; recall@3 = 1.0 means all of them made the top three. It's a fraction between 0 and 1, higher is better, and it maps directly onto the thing that matters — because the LLM only reads the top few chunks, recall@k is "did the answer even reach the model?"

Two habits make this metric trustworthy. First, hold the query set fixed and compare pipelines on identical inputs — otherwise you're measuring the questions, not the system. Second, watch recall at small k especially closely, because that's where the LLM actually lives; a gain at k=50 is cold comfort if you only feed the model three chunks. The payoff is that you can now attribute improvement: add one stage, re-measure, and if recall@k doesn't move, that stage isn't paying for its latency and complexity and should be cut. This is also how you catch regressions before your users do — a config change that quietly drops recall@1 shows up as a number, not as a support ticket three weeks later. (Recall@k is developed further as an eval discipline in FA5 · RAG evaluation.) The lab below runs the exact before/after that turns "it feels better" into "recall@1 went from 0.33 to 1.0."

Every technique here is a hypothesis; prove it with an eval. The workhorse metric for retrieval is recall@k: of the queries whose known-relevant document was retrieved in the top-k, what fraction? (Covered as an eval discipline in FA5 · RAG evaluation.) Compare the naive pipeline against the advanced one on the same query set — if recall@k doesn't move, the added complexity isn't earning its keep.

recall_at_k.pydef recall_at_k(results, relevant, k):
    """results: {query: ranked doc-id list}. relevant: {query: gold doc-id}.
    Returns fraction of queries whose gold doc is in the top-k."""
    hits = sum(1 for q, gold in relevant.items() if gold in results[q][:k])
    return round(hits / len(relevant), 2)

relevant = {"q1": "d_reimburse", "q2": "d_err5021", "q3": "d_refund"}

# Naive dense-only top-k misses the paraphrase and the exact error code.
naive = {"q1": ["d_shipping", "d_hours", "d_reimburse"],
         "q2": ["d_refund", "d_hours", "d_shipping"],
         "q3": ["d_refund", "d_reimburse", "d_hours"]}

# Advanced (contextual + hybrid + rerank) surfaces the right doc first.
advanced = {"q1": ["d_reimburse", "d_shipping", "d_hours"],
            "q2": ["d_err5021", "d_refund", "d_hours"],
            "q3": ["d_refund", "d_reimburse", "d_hours"]}

print("recall@1 naive    :", recall_at_k(naive, relevant, k=1))
print("recall@1 advanced :", recall_at_k(advanced, relevant, k=1))
print("recall@3 naive    :", recall_at_k(naive, relevant, k=3))
print("recall@3 advanced :", recall_at_k(advanced, relevant, k=3))

recall@1 naive    : 0.33
recall@1 advanced : 1.0
recall@3 naive    : 0.67
recall@3 advanced : 1.0

At k=1 the naive pipeline finds the right doc for only 1 of 3 queries; the advanced pipeline gets all three. The gap is largest at small k — exactly where it matters, because the LLM only reads the top few. Without this measurement you would be guessing whether the extra stages help.

8 · Tech-lead — designing the retrieval pipeline tech-lead

A junior engineer collects retrieval tricks; a lead sequences them. The difference shows up in a real conversation: someone proposes "let's add a reranker," and the lead's first question isn't "which reranker?" but "have we fixed chunking and added context lines yet, and what does recall@k say before we spend that latency?" Everything in this lesson is a component, and a pile of components isn't a system — the value is in the order you assemble them and the gate you put on each addition. Owning that order, and defending it, is the tech-lead's actual job here.

The order isn't arbitrary; it follows the flow of the data and the cost of the fix. Get chunking right first, because it's the most upstream decision and it silently sets the ceiling on everything downstream — bad boundaries manufacture broken chunks no later stage fully repairs. Then add a context line per chunk so each one can stand alone. Index that contextualized text for both dense and BM25 so you have two retrievers that fail differently. Fuse their rankings with RRF for wide recall, then rerank the shortlist with a cross-encoder for precision. And gate every one of those additions on recall@k over a fixed eval set: a stage that doesn't move the number gets cut, no matter how fashionable it is. Fine-tuning embeddings sits dead last — only if a measured domain gap survives all of the above.

The through-line a lead keeps repeating to the team is the reason any of this matters: retrieval quality caps everything downstream. The most eloquent prompt in the world can't fix an answer built from the wrong chunks, so retrieval is the highest-leverage part of a RAG system — and it deserves to be treated as an engineered, staged, measured subsystem rather than a bag of tricks bolted on by vibes. The final piece of the lead's judgment is picking the vector store and framework that make this whole composition cheap to run and cheap to evolve, because the pipeline you ship this quarter is the one you'll be tuning next quarter (AE7 · Vector DB frameworks). The build-order list below is the artifact a lead actually enforces in review — read it as the sequence, and the reasons above as the why behind each step.

A lead owns the whole pipeline as a system, not a bag of tricks. The disciplined build order is: get chunking right (structure-aware), add a context line per chunk at ingestion, index for both dense and BM25, fuse with RRF, rerank the shortlist with a cross-encoder, and gate every change on recall@k over a fixed eval set. Fine-tune embeddings only if a measured domain gap survives all of that. Choose the vector store and framework that make this composition cheap to run and evolve (AE7 · Vector DB frameworks).

The build order a lead enforces

1. Chunk by structure — never cut mid-thought (chunking.py).
2. Contextualize — prepend a situating line before embedding (contextual_chunk.py).
3. Dual-index — embed for dense search and build a BM25 index over the same contextualized text.
4. Fuse — retrieve wide from both, combine ranks with RRF (hybrid_rrf.py).
5. Rerank — cross-encoder narrows the wide set to a precise top-k (rerank_stub.py).
6. Measure — recall@k before/after on a fixed query set (recall_at_k.py); only keep stages that move it.
7. Fine-tune embeddings — last resort, only for a proven off-distribution domain gap.

def contextualize(doc_title, section, chunk):
    context = f"This chunk is from '{doc_title}', section '{section}'."
    return context + " " + chunk

raw = "It grew 3% quarter over quarter."
print(contextualize("FY24 Report", "Revenue", raw))
# This chunk is from 'FY24 Report', section 'Revenue'. It grew 3% quarter over quarter.

def rrf(rankings, k=60):
    scores = {}
    for ranked in rankings:
        for rank, doc in enumerate(ranked):
            scores[doc] = scores.get(doc, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=lambda d: scores[d], reverse=True)

dense  = ["d2", "d1", "d5"]   # by meaning
sparse = ["d5", "d3", "d2"]   # by keyword
print(rrf([dense, sparse]))
# d2 first: ranks high in one list AND appears in both -> floats to the top

def first_stage(query, corpus):                 # cheap: raw keyword frequency
    q = query.lower().split()
    scored = [(d, sum(t.lower().split().count(w) for w in q)) for d,t in corpus]
    return [d for d,_ in sorted(scored, key=lambda x:x[1], reverse=True)]

def cross_encoder(query, text):                  # richer: fraction of query terms covered
    q, t = set(query.lower().split()), set(text.lower().split())
    return len(q & t) / len(q)                    # stand-in for a real cross-encoder

corpus = [
    ("d1", "refund refund refund refund process"),        # spams 'refund', misses 'window'
    ("d2", "how to enable two factor authentication"),
    ("d3", "the refund window is five business days"),     # actually answers the query
]
wide = first_stage("refund window", corpus)      # wide, imprecise
texts = dict(corpus)
reranked = sorted(wide, key=lambda d: cross_encoder("refund window", texts[d]),
                  reverse=True)
print("first-stage:", wide)      # ['d1', 'd3', 'd2'] — repetition inflates d1
print("reranked   :", reranked)  # ['d3', 'd1', 'd2'] — d3 promoted (covers both terms)

def recall_at_k(retrieved, relevant, k):
    return len(set(retrieved[:k]) & set(relevant)) / len(set(relevant))

evalset = [
    # query id: (naive top-k ids, hybrid+rerank top-k ids, relevant ids)
    ("q1", ["d5","d2","d9"], ["d1","d5","d2"], ["d1"]),
    ("q2", ["d3","d7","d8"], ["d3","d4","d7"], ["d4"]),
]
def mean_recall(idx, k=3):
    return sum(recall_at_k(row[idx], row[3], k) for row in evalset) / len(evalset)

print(f"naive  recall@3: {mean_recall(1):.0%}")   # 0%  (missed both)
print(f"hybrid recall@3: {mean_recall(2):.0%}")   # 100% (both in top-3)

def should_finetune_embeddings(domain_vocab_gap, have_labeled_pairs,
                               tried_hybrid_and_rerank, volume_high):
    if not tried_hybrid_and_rerank:
        return "NO — exhaust contextual chunking + hybrid + rerank first (cheaper)"
    if not have_labeled_pairs:
        return "NO — no (query, relevant-doc) pairs to train/eval on"
    if domain_vocab_gap and volume_high:
        return "YES — off-distribution domain + volume justifies the retrain cost"
    return "MAYBE — measure recall@k gain vs cost before committing"

print(should_finetune_embeddings(True, False, True, True))   # NO — no pairs
print(should_finetune_embeddings(True, True, True, True))    # YES

def design_pipeline(ambiguous_chunks, vocab_mismatch, exact_term_misses):
    stages = ["dense retrieval (baseline)"]
    if ambiguous_chunks:
        stages.insert(0, "contextual chunking (situate before embed)")
    if exact_term_misses:
        stages.append("sparse/BM25 + RRF fusion (catch exact tokens)")
    elif vocab_mismatch:
        stages.append("hybrid dense+sparse + RRF (bridge vocabulary)")
    stages.append("cross-encoder rerank -> precise top-k")
    stages.append("measure recall@k before/after")
    return stages

for s in design_pipeline(ambiguous_chunks=True, vocab_mismatch=True,
                         exact_term_misses=True):
    print("-", s)

Knowledge check check yourself