Retrieval quality & re-ranking

1 · Why the first pass isn't enough

Here's the failure that sends teams looking for this chapter. Your RAG bot demos beautifully. You ask it a question, the right passage comes back, the answer is crisp. Then real users arrive with real questions, and the complaints trickle in: "it gave me the wrong answer," "it's confidently citing the wrong doc." You dig in, and the maddening part is this — the correct chunk was retrieved. It was sitting at rank 5. The model just never read that far, or it anchored on the noisy chunk that landed at rank 1. Nothing is broken. Retrieval did its job. It just wasn't careful enough about which of its good-enough hits went first.

Think of first-pass retrieval like a librarian who's brilliant at speed but only skims. Ask for "books about restarting a checkout service" and in seconds she pulls twenty books off the shelves that all mention those words — that's her superpower, she can scan the whole library. But she hasn't read any of them closely enough to know which one actually tells you the steps versus which one just name-drops "checkout" and "queue" in passing. To pick the single best book she'd have to sit down and read each candidate against your exact question — slow, careful work she can't afford to do across the whole library. That split — a fast skim over everything, then a careful read of the shortlist — is the entire architecture of this chapter.

The common mistake here is to blame the wrong thing. When the top answer is wrong, the instinct is "my embeddings are bad" or "I need a bigger model," and people go re-index the whole corpus. But usually the answer is already in the candidate set — recall is fine — and the real defect is ordering. That distinction is worth its own vocabulary, because you cannot fix a problem you've misdiagnosed: recall asks "did we even fetch the right chunk?", while precision asks "of the chunks we fetched, are the good ones on top?" Confuse them and you'll spend a week improving recall when your recall was never the bottleneck. The table below is the precise language for talking about retrieval quality — memorize these three, because every decision in the rest of the chapter comes back to which one you're actually short on.

First-pass retrieval — vector search or BM25 — is built for speed over a huge corpus. It embeds each document once and compares vectors independently, so it scales to millions of chunks. The price is precision: it finds things that are topically near the question, but it can't tell that chunk #5 actually answers the question while chunk #1 merely mentions the same words. That is a precision@k problem — the right chunk is somewhere in the top 20, just not at the top.

2 · Re-ranking with a cross-encoder

Everything good about your RAG answer rides on getting the right chunk to the top of the list. The model reads the first chunk most attentively and the last chunk second-most; anything crowded into the middle it may skim right past. So if the chunk that truly answers the question is at rank 5, it might as well not exist. Re-ranking is the one move that reliably fixes this — and understanding why it works comes down to a single difference in how two kinds of models look at text.

Here's the analogy that makes it click. Imagine speed-dating versus a real interview. First-pass retrieval is speed-dating: everyone fills out a profile card in advance (that's the embedding), and matching is just comparing cards — fast, and you can compare thousands of cards at once, but nobody actually talked to anybody. A cross-encoder is the real interview: it sits the query and one candidate chunk down in the same room, at the same time, and judges how well they actually respond to each other. Of course the interview is far more accurate — the two parties are genuinely interacting rather than being matched on pre-written cards. And of course it's far slower — you have to hold a fresh interview for every single candidate, and you can't prepare the answer ahead of time because it depends on this query paired with this chunk. That's the whole trade in one image: cards for scale, interviews for judgment.

In precise terms: a bi-encoder (what first-pass vector search uses) embeds the query and each chunk into vectors separately, then compares those vectors — so a chunk's vector can be computed once, at indexing time, and reused for every future query. That's why it scales to millions of chunks. But because the model never sees the query and the chunk side by side, it can only judge "are these two texts about similar topics?", not "does this text answer that question?" A cross-encoder feeds the pair (query, chunk) through the model jointly, letting every word of the query attend to every word of the chunk, and outputs one relevance score. That joint view is what catches "this chunk uses all the right words but doesn't actually give the steps."

The trap that ruins re-ranking setups is trying to run the cross-encoder on the whole corpus — "if it's more accurate, use it everywhere." You can't: it's not precomputable, so scoring a million chunks per query would take minutes and cost a fortune. The discipline is a two-stage funnel: let the cheap bi-encoder cast a wide net and hand you the top-N candidates (say 25–100), then spend the expensive cross-encoder only on rescoring those N. You get the bi-encoder's scale and the cross-encoder's judgment, and you only pay the careful cost on the tiny shortlist — never the whole library. The diagram and table below lay out that funnel; keep the "cheap-wide-then-careful-narrow" shape in mind as you read them.

Below is the whole idea, runnable. First-pass retrieval is a simple shared-term count (standing in for the bi-encoder) — which a keyword-stuffed chunk can game. Then a richer re-ranker rescores the top-N with signals the first pass ignored — term coverage plus exact phrase/bigram matches — and re-orders them. Watch the truly-relevant chunk climb from behind a noisy one to the top.

rerank.pyimport re
from collections import Counter

def toks(s):
    return re.findall(r'[a-z]+', s.lower())

# ---- FIRST PASS: cheap shared-term count (stands in for a bi-encoder) ----
def first_pass_score(query, chunk):
    q, c = Counter(toks(query)), Counter(toks(chunk))
    return sum(min(q[w], c[w]) for w in q)   # raw shared-term count

# ---- RE-RANKER: richer joint score (stands in for a cross-encoder) ----
# A real cross-encoder replaces this function; the pipeline shape is identical.
def rerank_score(query, chunk):
    cl = chunk.lower()
    qset = set(toks(query))
    coverage = sum(1 for w in qset if w in set(toks(chunk))) / (len(qset) or 1)
    bigram = 1.0 if 'consumer group' in cl else 0.0   # meaningful phrase, not just words
    phrase = 1.0 if 'restart checkout' in cl else 0.0  # answers the actual ask
    return round(2.0*coverage + bigram + phrase, 3)

chunks = [
    'Restart the queue, restart the group, restart the checkout queue consumer.',  # keyword-stuffed noise
    'To restart checkout, drain the queue then scale the consumer group up.',       # the real answer
    'Checkout reads orders; the queue and consumer group appear in Grafana.',
    'Never restart the database to fix a queue backlog.',
]
query = 'restart checkout consumer group queue'

# stage 1: first-pass top-N (cheap, favors recall)
N = 4
first = sorted(chunks, key=lambda c: first_pass_score(query, c), reverse=True)[:N]
print('FIRST PASS (top-4):')
for i, c in enumerate(first, 1):
    print(f'  {i}. [{first_pass_score(query, c)}] {c}')

# stage 2: re-rank ONLY those N (richer, favors precision)
reranked = sorted(first, key=lambda c: rerank_score(query, c), reverse=True)
print('\nAFTER RE-RANK:')
for i, c in enumerate(reranked, 1):
    print(f'  {i}. [{rerank_score(query, c)}] {c}')

FIRST PASS (top-4):
  1. [5] Restart the queue, restart the group, restart the checkout queue consumer.
  2. [5] To restart checkout, drain the queue then scale the consumer group up.
  3. [4] Checkout reads orders; the queue and consumer group appear in Grafana.
  4. [2] Never restart the database to fix a queue backlog.

AFTER RE-RANK:
  1. [4.0] To restart checkout, drain the queue then scale the consumer group up.
  2. [2.6] Checkout reads orders; the queue and consumer group appear in Grafana.
  3. [2.0] Restart the queue, restart the group, restart the checkout queue consumer.
  4. [0.8] Never restart the database to fix a queue backlog.

First-pass tied a keyword-stuffed noise chunk with the real answer (both score 5), and a plain sort left the noise at #1. The re-ranker — rewarding coverage plus the meaningful consumer group and restart checkout phrases rather than raw repeats — pushed the actionable answer to #1 and demoted the noise to #3. That is precision@k improving in one sort — and we only paid the richer cost on 4 candidates, not the whole corpus.

3 · Query transformation

Re-ranking fixes the order of what you found — but what if the answer was never found at all? No amount of rescoring can rescue a chunk that the first pass didn't retrieve. When recall itself is the problem, you have to attack it from the other end: not the results, but the query. And the reason queries fail is almost poetic — the person asking and the document answering rarely speak the same way.

Picture the mismatch. A stressed engineer types "checkout is stuck" into your bot. The runbook that solves it says "Drain the order queue, then scale the consumer group to resume processing." Not one important word overlaps — "stuck" vs. "drain," "checkout" vs. "queue" and "consumer group." To a keyword search these look unrelated, and even to an embedder they can drift apart, because the terse panic of a question is phrased nothing like the calm procedural prose of an answer. It's like searching a cookbook for "dinner is boring" — the recipe you need never uses those words. The fix is to stop searching with the raw question and instead search with something that looks more like the answer you're hoping to find.

That single insight branches into three published techniques, and the mistake to avoid is reaching for all three at once. They solve different failures, so you pick by diagnosis. Multi-query handles "the question could be phrased ten ways" — you generate several paraphrases and union their hits, widening recall cheaply. HyDE handles the question-vs-answer vocabulary gap directly — you have an LLM draft a fake answer and search with that, because an answer is phrased like your documents. Decomposition handles the compound question — "how do I restart checkout and where are invoices stored?" needs facts from two different chunks, so you split it into sub-questions and retrieve each. Match the technique to the symptom; don't bolt on all three and hope. The table names each precisely, and the runnable lab below builds multi-query so you can watch the union gather chunks no single query returned.

The other way to improve retrieval is to fix the query before you search. A user's question is often short, ambiguous, or phrased unlike the documents. Three published techniques rewrite it:

Here is a runnable multi-query simulation: several paraphrases each run first-pass retrieval, and we union + dedup the hits into one candidate set. In production the paraphrases come from an LLM; here we hand-write them so it runs offline.

multi_query.pyimport re
from collections import Counter

def toks(s):
    return re.findall(r'[a-z]+', s.lower())

def score(query, chunk):
    q, c = Counter(toks(query)), Counter(toks(chunk))
    return sum(min(q[w], c[w]) for w in q)

def retrieve(query, chunks, k=1):
    ranked = sorted(range(len(chunks)), key=lambda i: score(query, chunks[i]), reverse=True)
    return [i for i in ranked if score(query, chunks[i]) > 0][:k]

chunks = [
    'Restart checkout by draining the order queue.',
    'The vehicle will not crank when the battery is flat.',
    'Reboot the stuck payment worker to clear the consumer group.',
    'Invoices are stored in Postgres by the billing service.',
]

# One LLM would generate these paraphrases; we hand-write them for an offline run.
paraphrases = [
    'how to restart checkout queue',
    'reboot the payment worker consumer group',
    'clear a stuck checkout',
]

seen, union = set(), []          # dedup by chunk index, keep first-seen order
for pq in paraphrases:
    hits = retrieve(pq, chunks, k=1)
    print(f'{pq!r:45} -> chunks {hits}')
    for i in hits:
        if i not in seen:
            seen.add(i); union.append(i)

print('\nUNION (deduped):', union)
for i in union: print('  -', chunks[i])

'how to restart checkout queue'               -> chunks [0]
'reboot the payment worker consumer group'    -> chunks [2]
'clear a stuck checkout'                      -> chunks [2]

UNION (deduped): [0, 2]
  - Restart checkout by draining the order queue.
  - Reboot the stuck payment worker to clear the consumer group.

Each paraphrase, taking the single best hit, saw the question from a different angle: the first surfaced the “drain the queue” chunk, the other two both surfaced the “reboot the worker” chunk. The union gathered both good chunks that no single query returned together, and dedup kept the repeated one just once. That is multi-query in miniature: several angles on the question, merged. (RAG 3.2 covered fusing rankings with RRF; here we fuse sets from paraphrased queries — a complementary move.)

4 · Context packing — order, dedup, budget

You've done the hard work — you have the right chunks, in the right order. Here's where teams throw it away. They take those beautiful ranked results and just paste all of them into the prompt, in rank order, and assume the model will find the good parts. It often doesn't. The chunk that answers the question sits at position 6 of 12, buried in the middle of a long context, and the model glides right over it — while a redundant near-duplicate two slots up ate the budget that could have held a genuinely new fact. Retrieval succeeded; packing failed.

The mental model to unlearn is "context is a bucket — pour everything in." It's closer to arranging a one-page briefing for a busy executive who reads the top and bottom carefully and skims the middle. Where you place a fact matters as much as whether you include it. This isn't a metaphor for the model's laziness — it's a real, published effect called "lost in the middle," where language models reliably attend best to text at the start and end of a long context and measurably worse to what's stranded in the center. So the naive instinct — "put the single best chunk right in the middle where it's central" — is exactly backwards; the center is the weakest seat in the house.

Three forces fight over the prompt, and packing is the art of balancing them. First, lost-in-the-middle: position is not neutral, so the strongest chunks belong at the edges. Second, near-duplicates: retrieval loves to hand you the same fact phrased twice — overlapping chunks, the same doc indexed under two headings — and each copy burns budget while adding zero information, so you dedup before packing. Third, the token budget: the context window is finite and every token you spend costs latency and money, so past a point you truncate and drop the tail. The common failure is to optimize only the third (stuff in as much as fits) while ignoring the first two — you end up paying for a bloated, duplicate-ridden context whose best fact is hidden dead-center. The table names each force and its fix; the runnable packer below does all three at once so you can watch a near-duplicate get dropped and the second-best chunk migrate to the end.

You now have good candidates. The last decision is how to lay them into the prompt, and it is not “paste all of them.” Three forces shape the packing:

Here is a runnable packer that takes ranked chunks, drops near-duplicates, orders them so the best sit at the edges (fighting lost-in-the-middle), and truncates to a token budget. We approximate tokens as words to keep it offline; a real tokenizer swaps in without changing the shape.

pack.pyimport re

def approx_tokens(s):
    return len(s.split())          # a real tokenizer replaces this

def near_dup(a, b, thresh=0.8):
    wa, wb = set(re.findall(r'[a-z]+', a.lower())), set(re.findall(r'[a-z]+', b.lower()))
    if not wa or not wb: return False
    jaccard = len(wa & wb) / len(wa | wb)
    return jaccard >= thresh

def pack(ranked, budget=40):
    # ranked: chunks already sorted best -> worst (e.g. from the re-ranker)
    kept = []
    for c in ranked:                                   # 1) dedup near-duplicates
        if not any(near_dup(c, k) for k in kept):
            kept.append(c)
    trimmed, used = [], 0                              # 2) truncate to token budget
    for c in kept:
        t = approx_tokens(c)
        if used + t > budget: break
        trimmed.append(c); used += t
    # 3) order: best first, 2nd-best LAST, rest in the middle (fight lost-in-middle)
    ordered = trimmed[:]
    if len(ordered) >= 3:
        ordered = [trimmed[0]] + trimmed[2:] + [trimmed[1]]
    return ordered, used

ranked = [
    'Drain the queue then scale the consumer group to restart checkout.',
    'Drain the queue then scale the consumer group to restart the checkout.',  # near-dup of #0
    'Never restart the database to fix a queue backlog.',
    'Checkout orders flow through the queue into the consumer group.',
    'Grafana dashboards track queue depth and consumer lag over time.',
]
ordered, used = pack(ranked, budget=40)
print(f'packed {len(ordered)} chunks, ~{used} tokens:')
for i, c in enumerate(ordered, 1): print(f'  [{i}] {c}')

packed 4 chunks, ~40 tokens:
  [1] Drain the queue then scale the consumer group to restart checkout.
  [2] Checkout orders flow through the queue into the consumer group.
  [3] Grafana dashboards track queue depth and consumer lag over time.
  [4] Never restart the database to fix a queue backlog.

The near-duplicate second chunk (identical words) was dropped, the packer filled right up to the 40-token budget, and the second-strongest chunk was moved to the end so the two most important chunks sit at the edges where the model reads best. Same facts, no redundancy, better placement — the middle holds the weaker context that matters least if the model skims it.

5 · When re-ranking is worth the latency

Now the honest part, the one that separates engineers from technique-collectors. Every method in this chapter makes retrieval better — and every one of them costs you something: an extra model pass, sometimes an extra LLM call, tens to hundreds of milliseconds added to every single query. It is genuinely tempting, having just learned all this machinery, to bolt it onto everything. That's the trap. The mark of a senior engineer isn't knowing re-ranking exists; it's knowing when not to use it.

Think of it like airport security. A wrong "top result" at a border checkpoint or in a medical triage system is catastrophic, so you happily accept the slow, careful screening — precision is worth any reasonable delay. But you would never put full body-scan security on the turnstile into a subway, where a hundred thousand people a minute need to pass and the cost of an occasional wrong guess is trivial. Same tool, opposite verdict, and the deciding factor is the same two questions every time: how expensive is a wrong top answer, and how tight is the latency budget? High-stakes and slack latency → screen everything carefully. Low-stakes and high traffic → wave them through with cheap top-k.

There's also a diagnosis that overrides the whole cost-benefit question: re-ranking cannot fix a recall problem. If the answer never made it into the candidate set, no amount of rescoring conjures it back — you'd be spending latency to more carefully sort a list that doesn't contain the answer. So before you reach for a cross-encoder, confirm the chunk is actually in the top-N; if it isn't, you have a retrieval/recall problem and query transformation or better chunking is the real fix. And the closing discipline, the one that outlives any specific technique: measure, don't assume. The gains here are real but corpus-specific — run precision@k and MRR with and without each addition on your own questions before you ship it, or you may be paying latency for a number that never moved. The table below maps common situations to a verdict; read it as a decision aid, not a mandate to enable everything.

Re-ranking and query transformation both cost time — an extra model pass, sometimes an extra LLM call. That's a real budget you spend on every query. Spend it where it pays off: