Debug a broken RAG

The RAG failure decision tree essential

Picture the scene you'll actually walk into. A product manager forwards you three screenshots: one answer is confidently wrong, one is a hallucinated policy that doesn't exist, one just says "I don't have that information" for a question the handbook clearly covers. Slack is on fire, and the loudest voice in the channel says "the model is bad — can we try GPT-5 / a bigger Claude?" You reach for the model dial. Stop. Swapping the model is the most expensive, slowest, least likely fix, and nine times out of ten it changes nothing — because the model was never the broken part.

Here's the mental model that saves you. A RAG answer is the output of a pipeline, not a single black box. Text is chunked, each chunk is embedded, a query retrieves candidates, those candidates are ranked, and only the top few are pasted into a prompt the model then answers. That's five distinct stages — chunking, embedding, retrieval, ranking, generation — and a defect in any one of them surfaces at the very end as "the answer is wrong." The output looks identical whether the true chunk was never retrieved or was retrieved-but-ignored. That's precisely why "the model is bad" is such a seductive misdiagnosis: the symptom is downstream of five possible causes, and the model is only the last one.

Think of it like debugging a leak in a house with water pooling on the ground floor. You don't rip out the ground-floor ceiling first — you start at the roof and work down, because a leak upstream presents as damage downstream. RAG has the same property, and it has a name worth remembering: a fault upstream masquerades as a fault downstream. A chunk that never got retrieved looks exactly like a hallucination. A chunk buried at rank 40 looks exactly like a weak prompt. If you patch the symptom you see (the prompt) while the real leak is upstream (retrieval), you'll burn a week and the bug comes right back.

So the discipline is a tree, and you walk it in pipeline order, top to bottom — the first "no" is your bug, and you stop there:

• Was the right chunk retrieved at all? No → the generator never saw the answer; the fault is retrieval (bad chunking, an embedding mismatch, or vocabulary the query didn't match). §1–2.
• Was it ranked high enough to be read? It came back but sits below fluffier chunks the model never gets to → a ranking problem; add a reranker. §3.
• Is the chunk itself usable? Retrieved and ranked, but the chunk is junk — a fact split across a boundary, a table shredded into nonsense → a chunking problem.
• Did generation actually use it? The correct fact is provably in the prompt, yet the answer contradicts it → only now is it a generation / prompt problem. §4.

The single most common mistake — the one that separates a junior from a staff engineer here — is jumping straight to the last question because the symptom "feels like" a hallucination. Resist it. The whole point of the tree is that you earn the right to blame the prompt only after retrieval and ranking have each passed.

Almost every bad RAG answer maps to exactly one layer. Walk the tree top to bottom — each question isolates the next layer, and the first "no" is your bug. The cardinal rule from Ch 3: most "hallucinations" are retrieval failures — the model answered honestly from the wrong or missing context.

The branches: not retrieved → chunking / embedding / query mismatch (§1–2). Retrieved but ranked low → add reranking (§3). Chunk retrieved and ranked but the chunk is junk → chunking. Right chunk in context but ignored → generation / prompt (§4). When you can't tell, §5 shows how to test each layer in isolation.

§1 · Symptom: empty results essential

This is the failure that makes users trust you least. A support rep types "how do I reset my password," gets back "I don't have that information," then opens the very handbook page that answers it and screenshots it to your channel with a red circle. Nothing looks more broken than a system that denies knowing something it demonstrably ingested. And the reflex — "our docs must be missing the answer, let's add more content" — is almost always wrong. The content is there. The retriever never handed it over.

Here's the counter-intuitive part worth sitting with: an empty answer is nearly always an empty retrieval. The generator can only speak about chunks it was given; hand it nothing and an honest model correctly refuses. So a blank answer isn't the model failing — it's the model succeeding at the wrong job, because the layer before it returned zero candidates. That reframes the whole investigation: you don't look at the prompt, you look at what came out of the retriever.

Why does a retriever return nothing when the answer is right there? The classic culprit is a vocabulary mismatch combined with an over-strict matching rule. Imagine a retriever that only accepts a chunk if every word of the query appears in it — an AND-match. The user asks about "reset password"; the doc talks about "change credentials." Same meaning, zero shared words, so the AND-match returns an empty list every single time. The words are synonyms to a human and total strangers to a naive lexical filter. This is the gap between how people phrase questions and how documentation phrases answers, and it is the number-one source of phantom "we don't cover this" reports.

The trap now is the lazy fix, and naming it is half the lesson. The obvious patch is to loosen AND into OR — match any query word. But then filler words like "how," "do," and "I" match nearly every chunk, and you've swapped an empty result for a firehose of noise: technically non-empty, practically useless. The real repair has two moves: drop stopwords so junk terms can't dominate, and score by overlap instead of hard-requiring all terms, so a chunk that shares the meaningful words rises even if it doesn't share all of them. That turns a brittle yes/no filter into a ranked list — which is what retrieval should have been all along. (Even this lexical fix can't bridge reset→change; that's the job of the semantic embedding you'll reach for in §2. For now the goal is narrower and vital: prove the failure was retrieval, not the model.)

Report: "It says I don't have that information for questions the docs clearly answer." An empty answer is almost always an empty retrieval — the generator never got a chunk. Reproduce it, then inspect why the retriever returns nothing.

s1_empty.py# SYMPTOM 1 — "no results / empty". User asks "How do I reset my password?"
# The corpus DOES contain the answer, but the retriever returns nothing.
CORPUS = [
    "To change your account credentials, open Settings then Security.",
    "Billing questions are handled by the finance portal.",
    "Our office hours are 9am to 5pm on weekdays.",
]

def retrieve_broken(query, k=2):
    # BUG: require EVERY query word to appear in the chunk (AND match).
    terms = query.lower().split()
    hits = [c for c in CORPUS if all(t in c.lower() for t in terms)]
    return hits[:k]

print("BROKEN:", retrieve_broken("how do I reset my password"))

# Root cause: the answer chunk shares NO words with the query
# ("reset/password" vs "change/credentials"). AND-matching guarantees zero hits.
# The lazy patch (OR every word) makes "how","do","i" match everything -> noise.
STOP = {"how", "do", "i", "the", "a", "my", "to"}

def retrieve_fixed(query, k=2):
    # FIX: drop stopwords, SCORE by term overlap, never hard-require all terms.
    terms = [t for t in query.lower().split() if t not in STOP]
    scored = []
    for c in CORPUS:
        cl = c.lower()
        score = sum(1 for t in terms if t in cl)
        if score:
            scored.append((score, c))
    scored.sort(key=lambda x: -x[0])
    return [c for _, c in scored[:k]]

print("FIXED (lexical):", retrieve_fixed("password reset settings security"))

BROKEN: []
FIXED (lexical): ['To change your account credentials, open Settings then Security.']

§2 · Symptom: retrieves the wrong thing intermediate

The empty-result bug at least announces itself. This one is nastier because it's confident. A developer asks "how do I reset the API token," and the system returns a crisp, well-written answer — about resetting a password. Everything downstream worked: a chunk was retrieved, it scored high, generation used it faithfully. The only problem is it was the wrong chunk. Confident-but-wrong is more dangerous than empty, because empty makes users double-check and wrong makes them act.

To see why pure vector search does this, you have to understand what an embedding actually measures. Embeddings map text into a space where closeness means similarity of meaning — and "reset the API token" and "reset your password" are, in meaning-space, near neighbors. Both are about a user, a credential, and the act of resetting it. The embedding faithfully reports "these are very similar," ranks the password chunk at the top, and is right by its own metric — it just answered a slightly different question than the user asked. This is semantic adjacency: two things so close in meaning that the vector geometry can't tell which one the user actually wanted.

The specific weakness underneath is that a single embedding vector blurs rare, exact tokens. Words like api, token, or an error code like ERR-42 carry almost all the discriminating signal in the query — but a dense vector smears them into the overall gist, so a distractor that's topically close can edge out the chunk containing the exact term. Think of it as summarizing a sentence so well that you lose the one word that mattered. This is exactly where the old technology — plain keyword search — wins: a keyword matcher weighted by inverse document frequency (IDF) gives a rare word like token enormous weight precisely because it appears in almost no chunks. The word the embedding threw away is the word IDF prizes.

So the mistake to avoid isn't "using vectors" — vectors are what let paraphrases like reset→change match at all. The mistake is treating dense retrieval as complete. The production answer is neither "vector" nor "keyword" but hybrid: run both, then fuse their rankings (typically with reciprocal rank fusion, RRF) so a chunk that either side ranks highly bubbles up. Semantic search catches the paraphrases; keyword search catches the exact rare terms; fusion gets you the correct chunk in both the "reset password" paraphrase case and the "ERR-42" exact-token case with one mechanism. When someone proposes "let's just switch to keyword search," the right correction is: don't trade one blind spot for another — combine them.

Report: "It confidently answers, but about the wrong topic." The chunk came back with a high score — it's just the wrong chunk. Pure vector search can be fooled by semantic adjacency: a query about an API token lands nearest a password chunk. This is exactly the case where keyword search would have won.

s2_wrong.py# SYMPTOM 2 — "retrieves the WRONG thing". Query: "reset the API token".
# The embedding puts the query CLOSEST to a chunk about resetting a *password*
# (semantically adjacent but wrong), beating the chunk that actually documents
# API-token reset. Keyword search, which sees the exact words, fixes it.
import math

# Per-chunk/query embeddings assigned directly so the failure is deterministic
# and inspectable. Axis intuition: [account-security, api-credentials].
Q_VEC = (0.80, 0.60)                                             # "reset the api token"
CORPUS = [   # (text, embedding)
    ("To reset your password, open Settings > Security.",       (0.98, 0.20)),  # distractor
    ("Reset an API token via the developer portal token page.", (0.30, 0.95)),  # THE ANSWER
]

def cos(a, b):
    dot = sum(x*y for x, y in zip(a, b))
    na = math.sqrt(sum(x*x for x in a)); nb = math.sqrt(sum(y*y for y in b))
    return round(dot/(na*nb), 3)

# BUG: semantic ranking. 'password reset' sits at a SMALLER angle to the query
# than the true 'api token' chunk -> the WRONG chunk is retrieved at rank 1.
sem = sorted(((cos(Q_VEC, v), t) for t, v in CORPUS), reverse=True)
print("SEMANTIC top:", sem[0][0], "->", sem[0][1])

def keyword(query):
    terms = query.lower().split()
    N = len(CORPUS)
    df = {t: sum(1 for txt, _ in CORPUS if t in txt.lower()) for t in terms}
    scored = [(round(sum(math.log((N+1)/(df[t]+0.5)) for t in terms if t in txt.lower()), 3), txt)
              for txt, _ in CORPUS]
    return sorted(scored, reverse=True)

# FIX: keyword search matches the rare exact words 'api'+'token' (df=1), which
# only the correct chunk contains -> it wins. In production you FUSE the two (hybrid).
kw = keyword("reset the api token")
print("KEYWORD top:", kw[0][0], "->", kw[0][1])

SEMANTIC top: 0.904 -> To reset your password, open Settings > Security.
KEYWORD top: 2.262 -> Reset an API token via the developer portal token page.

The embedding ranks password above API token because the two are close in meaning-space, while the exact rare words api/token only live in the correct chunk. The fix isn't "keyword instead of vector" — it's hybrid: fuse both rankings (RRF, from Ch 3 Lab 3.5 / FA6) so exact terms and paraphrases both surface.

§3 · Symptom: right chunk retrieved but ranked low advanced

This is the symptom that makes engineers doubt their own eyes. You dump the top-10 retrieved chunks to a log, scroll through, and — there it is, the exact chunk that answers the question, sitting at rank 3 or rank 8. The data is clearly present. So why is the final answer still wrong? Because you read all ten; the model only read the top one or two. There is a silent gap between "retrieved" and "used," and it's called ranking.

The key idea is that RAG has two budgets, not one. Retrieval decides what could be relevant and can afford to be generous — pull back 10, 20, 30 candidates. But you can't paste thirty chunks into a prompt: it's slow, it's expensive, and it dilutes the model's attention. So a second stage picks the handful that actually go into the context — usually the top 3 to 5. If the answer-bearing chunk is in the recall set but not in that final handful, it might as well never have been retrieved. It was found and then dropped on the floor.

Here's the distinction that makes this its own layer, separate from §1–2. First-stage retrieval optimizes for recall — "don't miss the answer, cast a wide net." That's why it returns a broad set and why fast, approximate similarity is fine for it. But recall says nothing about order. A generic "security best practices" chunk can score marginally higher on first-stage similarity than the specific "Settings > Security > Reset" chunk, purely because it's broadly on-topic. First-stage scoring is a cheap approximation; it's good at "is this in the neighborhood" and bad at "which of these neighbors is the precise answer." Optimizing recall and optimizing precision are different jobs.

That's what a reranker is for, and why it belongs after retrieval rather than replacing it. A reranker (in production, a cross-encoder that reads the query and each candidate together rather than comparing two pre-computed vectors) rescores the whole candidate set by true query-chunk relevance and reorders it, so the precise answer climbs to rank 1. The pattern to internalize is the retrieval funnel: over-fetch wide and cheap, then rerank precisely down to the few chunks generation will actually read. The common mistake here is misreading the symptom as "retrieval missed it" and responding by fetching even more chunks — but more recall doesn't help if the answer already came back and got buried. More candidates without a reranker just means a longer list the model still won't read past. Skipping the rerank is the single most frequent reason a chunk that was retrieved never gets used.

Report: "When I dump the top-10 I can see the answer in there — but the final answer is still wrong." The answer-bearing chunk is retrieved, just buried below fluffier chunks, and generation only reads the top few. This is a ranking failure — the fix is a reranker over a wide recall set, not more retrieval.

s3_rank.py# SYMPTOM 3 — "the right chunk IS retrieved, but ranked too low to be used".
# First-stage retrieval returns a wide set; the answer-bearing chunk sits at rank 3.
# Generation only reads the top-1 chunk, so the answer is wrong. A reranker that
# scores query-chunk relevance promotes the right chunk to rank 1.
CANDIDATES = [   # (first_stage_score, text) -- as retrieval returned them
    (0.71, "General security best practices: rotate credentials regularly."),
    (0.68, "Passwords must be at least 12 characters with a symbol."),
    (0.66, "To reset your password: Settings > Security > Reset, then check email."),  # answer
]
QUESTION = "how do I reset my password"

def top1(cands):
    return max(cands, key=lambda c: c[0])[1]

# BUG: feed only the rank-1 chunk to the model. It's the vague one, so the model
# can't answer (or invents). The answer chunk WAS retrieved but never used.
print("USED (no rerank):", top1(CANDIDATES))

def rerank(question, cands):
    # Cheap cross-encoder stand-in: relevance = overlap of meaningful query terms.
    STOP = {"how", "do", "i", "the", "a", "my", "to"}
    q = {w for w in question.lower().split() if w not in STOP}
    scored = [(sum(t in text.lower() for t in q), text) for _, text in cands]
    scored.sort(key=lambda x: -x[0])
    return scored

# FIX: rerank the WHOLE candidate set by real query relevance before selecting.
ranked = rerank(QUESTION, CANDIDATES)
print("RERANKED top:", ranked[0][1])
print("USED (rerank):", ranked[0][1])

USED (no rerank): General security best practices: rotate credentials regularly.
RERANKED top: To reset your password: Settings > Security > Reset, then check email.
USED (rerank): To reset your password: Settings > Security > Reset, then check email.

§4 · Symptom: right chunk in context, answer ignores it professional

Now you've done everything right and it's still wrong — which is the most maddening kind of bug. You paste the literal prompt that was sent to the model into your terminal. The correct fact is right there in the context block: "Refunds are processed within 5 business days." And the model answers "30 days." The retriever found it, the reranker surfaced it, it's provably in the prompt — and the model looked past it. Only after clearing the three upstream layers do you get to say the words "this is a generation problem," and now you've earned them.

Why would a model ignore a fact it was handed? Because an LLM has two sources of knowledge, and left to itself it doesn't know which to trust. There's the context you supply (the retrieved chunk), and there's its parametric memory — everything baked into its weights during training. If the training data said refunds take 30 days and your current policy says 5, a weakly-instructed model will happily answer from its stale prior, because to the model both are just "things it knows." Retrieval put the right answer on the desk; the model read from its own memory instead. The context and the parameters conflict, and nothing told the model the context wins.

That "nothing told it" is the actual bug, and it lives in the prompt. A vague system prompt like "You are a helpful assistant" gives the model no rule for resolving the conflict, so it defaults to its priors — which feel authoritative because they're fluent. The fix is two grounding rules, and they're worth memorizing because they solve a startling fraction of "hallucination" reports: (1) answer only from the supplied context, and (2) if the context doesn't contain the answer, say you don't know. The first pins the model to the retrieved fact and overrides the stale prior; the second creates an honest exit instead of an invented one.

Rule (2) surfaces something beginners get backwards: a refusal is a feature, not a failure. When the context is empty or irrelevant, "I don't have that information" is the correct output — far better than a confident fabrication that a user will act on. So the way you verify this fix isn't only "does it now quote the right answer?" but also "does it refuse when I hand it empty context?" A grounded system does both. The mistake to avoid is the reflex to reach back into the retriever — swapping embedders, tuning k — when the retriever already did its job. If the fact is demonstrably in the context and the answer contradicts it, no amount of retrieval tuning will help; the repair is in the prompt, and the empty-context refusal test is how you prove it landed.

Report: "I logged the prompt — the correct fact is right there in the context — and the model still gave a different, outdated answer." Now you've cleared retrieval and ranking; this is a generation / prompt failure. A weak prompt lets the model answer from its parametric memory instead of the supplied context.

s4_prompt.py# SYMPTOM 4 — "the right chunk IS in context, but the answer ignores it / hallucinates".
# This is a GENERATION/PROMPT failure, not retrieval. We stub the LLM as a deterministic
# function so the bug is reproducible offline: a weak prompt lets the model fall back on
# its (stale) prior; a strict grounded prompt forces it to answer from context or refuse.

CONTEXT = "Refunds are processed within 5 business days."   # correct, retrieved chunk
PRIOR   = "Refunds are processed within 30 days."           # model's stale training memory

def llm(prompt, context):
    # Stub 'model': if the prompt does NOT pin it to the context, it answers from PRIOR.
    grounded = "answer only using the context" in prompt.lower()
    if grounded:
        if context.strip():
            return context                       # obeys: quotes the retrieved fact
        return "I don't have that information."   # honest refusal when context is empty
    return PRIOR                                   # weak prompt -> parametric hallucination

WEAK   = "You are a helpful assistant. Question: how long do refunds take?"
STRICT = ("You are a support assistant. Answer only using the context below; "
          "if it is not there, say you don't know.\nQuestion: how long do refunds take?")

# BUG: the correct chunk is right there in context, but the weak prompt lets the
# model answer from memory -> WRONG '30 days'.
print("WEAK prompt   ->", llm(WEAK, CONTEXT))

# FIX: a grounding prompt forces the answer to come from the retrieved context.
print("STRICT prompt ->", llm(STRICT, CONTEXT))

# VERIFY the escape hatch: strict prompt AND empty context -> honest refusal, no invention.
print("STRICT + empty->", llm(STRICT, ""))

WEAK prompt   -> Refunds are processed within 30 days.
STRICT prompt -> Refunds are processed within 5 business days.
STRICT + empty-> I don't have that information.

The two grounding rules from Ch 3 Lab 3.4 are the whole fix: (1) answer only from the supplied context, and (2) say you don't know when it's absent. The third print proves the escape hatch — a strict prompt with empty context refuses instead of inventing. That refusal is a feature, not a failure.

§5 · The systematic method: isolate the layer expert

The first four sections gave you clean symptoms. Production won't. Real reports arrive as "the answers got worse this week" or "customers say it's unreliable" — no tidy screenshot pointing at one layer. Under that ambiguity the temptation is to start changing things: tweak the prompt, bump k, try a new embedder, and see what helps. That's guessing, and guessing on a five-stage pipeline means you'll often "fix" a layer that was never broken and declare victory right before the bug returns.

The professional move is to borrow the oldest trick in debugging: bisection. Instead of poking the system end-to-end, you cut it into stages and test each in isolation against a case whose answer you already know. The moment one stage fails the check, you've localized the fault — everything upstream is proven good, everything downstream is irrelevant until you fix this. It's git bisect for a data pipeline: you don't read every commit, you binary-search to the one that broke.

What makes bisection possible for RAG is a golden case: a single trusted triple of (question, expected_chunk_id, expected_answer). Because you know exactly which chunk should come back and what the answer should contain, you can interrogate the pipeline in order and get a yes/no at each stage: Did retrieval surface the expected chunk into the top-k? (No → retrieval bug.) Was it at rank 1, or buried? (Buried → ranking bug.) Given only that chunk, does generation actually use it? (No → prompt bug.) The first failing check is your layer, and you stop — you don't keep testing downstream stages of a pipeline you've already proven broken upstream.

The critical detail — the one that makes this method trustworthy — is testing generation on the gold chunk alone, not on whatever retrieval happened to return. If you feed the generator the real retriever's output and the answer is wrong, you still can't tell whether the prompt is weak or the retrieval was bad. By handing generation the known-correct chunk, you remove retrieval from the equation entirely: now a wrong answer can only be the prompt's fault. That's what "isolate the layer" really means — you don't just observe the pipeline, you sever it, so each stage is judged on its own. The mistake this kills is the most expensive one in RAG work: patching the prompt when retrieval is the real problem, because no prompt can rescue a chunk that was never retrieved.

When the symptom is ambiguous — and in production it usually is — stop guessing and bisect the pipeline. Hold one known-good (question, expected_chunk, expected_answer) case and test each layer against it in order: did retrieval surface the expected chunk? was it ranked #1? given only that chunk, does generation use it? The first failing check is your layer.

s5_isolate.py# SYMPTOM 5 — "the answer is wrong, but WHICH layer failed?" The systematic method:
# test retrieval and generation SEPARATELY against a known expected chunk, so you
# bisect the pipeline instead of guessing. One gold (question, expected_id, expected).

GOLD = {"q": "how do I reset my password",
        "expected_id": "faq#3",
        "must_contain": "Settings > Security"}

# --- stubs you can swap for the real retriever / generator ---
def retrieve(q):
    return [("faq#3", "To reset your password: Settings > Security > Reset."),
            ("faq#1", "Passwords must be 12+ characters.")]

def generate(q, context):
    # grounded stub: answers from context if a chunk is present, else refuses
    return context[0][1] if context else "I don't have that information."

def diagnose(gold, retrieve, generate, k=2):
    hits = retrieve(gold["q"])
    ids = [i for i, _ in hits][:k]
    # LAYER 1 — retrieval: did the expected chunk make the top-k at all?
    if gold["expected_id"] not in ids:
        return f"RETRIEVAL failed: {gold['expected_id']} not in top-{k} {ids} -> fix chunking/embedding/hybrid"
    # LAYER 2 — ranking: is it at rank 1, or buried?
    rank = ids.index(gold["expected_id"]) + 1
    # LAYER 3 — generation: feed ONLY the gold chunk; does the answer use it?
    gold_chunk = [h for h in hits if h[0] == gold["expected_id"]]
    ans = generate(gold["q"], gold_chunk)
    if gold["must_contain"] not in ans:
        return f"GENERATION failed: right chunk in context but answer missing '{gold['must_contain']}' -> fix prompt"
    if rank != 1:
        return f"RANKING weak: correct chunk retrieved at rank {rank} (not 1) -> add a reranker"
    return "ALL LAYERS PASS on this case"

print(diagnose(GOLD, retrieve, generate))

# Inject a RANKING regression: same chunk, but retrieval now returns it at rank 2.
def retrieve_buried(q):
    return [("faq#1", "Passwords must be 12+ characters."),
            ("faq#3", "To reset your password: Settings > Security > Reset.")]
print(diagnose(GOLD, retrieve_buried, generate))

ALL LAYERS PASS on this case
RANKING weak: correct chunk retrieved at rank 2 (not 1) -> add a reranker

§6 · Tech-lead: a standing RAG debugging discipline tech-lead

Here's the failure mode that outlives any single bug you fix. You heroically root-cause a retrieval regression on Tuesday, ship the fix, and move on. Three weeks later a different engineer changes the chunk size for an unrelated reason, and your bug silently comes back — because the only place your hard-won diagnosis lived was in your head. A fix that isn't encoded in a test will regress; it's not a question of if, only when. The difference between an engineer who debugs RAG and a lead who owns it is that the lead makes the fix permanent and other-people-proof.

The way you do that is to stop treating the §5 harness as a one-off diagnostic and turn it into standing infrastructure. Keep a versioned golden set — 15 to 20 (question, expected_chunk_id, expected_answer) cases — in the repo, and make every bug you ever fix become a new row in it. Run the layer-bisecting harness across the whole set on every change and report per-layer pass rates, so a regression doesn't just say "worse," it says which layer got worse. This is the same reason the eval chapters insist on separate metrics — Recall@k for retrieval, context precision for ranking, groundedness for generation — because a single end-to-end "is the answer good?" score can't tell you where to look.

The payoff is a translation that changes how your team operates. An unactionable report — "the RAG feels worse this week" — becomes an actionable one: "retrieval Recall@5 fell from 0.9 to 0.7 right after the chunking change." The first sentence starts a week of guessing; the second points straight at the fix. Wire the golden set into CI as an eval gate: if retrieval recall or generation groundedness drops below the last release, the merge is blocked, and the regression is caught by a machine before a user ever sees it. That's the whole arc of this challenge, hardened into a process — diagnose in pipeline order, reproduce, root-cause, verify, and then gate so it can't quietly return.

A lead doesn't debug RAG by intuition — they institutionalize the bisection. Keep a golden set of 15–20 (question, expected_chunk_id, expected_answer) cases in version control. On every change, run the layer harness from §5 across the set and report per-layer pass rates, so a regression names its own layer. Wire it into CI as an eval gate (FA5): retrieval Recall@k and generation groundedness must not drop. This turns "the RAG feels worse this week" — an unactionable report — into "retrieval Recall@5 fell from 0.9 to 0.7 after the chunking change," which points straight at the fix.