RAG evaluation

1 · Why RAG needs its own evaluation essential

Here's the on-call story that teaches this lesson. A user reports that your support bot gave a confidently wrong answer about the refund policy. You open the trace. The answer is fluent, cites nothing, and is simply false. So — is the model broken? You spend an afternoon rewriting the system prompt, ship it, and the same class of bug reappears a week later. The prompt was never the problem: the retriever had handed the model a chunk about shipping, not refunds, and the model dutifully improvised on what it was given. You optimized the wrong half of the system, because your one accuracy number couldn't tell you which half was bleeding.

That is the whole reason RAG earns its own evaluation. Picture a RAG pipeline as a restaurant kitchen: a runner fetches ingredients from the pantry, and a chef cooks with whatever lands on the counter. When a dish comes back, "the food was bad" is useless feedback — you need to know whether the runner grabbed the wrong ingredients or the chef ruined good ones, because one means retraining the runner and the other means retraining the chef. A single "customer disliked it" score fuses two unrelated failures that have two unrelated fixes.

Stated precisely: a RAG system has two components that fail independently. The retriever turns the question into a search and returns some chunks; the generator (the LLM) writes an answer conditioned on those chunks. Retrieval can be perfect while generation hallucinates, and generation can be flawless while retrieval starves it of the one chunk it needed. Because the two failures are independent, a single scalar accuracy score is underdetermined — the same low number can be produced by two opposite root causes, so it cannot point you at the lever that fixes it. RAG evaluation's core move is to decompose the score along the pipeline's seam: metrics that grade retrieval, kept separate from metrics that grade generation.

The common mistake is treating a RAG app as one black box and eyeballing final answers. It feels efficient and quietly burns days: you tune prompts to fix retrieval bugs, widen top-k to fix generation bugs, and never converge because you never localized the fault. The discipline in this lesson — four metrics, split cleanly across the seam — is what turns "the answer was wrong" into "context recall is 0.4, go fix chunking." Keep the runner-and-chef split in mind: every metric that follows is grading one or the other.

A RAG system has two moving parts that can fail independently: the retriever (did it find the right chunks?) and the generator (did the model answer correctly from those chunks?). A plain accuracy score can't tell them apart — a wrong answer might mean the retriever missed the doc, or the model ignored a doc it was handed. RAG evaluation splits the pipeline so you know which half to fix. This builds on the retrieval pipeline in Ch 3 · RAG and the general eval discipline in Ch 5 · Evaluation.

2 · The four core metrics essential

You've agreed the score must split across the pipeline's seam — now the question is how many pieces. The tempting answer is two: one number for the retriever, one for the generator. But each half can fail in two different ways, and lumping them together hides which one bit you. So the field settled on four metrics — two per component — and this is the section where they stop being jargon and become tools you reach for by name.

Start with the retriever, and borrow the two words every search system has argued about for decades: precision and recall. They pull in opposite directions, which is exactly why you need both. Context precision asks "of the chunks I fetched, how many were actually relevant?" — it punishes a retriever that pads the context with junk. Context recall asks the mirror question, "of the chunks I needed, how many did I fetch?" — it punishes a retriever that missed the answer-bearing doc entirely. Here's why one alone lies to you: a retriever that returns your entire corpus scores perfect recall (it found everything) and catastrophic precision (almost all of it is noise); a retriever that returns a single lucky chunk can score perfect precision and miserable recall. You only trust a retriever when both are high.

Now the generator, graded against the context it was handed. Faithfulness asks "is every claim in the answer supported by the retrieved context?" — it is your hallucination detector, and it catches the most dangerous failure of all: a fluent, confident answer that the documents never actually said. Answer relevance asks a different thing, "does the answer actually address the question that was asked?" — it catches evasive, padded, or off-topic replies that may be perfectly faithful yet useless. The subtlety worth internalizing: faithfulness and relevance are orthogonal. An answer can be faithful but irrelevant (it accurately recites the context but dodges the question) or relevant but unfaithful (it answers the question by making things up). You need both because they fail on different axes.

The mistake to avoid is collapsing these back into one "quality" average. Averaging destroys the diagnostic power you built them for — a mean of 0.75 could be four mediocre metrics or two perfect ones dragging up two broken ones, and those demand opposite responses. Read the four as a 2×2 grid (retriever vs generator, and the two ways each can fail) and keep them separate all the way to the scorecard.

Ragas-style evaluation scores each question on four axes. Two grade the generator and two grade the retriever:

Read them as a grid: precision/recall judge the retriever against ground-truth relevant chunks; faithfulness/relevance judge the answer against the context and the question. A healthy system needs all four high — a great answer built on lucky guessing (low recall) is fragile, and perfect retrieval wasted by a hallucinating model (low faithfulness) is unsafe.

3 · The RAG eval pipeline essential

You know the four metrics — but a metric floating in isolation doesn't stop a bad deploy. The failure this section prevents is the one every team hits after their first eval script: someone computes faithfulness in a notebook, nods, and merges a change that quietly tanks recall, because the eval lived in a notebook instead of the pipeline. Metrics only protect you when they're wired into a repeatable machine that ends in a yes/no decision. That machine is the RAG eval pipeline, and its shape is the same every time.

Think of it as an assembly line with four stations. At the first station a single eval sample arrives — and here's the detail beginners under-pack: a sample is not just "the question." It's the question, the answer your app generated, the contexts your retriever returned, and the ground-truth riding alongside (the reference answer plus the ids of the chunks that should have been fetched). You need all of it because the metrics compare what happened against what should have happened. At the second station, the sample hits the scorers — the four graders from §2 run in parallel, each emitting a number. At the third, you aggregate: run many samples and take the mean of each metric across the whole set, because one sample is noise and only the average is signal. At the fourth, the gate compares each mean to a threshold and emits a single pass/fail — in practice, a process exit code.

Why draw it as a line at all? Because the decomposition you fought for in §1 has to survive all the way to the end. If you collapsed the four scores into one before the gate, you'd be right back to the underdetermined number that can't tell you which half broke. Keeping the metrics separate through aggregation means a red gate arrives with a label attached — "context precision failed" — pointing straight at the retriever. The whole point of the pipeline is that the diagnostic signal you designed doesn't leak away between "we scored it" and "we blocked the merge." The diagram below is that assembly line; the sections after it build each scorer so you can see exactly what number each station emits.

Every eval sample is a triple — question + generated answer + retrieved contexts — plus ground-truth (the reference answer and the ids of the chunks that should have been retrieved). Each metric is a scorer; the per-question scores are aggregated, then compared to thresholds to produce a single pass/fail gate:

The scorers are the interesting part. The next sections model each one offline with plain Python so you can see exactly what the number means before trusting a framework to compute it.

4 · Intermediate — retrieval metrics as set overlap intermediate

People hear "evaluate retrieval" and reach for a model — and freeze, because calling an LLM to grade every chunk sounds slow, expensive, and hard to trust. The liberating surprise of this section is that the two retriever metrics need no model at all. Once you've labeled which chunk ids are truly relevant, precision and recall are pure counting. This is the cheapest, most deterministic corner of the whole eval stack, and it's where you should always start debugging.

The mental picture is two overlapping circles — a Venn diagram. One circle is the set of chunks the retriever returned; the other is the set of chunks that are actually relevant (your ground-truth labels). The overlap in the middle is the chunks you got right. Everything else is a mistake of one of two kinds. Precision and recall simply divide that overlap by a different denominator, and choosing the denominator is the entire idea:

• Context precision = overlap / retrieved. Of everything you fetched, what fraction was relevant? The denominator is what the retriever did, so precision drops the moment you pad the context with junk chunks. It's the "am I being noisy?" number.
• Context recall = overlap / relevant. Of everything you needed, what fraction did you fetch? The denominator is what you should have gotten, so recall drops the moment an answer-bearing chunk goes missing. It's the "am I missing things?" number.

Read that carefully, because the two denominators are the whole distinction and it's the thing beginners blur. Same numerator (the overlap), different denominator, opposite meaning. That's also why they trade off against top-k: fetch more chunks and recall can only rise or hold (you can't lose a hit by looking further), while precision usually falls (each extra chunk is more likely to be noise than gold). Tuning a retriever is largely the act of steering this tension.

The mistake to watch for is reporting one and ignoring the other — usually recall, because "did we find it?" feels like the whole job. But a retriever that returns the entire corpus has perfect recall and worthless precision, and it will bury your generator in noise and blow your context budget. Both numbers, always, before you trust the retriever. The lab below computes both from nothing but Python sets, so you can watch the arithmetic move.

Context precision and recall are, at heart, set arithmetic over chunk ids. If you know which chunk ids are truly relevant (ground-truth) and which the retriever returned, precision and recall fall straight out — no model needed. Precision = fraction of retrieved that are relevant; recall = fraction of relevant that were retrieved.

retrieval_metrics.pydef context_precision(retrieved, relevant):
    if not retrieved:
        return 0.0
    hits = [c for c in retrieved if c in relevant]
    return round(len(hits) / len(retrieved), 2)

def context_recall(retrieved, relevant):
    if not relevant:
        return 1.0
    got = [c for c in relevant if c in retrieved]
    return round(len(got) / len(relevant), 2)

relevant  = {"doc7", "doc12"}
retrieved = ["doc7", "doc3", "doc9"]

print("precision:", context_precision(retrieved, relevant))
print("recall:   ", context_recall(retrieved, relevant))

precision: 0.33
recall:    0.5

Precision 0.33 means only 1 of 3 retrieved chunks was relevant (noisy retrieval); recall 0.5 means we found 1 of the 2 chunks we needed (a miss). Both are low here — a retriever this bad would starve even a perfect generator. This is the same precision/recall you'd tune top-k and chunking against in Ch 3.

5 · Advanced — faithfulness as an LLM-as-judge proxy advanced

Retrieval was easy to score because "relevant" is a label you can pin to a chunk id. Faithfulness is where it gets hard, and the difficulty is worth naming out loud: there is no set of ids that tells you whether an answer is grounded. Grounding is a claim about meaning — does the sentence the model wrote actually follow from the text it was shown? — and meaning doesn't reduce to counting. This is the metric that catches the scariest failure in all of RAG: the confident, fluent, well-formatted answer that the documents never supported. So how do you grade something only a reader can judge?

The production answer is LLM-as-judge: you hand a second model the answer and the retrieved context and ask it to rule, claim by claim, whether each statement is entailed by the context. It works because the same language understanding that lets a model generate fluent text lets another model check whether a sentence is supported — the judge is doing reading comprehension, the one task LLMs are genuinely good at. Faithfulness then becomes the fraction of the answer's claims the judge marks as supported: decompose the answer into atomic claims, check each against the context, and divide.

To make the mechanism visible without a model call, the lab below builds a deliberately crude stand-in: split the answer into sentences, and count a claim as "supported" if most of its content words appear somewhere in the context. This is a teaching prop, not a real judge — and understanding why it's a prop is half the lesson. Token overlap is a bag-of-words proxy for meaning, and meaning laughs at bags of words. It cannot see paraphrase ("issued within a month" ↔ "30-day window" share almost no tokens yet say the same thing, so a true claim scores as unsupported). Far worse, it is blind to negation: "refunds are allowed" and "refunds are not allowed" share nearly every token, so the proxy happily "supports" the exact opposite of what the context says — a false positive on the one failure faithfulness exists to catch.

The mistake this guards against is shipping the cheap proxy to production because it "mostly agrees" on your test cases. It agrees until an adversarial or subtly-negated answer walks through, and then it waves a hallucination straight past the gate. Use the token-overlap version to understand the shape of the metric — claims in, supported-fraction out — and use a reasoning LLM-as-judge when you actually need to trust the number. The lab shows the proxy so the idea is concrete; the callout after it draws the line you must not cross.

Faithfulness asks: is every claim in the answer supported by the retrieved context? In production this is scored by an LLM-as-judge — a second model reads the answer and the context and rules on each claim (Ch 5 covers judge design). We can model the shape of that judge offline with a token-overlap proxy: split the answer into claims, and count a claim as supported if most of its content words appear in the context.

faithfulness.pyimport re

def tokens(text):
    return set(re.findall(r"[a-z0-9]+", text.lower()))

def faithfulness(answer, contexts):
    """Proxy: split the answer into claims (sentences); a claim is 'supported'
    if most of its content tokens appear somewhere in the retrieved context."""
    context_tokens = set()
    for c in contexts:
        context_tokens |= tokens(c)
    claims = [s.strip() for s in re.split(r"[.!?]", answer) if s.strip()]
    supported = 0
    for claim in claims:
        ct = tokens(claim)
        if not ct:
            continue
        overlap = len(ct & context_tokens) / len(ct)
        if overlap >= 0.6:
            supported += 1
    return round(supported / len(claims), 2) if claims else 0.0

contexts = ["The refund window is 30 days from delivery.",
            "Refunds are issued to the original payment method."]

grounded     = "Refunds go to the original payment method within 30 days."
hallucinated = "Refunds are issued as store credit only and expire after a year."

print("grounded answer:    ", faithfulness(grounded, contexts))
print("hallucinated answer:", faithfulness(hallucinated, contexts))

grounded answer:     1.0
hallucinated answer: 0.0

Below is what the real thing looks like with the Ragas framework — it wires up the metrics and runs the judge calls for you. It does not run here:

ragas_eval.py# needs the ragas library:  pip install ragas
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (faithfulness, answer_relevancy,
                           context_precision, context_recall)

data = Dataset.from_dict({
    "question":     ["When are refunds issued?"],
    "answer":       ["Refunds go to the original payment method within 30 days."],
    "contexts":     [["The refund window is 30 days from delivery.",
                      "Refunds are issued to the original payment method."]],
    "ground_truth": ["Refunds go to the original payment method within 30 days."],
})

result = evaluate(data, metrics=[faithfulness, answer_relevancy,
                                 context_precision, context_recall])
print(result)   # {'faithfulness': 1.0, 'answer_relevancy': 0.98, ...}

6 · Professional — building an eval set & the scorecard professional

Every metric so far quietly assumed a thing you don't have yet: ground-truth. Precision and recall compared against "the relevant chunk ids"; faithfulness compared against "the retrieved context"; but who decided which chunks are relevant, and what a correct answer looks like? Nobody, until you build it. The eval set is that missing foundation, and the failure it prevents is the most demoralizing one in this whole lesson: a beautiful four-metric dashboard computed over garbage labels, producing numbers that look rigorous and mean nothing. Metrics are a lens; the eval set is what you point it at, and a dirty lens on the wrong subject just gives you confident nonsense.

Think of the eval set as the exam you write for your RAG system — and like any fair exam, three properties matter. First, each question needs an answer key: a realistic question, the reference answer a human expert would give, and the ids of the chunks that actually contain that answer (that last part is what makes context recall computable at all). Second, it must cover the distribution you actually serve — easy single-fact lookups, multi-hop questions that need two chunks stitched together, and the known-hard edge cases that embarrass you in production. An exam of only easy questions certifies nothing. Third, it must be frozen in version control, so a score today is comparable to a score last month; an eval set that drifts silently makes every "did this change help?" question unanswerable.

Once each sample can be scored, you aggregate — mean per metric across the set — and compare each mean to a per-metric threshold. That's the scorecard: the compact table a human reads and CI enforces. And notice the thresholds are deliberately not equal (faithfulness held highest, because a hallucination is the worst outcome you can ship); the bar encodes what your product can and cannot tolerate. Crucially, the overall verdict passes only if every metric clears its bar — one weak axis fails the whole run, which is exactly the point: you decomposed the score so a single broken half can't hide behind three healthy ones.

The mistake professionals still make is treating the eval set as write-once. It isn't a monument, it's a living regression suite — every real production failure that slips through becomes a new labeled sample, so the same bug can never ship twice. Build it small and honest first, then grow it from incidents. The steps and lab below turn this into a concrete recipe and a runnable scorecard.

An eval set is the foundation — metrics are meaningless without ground-truth. Each sample needs a realistic question, a reference answer, and the relevant chunk ids. Cover the distribution you actually serve (easy lookups, multi-hop, and known-hard edge cases), and keep it version-controlled so runs are comparable over time.

How to build a RAG eval set

1. Sample real user questions (or write representative ones per topic).
2. Write the ground-truth answer a human expert would give.
3. Label which chunk id(s) in your corpus contain that answer (for context recall).
4. Freeze it in version control; grow it whenever a bug slips through to production.
5. Re-run the whole set on every retriever/prompt/model change.

Once you can score each sample, you aggregate across the set (mean per metric) and compare to per-metric thresholds. That's the scorecard a human reads and CI enforces:

scorecard.pyTHRESHOLDS = {"faithfulness": 0.90, "answer_relevance": 0.80,
              "context_precision": 0.70, "context_recall": 0.80}

def scorecard(per_question):
    """per_question: list of dicts, one per eval question, with each metric 0..1.
    Aggregate = mean per metric; a metric passes if its mean >= its threshold."""
    metrics = THRESHOLDS.keys()
    n = len(per_question)
    agg = {m: round(sum(q[m] for q in per_question) / n, 3) for m in metrics}
    verdict = {m: ("PASS" if agg[m] >= THRESHOLDS[m] else "FAIL") for m in metrics}
    overall = "PASS" if all(v == "PASS" for v in verdict.values()) else "FAIL"
    return agg, verdict, overall

eval_run = [
    {"faithfulness": 1.0, "answer_relevance": 0.9, "context_precision": 0.66, "context_recall": 1.0},
    {"faithfulness": 0.95, "answer_relevance": 0.85, "context_precision": 0.75, "context_recall": 0.8},
    {"faithfulness": 0.9, "answer_relevance": 0.8, "context_precision": 0.8, "context_recall": 0.7},
]

agg, verdict, overall = scorecard(eval_run)
for m in THRESHOLDS:
    print(f"{m:18} mean={agg[m]:.3f}  thr={THRESHOLDS[m]:.2f}  {verdict[m]}")
print("OVERALL:", overall)

faithfulness       mean=0.950  thr=0.90  PASS
answer_relevance   mean=0.850  thr=0.80  PASS
context_precision  mean=0.737  thr=0.70  PASS
context_recall     mean=0.833  thr=0.80  PASS
OVERALL: PASS

7 · Tech-lead — eval-gating in CI & avoiding overfitting tech-lead

A scorecard nobody is forced to look at is a scorecard that rots. The failure mode here is organizational, not technical: the team has beautiful metrics, everyone agrees they matter, and then a Friday-afternoon prompt tweak that shaves 8 points off context recall merges anyway — because reading the dashboard was optional and the deadline was not. The fix is to make the eval un-ignorable by turning the scorecard into a gate: CI runs the eval set on every change and blocks the merge if quality slips. This is nothing more exotic than a failing unit test, aimed at RAG quality instead of a function's return value. The gate is literally an exit code — 0 merges, non-zero blocks.

But a lead makes one non-obvious design choice that separates a gate that helps from one that everyone learns to disable: gate on regression, not an absolute floor. The naive version says "faithfulness must exceed 0.90, forever." That bar is wrong on both ends — too high for a young system that hasn't earned it (so the gate blocks all progress and gets bypassed), and too low for a mature one (so a real regression from 0.97 to 0.91 sails through). Regression gating instead compares each metric to the last known-good baseline and fails only if it drops more than a small tolerance — enough slack to absorb the normal noise of an LLM judge, tight enough to catch a genuine slide. It asks "did this change make things worse?" which is the only question a merge check can fairly answer, and it stays fair as the system honestly improves.

The subtler trap a lead has to defend against is overfitting to the eval set, and it's insidious precisely because it looks like success. The moment you start tuning the system against a fixed set of questions, the scores stop measuring quality and start measuring how well you've memorized the test — Goodhart's law wearing a lab coat. Every metric creeps toward 1.0, the dashboard glows green, and production quality is flat or worse, because you optimized the exam rather than the skill. The defenses are cultural: keep a held-out slice you never tune on, rotate fresh production failures into the set, and treat a suite where everything is pinned at 1.0 as a smell (your eval is too easy, not your system too good) rather than a trophy.

So the tech-lead's real deliverable isn't a script — it's the discipline around it: version-controlled eval set, regression gate in CI, growth from real incidents, and a firewall between the tuned-on and held-out questions. That discipline, not any particular framework, is what lets a team swap embeddings, rewrite chunking, or change models without silently shipping a regression. The lab below is the gate itself, in a dozen lines of exit-code logic.

A lead turns the scorecard into a gate: CI runs the eval set on every change and blocks the merge if any metric regresses below a tolerance of the last known-good baseline. This is the RAG equivalent of a failing unit test — a prompt tweak that quietly tanks context recall never reaches production. The gate is just an exit code the CI runner checks.

ci_gate.pyimport sys

BASELINE = {"faithfulness": 0.92, "answer_relevance": 0.82,
            "context_precision": 0.72, "context_recall": 0.81}
# regression tolerance: a metric may not drop more than this below baseline
TOLERANCE = 0.03

def ci_gate(current, baseline=BASELINE, tolerance=TOLERANCE):
    regressions = []
    for m, base in baseline.items():
        if current[m] < base - tolerance:
            regressions.append(f"{m}: {current[m]:.2f} < {base - tolerance:.2f}")
    if regressions:
        print("RAG EVAL GATE: FAIL")
        for r in regressions:
            print("  regressed ->", r)
        return 1
    print("RAG EVAL GATE: PASS")
    return 0

current = {"faithfulness": 0.93, "answer_relevance": 0.83,
           "context_precision": 0.60, "context_recall": 0.82}

exit_code = ci_gate(current)
print("exit code:", exit_code)
sys.exit(exit_code)

RAG EVAL GATE: FAIL
  regressed -> context_precision: 0.60 < 0.69
exit code: 1

Here faithfulness and relevance held, but context precision cratered — the gate fails the build and names the retriever as the culprit. That's the payoff of four metrics: the failure message points at the half to fix. (Run echo $? after this file and you'll see 1 — exactly what a CI runner reads to fail a job.)

def context_recall(relevant_ids, retrieved_ids):
    relevant = set(relevant_ids)
    if not relevant:
        return 1.0
    hit = relevant & set(retrieved_ids)
    return len(hit) / len(relevant)

print(context_recall(["c1","c3"], ["c1","c2","c3","c4"]))  # 1.0  (both found)
print(context_recall(["c1","c3"], ["c2","c1","c5"]))       # 0.5  (missed c3)

def context_precision(relevant_ids, retrieved_ids):
    if not retrieved_ids:
        return 0.0
    hit = set(relevant_ids) & set(retrieved_ids)
    return len(hit) / len(retrieved_ids)

rel = ["c1","c3"]
print(context_precision(rel, ["c1","c3"]))            # 1.0  (tight)
print(context_precision(rel, ["c1","c3","c8","c9"]))  # 0.5  (padded with junk)

def faithfulness(answer, contexts):
    claims = [c.strip() for c in answer.split(".") if c.strip()]
    ctx = " ".join(contexts).lower()
    grounded = 0
    for claim in claims:
        # crude grounding proxy: majority of content words appear in context
        words = [w for w in claim.lower().split() if len(w) > 3]
        hits = sum(1 for w in words if w in ctx)
        if words and hits / len(words) >= 0.6:
            grounded += 1
    return grounded / len(claims) if claims else 1.0

ctx = ["Refunds are issued within 5 business days to the original card."]
ans = "Refunds are issued within 5 business days. You get store credit instead."
print(round(faithfulness(ans, ctx), 2))   # 0.5 — 2nd claim unsupported (hallucination)

def recall(rel, ret): return len(set(rel)&set(ret))/len(set(rel)) if rel else 1.0
def precision(rel, ret): return len(set(rel)&set(ret))/len(ret) if ret else 0.0

samples = [
    dict(rel=["c1"], ret=["c1","c2"], faith=1.0, relevance=0.9),
    dict(rel=["c3"], ret=["c9"],      faith=0.4, relevance=0.8),
]
agg = {"context_recall":0,"context_precision":0,"faithfulness":0,"answer_relevance":0}
for s in samples:
    agg["context_recall"]    += recall(s["rel"], s["ret"])
    agg["context_precision"] += precision(s["rel"], s["ret"])
    agg["faithfulness"]      += s["faith"]
    agg["answer_relevance"]  += s["relevance"]
agg = {k: round(v/len(samples),2) for k,v in agg.items()}
THRESH = {"context_recall":0.8,"context_precision":0.6,"faithfulness":0.8,"answer_relevance":0.7}
failed = [k for k,v in agg.items() if v < THRESH[k]]
print(agg)
print("GATE:", "PASS" if not failed else "FAIL -> " + ", ".join(failed))

def validate_evalset(rows):
    good, bad = [], []
    for r in rows:
        problems = []
        if not r.get("question"): problems.append("no question")
        if not r.get("reference_answer"): problems.append("no reference answer")
        if not r.get("relevant_chunk_ids"): problems.append("no relevant ids")
        (bad if problems else good).append((r.get("question","?"), problems))
        if not problems: good[-1] = r
    ok = [r for r in good if isinstance(r, dict)]
    return ok, [b for b in bad]

rows = [
    {"question":"Refund window?","reference_answer":"5 days","relevant_chunk_ids":["c1"]},
    {"question":"","reference_answer":"x","relevant_chunk_ids":["c2"]},   # bad
    {"question":"2FA?","reference_answer":"toggle","relevant_chunk_ids":[]}, # bad
]
ok, bad = validate_evalset(rows)
print(f"usable: {len(ok)}   rejected: {len(rows)-len(ok)}")

THRESH = {"faithfulness":0.85,"answer_relevance":0.75,
          "context_recall":0.80,"context_precision":0.60}

def ci_gate(scores, prev_scores=None):
    failed = [k for k,v in scores.items() if v < THRESH[k]]
    decision = "PASS" if not failed else "FAIL -> fix: " + ", ".join(failed)
    warn = ""
    # overfitting guard: suspiciously high on a static set that never rotates
    if prev_scores and all(scores[k] >= 0.97 for k in scores):
        warn = " WARN: near-perfect on a static eval set — rotate held-out questions"
    return decision + warn

now  = {"faithfulness":0.88,"answer_relevance":0.79,"context_recall":0.82,"context_precision":0.65}
print(ci_gate(now))   # PASS
perfect = {k:0.99 for k in THRESH}
print(ci_gate(perfect, prev_scores=perfect))  # PASS WARN: ... rotate

Knowledge check check yourself