RAG evaluation

1 · Why RAG needs its own evaluation essential

A RAG system has two moving parts that can fail independently: the retriever (did it find the right chunks?) and the generator (did the model answer correctly from those chunks?). A plain accuracy score can't tell them apart — a wrong answer might mean the retriever missed the doc, or the model ignored a doc it was handed. RAG evaluation splits the pipeline so you know which half to fix. This builds on the retrieval pipeline in Ch 3 · RAG and the general eval discipline in Ch 5 · Evaluation.

2 · The four core metrics essential

Ragas-style evaluation scores each question on four axes. Two grade the generator and two grade the retriever:

Read them as a grid: precision/recall judge the retriever against ground-truth relevant chunks; faithfulness/relevance judge the answer against the context and the question. A healthy system needs all four high — a great answer built on lucky guessing (low recall) is fragile, and perfect retrieval wasted by a hallucinating model (low faithfulness) is unsafe.

3 · The RAG eval pipeline essential

Every eval sample is a triple — question + generated answer + retrieved contexts — plus ground-truth (the reference answer and the ids of the chunks that should have been retrieved). Each metric is a scorer; the per-question scores are aggregated, then compared to thresholds to produce a single pass/fail gate:

The scorers are the interesting part. The next sections model each one offline with plain Python so you can see exactly what the number means before trusting a framework to compute it.

4 · Intermediate — retrieval metrics as set overlap intermediate

Context precision and recall are, at heart, set arithmetic over chunk ids. If you know which chunk ids are truly relevant (ground-truth) and which the retriever returned, precision and recall fall straight out — no model needed. Precision = fraction of retrieved that are relevant; recall = fraction of relevant that were retrieved.

retrieval_metrics.pydef context_precision(retrieved, relevant):
    if not retrieved:
        return 0.0
    hits = [c for c in retrieved if c in relevant]
    return round(len(hits) / len(retrieved), 2)

def context_recall(retrieved, relevant):
    if not relevant:
        return 1.0
    got = [c for c in relevant if c in retrieved]
    return round(len(got) / len(relevant), 2)

relevant  = {"doc7", "doc12"}
retrieved = ["doc7", "doc3", "doc9"]

print("precision:", context_precision(retrieved, relevant))
print("recall:   ", context_recall(retrieved, relevant))

precision: 0.33
recall:    0.5

Precision 0.33 means only 1 of 3 retrieved chunks was relevant (noisy retrieval); recall 0.5 means we found 1 of the 2 chunks we needed (a miss). Both are low here — a retriever this bad would starve even a perfect generator. This is the same precision/recall you'd tune top-k and chunking against in Ch 3.

5 · Advanced — faithfulness as an LLM-as-judge proxy advanced

Faithfulness asks: is every claim in the answer supported by the retrieved context? In production this is scored by an LLM-as-judge — a second model reads the answer and the context and rules on each claim (Ch 5 covers judge design). We can model the shape of that judge offline with a token-overlap proxy: split the answer into claims, and count a claim as supported if most of its content words appear in the context.

faithfulness.pyimport re

def tokens(text):
    return set(re.findall(r"[a-z0-9]+", text.lower()))

def faithfulness(answer, contexts):
    """Proxy: split the answer into claims (sentences); a claim is 'supported'
    if most of its content tokens appear somewhere in the retrieved context."""
    context_tokens = set()
    for c in contexts:
        context_tokens |= tokens(c)
    claims = [s.strip() for s in re.split(r"[.!?]", answer) if s.strip()]
    supported = 0
    for claim in claims:
        ct = tokens(claim)
        if not ct:
            continue
        overlap = len(ct & context_tokens) / len(ct)
        if overlap >= 0.6:
            supported += 1
    return round(supported / len(claims), 2) if claims else 0.0

contexts = ["The refund window is 30 days from delivery.",
            "Refunds are issued to the original payment method."]

grounded     = "Refunds go to the original payment method within 30 days."
hallucinated = "Refunds are issued as store credit only and expire after a year."

print("grounded answer:    ", faithfulness(grounded, contexts))
print("hallucinated answer:", faithfulness(hallucinated, contexts))

grounded answer:     1.0
hallucinated answer: 0.0

Below is what the real thing looks like with the Ragas framework — it wires up the metrics and runs the judge calls for you. It does not run here:

ragas_eval.py# needs the ragas library:  pip install ragas
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (faithfulness, answer_relevancy,
                           context_precision, context_recall)

data = Dataset.from_dict({
    "question":     ["When are refunds issued?"],
    "answer":       ["Refunds go to the original payment method within 30 days."],
    "contexts":     [["The refund window is 30 days from delivery.",
                      "Refunds are issued to the original payment method."]],
    "ground_truth": ["Refunds go to the original payment method within 30 days."],
})

result = evaluate(data, metrics=[faithfulness, answer_relevancy,
                                 context_precision, context_recall])
print(result)   # {'faithfulness': 1.0, 'answer_relevancy': 0.98, ...}

6 · Professional — building an eval set & the scorecard professional

An eval set is the foundation — metrics are meaningless without ground-truth. Each sample needs a realistic question, a reference answer, and the relevant chunk ids. Cover the distribution you actually serve (easy lookups, multi-hop, and known-hard edge cases), and keep it version-controlled so runs are comparable over time.

How to build a RAG eval set

1. Sample real user questions (or write representative ones per topic).
2. Write the ground-truth answer a human expert would give.
3. Label which chunk id(s) in your corpus contain that answer (for context recall).
4. Freeze it in version control; grow it whenever a bug slips through to production.
5. Re-run the whole set on every retriever/prompt/model change.

Once you can score each sample, you aggregate across the set (mean per metric) and compare to per-metric thresholds. That's the scorecard a human reads and CI enforces:

scorecard.pyTHRESHOLDS = {"faithfulness": 0.90, "answer_relevance": 0.80,
              "context_precision": 0.70, "context_recall": 0.80}

def scorecard(per_question):
    """per_question: list of dicts, one per eval question, with each metric 0..1.
    Aggregate = mean per metric; a metric passes if its mean >= its threshold."""
    metrics = THRESHOLDS.keys()
    n = len(per_question)
    agg = {m: round(sum(q[m] for q in per_question) / n, 3) for m in metrics}
    verdict = {m: ("PASS" if agg[m] >= THRESHOLDS[m] else "FAIL") for m in metrics}
    overall = "PASS" if all(v == "PASS" for v in verdict.values()) else "FAIL"
    return agg, verdict, overall

eval_run = [
    {"faithfulness": 1.0, "answer_relevance": 0.9, "context_precision": 0.66, "context_recall": 1.0},
    {"faithfulness": 0.95, "answer_relevance": 0.85, "context_precision": 0.75, "context_recall": 0.8},
    {"faithfulness": 0.9, "answer_relevance": 0.8, "context_precision": 0.8, "context_recall": 0.7},
]

agg, verdict, overall = scorecard(eval_run)
for m in THRESHOLDS:
    print(f"{m:18} mean={agg[m]:.3f}  thr={THRESHOLDS[m]:.2f}  {verdict[m]}")
print("OVERALL:", overall)

faithfulness       mean=0.950  thr=0.90  PASS
answer_relevance   mean=0.850  thr=0.80  PASS
context_precision  mean=0.737  thr=0.70  PASS
context_recall     mean=0.833  thr=0.80  PASS
OVERALL: PASS

7 · Tech-lead — eval-gating in CI & avoiding overfitting tech-lead

A lead turns the scorecard into a gate: CI runs the eval set on every change and blocks the merge if any metric regresses below a tolerance of the last known-good baseline. This is the RAG equivalent of a failing unit test — a prompt tweak that quietly tanks context recall never reaches production. The gate is just an exit code the CI runner checks.

ci_gate.pyimport sys

BASELINE = {"faithfulness": 0.92, "answer_relevance": 0.82,
            "context_precision": 0.72, "context_recall": 0.81}
# regression tolerance: a metric may not drop more than this below baseline
TOLERANCE = 0.03

def ci_gate(current, baseline=BASELINE, tolerance=TOLERANCE):
    regressions = []
    for m, base in baseline.items():
        if current[m] < base - tolerance:
            regressions.append(f"{m}: {current[m]:.2f} < {base - tolerance:.2f}")
    if regressions:
        print("RAG EVAL GATE: FAIL")
        for r in regressions:
            print("  regressed ->", r)
        return 1
    print("RAG EVAL GATE: PASS")
    return 0

current = {"faithfulness": 0.93, "answer_relevance": 0.83,
           "context_precision": 0.60, "context_recall": 0.82}

exit_code = ci_gate(current)
print("exit code:", exit_code)
sys.exit(exit_code)

RAG EVAL GATE: FAIL
  regressed -> context_precision: 0.60 < 0.69
exit code: 1

Here faithfulness and relevance held, but context precision cratered — the gate fails the build and names the retriever as the culprit. That's the payoff of four metrics: the failure message points at the half to fix. (Run echo $? after this file and you'll see 1 — exactly what a CI runner reads to fail a job.)

def context_recall(relevant_ids, retrieved_ids):
    relevant = set(relevant_ids)
    if not relevant:
        return 1.0
    hit = relevant & set(retrieved_ids)
    return len(hit) / len(relevant)

print(context_recall(["c1","c3"], ["c1","c2","c3","c4"]))  # 1.0  (both found)
print(context_recall(["c1","c3"], ["c2","c1","c5"]))       # 0.5  (missed c3)

def context_precision(relevant_ids, retrieved_ids):
    if not retrieved_ids:
        return 0.0
    hit = set(relevant_ids) & set(retrieved_ids)
    return len(hit) / len(retrieved_ids)

rel = ["c1","c3"]
print(context_precision(rel, ["c1","c3"]))            # 1.0  (tight)
print(context_precision(rel, ["c1","c3","c8","c9"]))  # 0.5  (padded with junk)

def faithfulness(answer, contexts):
    claims = [c.strip() for c in answer.split(".") if c.strip()]
    ctx = " ".join(contexts).lower()
    grounded = 0
    for claim in claims:
        # crude grounding proxy: majority of content words appear in context
        words = [w for w in claim.lower().split() if len(w) > 3]
        hits = sum(1 for w in words if w in ctx)
        if words and hits / len(words) >= 0.6:
            grounded += 1
    return grounded / len(claims) if claims else 1.0

ctx = ["Refunds are issued within 5 business days to the original card."]
ans = "Refunds are issued within 5 business days. You get store credit instead."
print(round(faithfulness(ans, ctx), 2))   # 0.5 — 2nd claim unsupported (hallucination)

def recall(rel, ret): return len(set(rel)&set(ret))/len(set(rel)) if rel else 1.0
def precision(rel, ret): return len(set(rel)&set(ret))/len(ret) if ret else 0.0

samples = [
    dict(rel=["c1"], ret=["c1","c2"], faith=1.0, relevance=0.9),
    dict(rel=["c3"], ret=["c9"],      faith=0.4, relevance=0.8),
]
agg = {"context_recall":0,"context_precision":0,"faithfulness":0,"answer_relevance":0}
for s in samples:
    agg["context_recall"]    += recall(s["rel"], s["ret"])
    agg["context_precision"] += precision(s["rel"], s["ret"])
    agg["faithfulness"]      += s["faith"]
    agg["answer_relevance"]  += s["relevance"]
agg = {k: round(v/len(samples),2) for k,v in agg.items()}
THRESH = {"context_recall":0.8,"context_precision":0.6,"faithfulness":0.8,"answer_relevance":0.7}
failed = [k for k,v in agg.items() if v < THRESH[k]]
print(agg)
print("GATE:", "PASS" if not failed else "FAIL -> " + ", ".join(failed))

def validate_evalset(rows):
    good, bad = [], []
    for r in rows:
        problems = []
        if not r.get("question"): problems.append("no question")
        if not r.get("reference_answer"): problems.append("no reference answer")
        if not r.get("relevant_chunk_ids"): problems.append("no relevant ids")
        (bad if problems else good).append((r.get("question","?"), problems))
        if not problems: good[-1] = r
    ok = [r for r in good if isinstance(r, dict)]
    return ok, [b for b in bad]

rows = [
    {"question":"Refund window?","reference_answer":"5 days","relevant_chunk_ids":["c1"]},
    {"question":"","reference_answer":"x","relevant_chunk_ids":["c2"]},   # bad
    {"question":"2FA?","reference_answer":"toggle","relevant_chunk_ids":[]}, # bad
]
ok, bad = validate_evalset(rows)
print(f"usable: {len(ok)}   rejected: {len(rows)-len(ok)}")

THRESH = {"faithfulness":0.85,"answer_relevance":0.75,
          "context_recall":0.80,"context_precision":0.60}

def ci_gate(scores, prev_scores=None):
    failed = [k for k,v in scores.items() if v < THRESH[k]]
    decision = "PASS" if not failed else "FAIL -> fix: " + ", ".join(failed)
    warn = ""
    # overfitting guard: suspiciously high on a static set that never rotates
    if prev_scores and all(scores[k] >= 0.97 for k in scores):
        warn = " WARN: near-perfect on a static eval set — rotate held-out questions"
    return decision + warn

now  = {"faithfulness":0.88,"answer_relevance":0.79,"context_recall":0.82,"context_precision":0.65}
print(ci_gate(now))   # PASS
perfect = {k:0.99 for k in THRESH}
print(ci_gate(perfect, prev_scores=perfect))  # PASS WARN: ... rotate

Knowledge check check yourself