Production RAG systems

1 · The full production architecture

Chapters 3.1–3.4 built the parts. A production system is those parts in a line, with two wrappers around them that beginners skip and professionals never do: observability (you can see every stage) and evaluation (you can prove the whole thing works). Here is the representative shape — the online query path, left to right:

That online path sits on top of an offline ingestion path (chunk → embed → vector store, from 3.2/3.4) that keeps the index fresh. And the whole loop — both paths — is wrapped in observability + evals: you log every query's retrieved chunks, scores, prompt, and answer, and you run an eval set against the system continuously. The wrappers are the difference between a demo and a system:

2 · Evaluation — how you know it works

“It looks good in the demo” is not evaluation. RAG has two things to measure, and they fail independently: did retrieval find the right chunks, and did the model answer faithfully from them? Split the metrics accordingly.

The retrieval metrics are just arithmetic over a labeled set — for a handful of questions you record which chunk ids are truly relevant, then compare against what the retriever returned. Here is recall@k and MRR from scratch, on fixed data so it runs offline:

eval_metrics.py# Retrieval evaluation from a small LABELED set — pure Python, deterministic.
# For each query we know which chunk ids are relevant (the 'gold' set), and we
# have the ranked list our retriever returned. We compute recall@k and MRR.

def recall_at_k(retrieved, relevant, k):
    """Fraction of the relevant chunks that appear in the top-k retrieved."""
    top = retrieved[:k]
    found = sum(1 for r in relevant if r in top)
    return found / (len(relevant) or 1)

def reciprocal_rank(retrieved, relevant):
    """1 / rank of the FIRST relevant chunk (rank counts from 1); 0 if none."""
    for i, doc_id in enumerate(retrieved, start=1):
        if doc_id in relevant:
            return 1.0 / i
    return 0.0

# A tiny labeled eval set: (query, ranked retrieved ids, set of relevant ids)
eval_set = [
    ('restart checkout',      [3, 1, 7, 2], {1}),        # relevant at rank 2
    ('where are invoices',    [5, 4, 9, 8], {5}),        # relevant at rank 1
    ('clear the queue',       [2, 6, 0, 1], {0, 1}),     # two relevant, ranks 3 & 4
    ('rollback a deploy',     [8, 3, 4, 6], {9}),        # relevant NOT retrieved
]

k = 3
recalls, rrs = [], []
for q, retrieved, relevant in eval_set:
    r = recall_at_k(retrieved, relevant, k)
    rr = reciprocal_rank(retrieved, relevant)
    recalls.append(r); rrs.append(rr)
    print(f'{q:20} recall@{k}={r:.2f}  RR={rr:.3f}')

print(f'\nmean recall@{k} : {sum(recalls)/len(recalls):.3f}')
print(f'MRR            : {sum(rrs)/len(rrs):.3f}')

restart checkout     recall@3=1.00  RR=0.500
where are invoices   recall@3=1.00  RR=1.000
clear the queue      recall@3=0.50  RR=0.333
rollback a deploy    recall@3=0.00  RR=0.000

mean recall@3 : 0.625
MRR            : 0.458

Read the failures, not just the average. The “clear the queue” query found only one of its two relevant chunks in the top-3 (recall 0.5), and “rollback a deploy” missed entirely (recall 0, RR 0) — chunk 9 was never retrieved, so no re-ranker or prompt could have saved that answer. The mean recall@3 of 0.625 and MRR of 0.458 are the numbers you track over time; a drop in either flags a retrieval regression. Building a labeled set is the unglamorous work that makes every later decision measurable.

The answer side is harder to score with arithmetic, because “is this claim supported?” needs judgement. The standard production trick is LLM-as-judge: you give a model the question, the retrieved context, and the answer, and ask it to rate faithfulness — is every statement grounded in the context? This needs a real model, so the code below is complete but does not run in the sandbox:

llm_judge.py# Faithfulness via LLM-as-judge. Complete and correct; needs YOUR API key to run.
# pip install anthropic ; export ANTHROPIC_API_KEY=sk-...
from anthropic import Anthropic          # runs in your own environment, not the sandbox

client = Anthropic()

JUDGE = '''You are grading FAITHFULNESS. Given the CONTEXT and an ANSWER,
reply with a single number 0.0-1.0: the fraction of the answer's claims that
are directly supported by the context. Do not use outside knowledge.

CONTEXT:
{context}

ANSWER:
{answer}

Score:'''

def faithfulness(context, answer):
    msg = client.messages.create(
        model='claude-opus-4-8',           # any current model id
        max_tokens=8,
        messages=[{'role': 'user',
                   'content': JUDGE.format(context=context, answer=answer)}],
    )
    return float(msg.content[0].text.strip())

# score = faithfulness(context, answer)   ->  e.g. 0.5 if half the claims are invented

3 · Evals in CI — gating deploys

Metrics you compute once and forget don't protect you. The professional move is to run the eval set in CI, exactly like a test suite: any change that touches retrieval quality must clear the bar before it ships. RAG has an unusually large blast radius — three very different changes can all silently wreck answers:

4 · Guardrails — keeping answers honest

Evals tell you how the system does on average. Guardrails protect the individual answer at request time. Four are standard in production RAG:

The grounding check is the one you can build in pure Python right now. The idea: split the answer into sentences and flag any sentence that no retrieved chunk supports. Here we approximate “supported” with content-word overlap; a real system swaps in an entailment model or the LLM judge, but the shape is identical:

grounding.py# Grounding check: flag any ANSWER sentence not supported by a retrieved chunk.
# 'Supported' = the sentence shares enough content words with some chunk.
# Pure Python, deterministic. A real system would use an entailment model here.
import re

def content_words(s):
    stop = {'the','a','an','to','and','or','is','are','of','in','on','it','then','up','by'}
    return {w for w in re.findall(r'[a-z]+', s.lower()) if w not in stop}

def sentences(text):
    return [s.strip() for s in re.split(r'(?<=[.!?])\s+', text.strip()) if s.strip()]

def supported(sentence, chunks, thresh=0.5):
    """A sentence is grounded if some chunk covers >= thresh of its content words."""
    sw = content_words(sentence)
    if not sw:
        return True                      # nothing to ground (e.g. 'OK.')
    for c in chunks:
        cov = len(sw & content_words(c)) / len(sw)
        if cov >= thresh:
            return True
    return False

chunks = [
    'To restart checkout, drain the queue then scale the consumer group.',
    'Never restart the database to fix a queue backlog.',
]
answer = ('Drain the queue and scale the consumer group to restart checkout. '
          'You should also email the CEO for approval first.')   # 2nd sentence is invented

flagged = []
for s in sentences(answer):
    ok = supported(s, chunks)
    print(f'[{"GROUNDED" if ok else "UNSUPPORTED"}] {s}')
    if not ok:
        flagged.append(s)

print(f'\n{len(flagged)} unsupported sentence(s) flagged for review.')

[GROUNDED] Drain the queue and scale the consumer group to restart checkout.
[UNSUPPORTED] You should also email the CEO for approval first.

1 unsupported sentence(s) flagged for review.

The first sentence overlaps the retrieved steps and passes; the invented “email the CEO” sentence shares no supporting chunk and is flagged. In production you decide what to do with a flag — block the answer, strip the unsupported sentence, or surface a warning — but first you have to detect it, and that is a cheap check you can run on every response. Combined with a retrieval-score threshold (refuse when the best chunk is too weak, from 3.1) and citation verification, this is the honesty layer.

5 · Failure modes & mitigations

Production RAG fails in specific, recurring ways. Knowing the catalogue is half the battle — most outages are one of these, not a novel mystery:

6 · Cost & latency budgets

Every production RAG query spends time and money across four stages. Thinking in an explicit per-query budget tells you where to optimise. The numbers below are illustrative placeholders to show the shape of the reasoning — plug in current pricing and measure your own; the relative ordering (generation usually dominates) is the durable lesson, not the digits:

7 · Operations — monitor, alert, improve

A production RAG system is never “done” — the corpus grows, queries drift, and models get swapped. The operations loop keeps quality from silently rotting: