Production RAG systems

1 · The full production architecture

Here is the moment that separates a weekend RAG demo from a system people trust with real questions. The demo works: you paste in three PDFs, ask a question, and a fluent answer comes back. So you ship it — and three weeks later someone in support forwards you a screenshot of your bot confidently citing a policy that was retired last year, to a customer who then acted on it. Nothing "broke." No exception, no red log line. The pipeline did exactly what you built it to do; you just never built the parts that would have caught it. That gap is what this whole chapter is about.

Think of it like the difference between a go-kart and a car you'd put your family in. Both have an engine, wheels, and a steering wheel — the go-kart even goes faster around the parking lot. What the car adds is everything that isn't the drivetrain: the speedometer so you know what's happening, the brakes and airbags for when it goes wrong, the annual inspection that proves it still works. Your 3.1–3.4 pipeline is the drivetrain. A production system is that same drivetrain plus the instruments and safety systems wrapped around it — and, crucially, the wrapping is where most of the engineering actually lives.

The common mistake is to treat those wrappers as "nice to have later," bolted on after launch when there's time. There never is time, and by then you're debugging blind: a user complains, and you have no log of what was retrieved, no metric to tell you whether retrieval or generation is at fault, and no baseline to compare against. The professional habit is to build the two wrappers first, or at least alongside the pipeline, precisely because they're the parts that pay off only when something goes wrong — and in front of real users, something always eventually goes wrong.

Chapters 3.1–3.4 built the parts. A production system is those parts in a line, with two wrappers around them that beginners skip and professionals never do: observability (you can see every stage) and evaluation (you can prove the whole thing works). Here is the representative shape — the online query path, left to right:

That online path sits on top of an offline ingestion path (chunk → embed → vector store, from 3.2/3.4) that keeps the index fresh. And the whole loop — both paths — is wrapped in observability + evals: you log every query's retrieved chunks, scores, prompt, and answer, and you run an eval set against the system continuously. The wrappers are the difference between a demo and a system:

2 · Evaluation — how you know it works

Ask yourself an uncomfortable question: right now, could you prove your RAG system got better after your last change — or do you just feel like it did? Most teams can't answer. They tweak a prompt, run a couple of questions by hand, nod, and ship. That's not evaluation; that's vibes. And vibes are exactly how a system silently degrades — each change feels fine in isolation, and three months later the answers are noticeably worse and nobody can say which change did it.

Evaluation is just the practice of measuring quality with a fixed, repeatable test instead of your gut. The analogy is a blood test versus "I feel okay." You might feel fine while a number quietly drifts into the danger zone; the test catches what your feelings can't. An eval set is that blood panel for your RAG system — the same questions, scored the same way, every time you change something — so "is this better?" becomes a number you can compare, not an argument you have in a meeting.

Here's the insight beginners miss, and it's the reason this section exists: a RAG system has two organs that can fail completely independently, and a single overall "does it seem good" judgement blurs them together. Retrieval can be perfect while the model hallucinates on top of good context; or the model can be beautifully faithful to garbage chunks that retrieval never should have surfaced. If you only look at the final answer, you can't tell which one is broken — so you can't fix it. The common mistake is measuring only the end result and then flailing when it's wrong, tuning the prompt when the real problem was recall, or vice versa. Split the measurement by organ and each failure points straight at its own fix.

“It looks good in the demo” is not evaluation. RAG has two things to measure, and they fail independently: did retrieval find the right chunks, and did the model answer faithfully from them? Split the metrics accordingly.

The retrieval metrics are just arithmetic over a labeled set — for a handful of questions you record which chunk ids are truly relevant, then compare against what the retriever returned. Here is recall@k and MRR from scratch, on fixed data so it runs offline:

eval_metrics.py# Retrieval evaluation from a small LABELED set — pure Python, deterministic.
# For each query we know which chunk ids are relevant (the 'gold' set), and we
# have the ranked list our retriever returned. We compute recall@k and MRR.

def recall_at_k(retrieved, relevant, k):
    """Fraction of the relevant chunks that appear in the top-k retrieved."""
    top = retrieved[:k]
    found = sum(1 for r in relevant if r in top)
    return found / (len(relevant) or 1)

def reciprocal_rank(retrieved, relevant):
    """1 / rank of the FIRST relevant chunk (rank counts from 1); 0 if none."""
    for i, doc_id in enumerate(retrieved, start=1):
        if doc_id in relevant:
            return 1.0 / i
    return 0.0

# A tiny labeled eval set: (query, ranked retrieved ids, set of relevant ids)
eval_set = [
    ('restart checkout',      [3, 1, 7, 2], {1}),        # relevant at rank 2
    ('where are invoices',    [5, 4, 9, 8], {5}),        # relevant at rank 1
    ('clear the queue',       [2, 6, 0, 1], {0, 1}),     # two relevant, ranks 3 & 4
    ('rollback a deploy',     [8, 3, 4, 6], {9}),        # relevant NOT retrieved
]

k = 3
recalls, rrs = [], []
for q, retrieved, relevant in eval_set:
    r = recall_at_k(retrieved, relevant, k)
    rr = reciprocal_rank(retrieved, relevant)
    recalls.append(r); rrs.append(rr)
    print(f'{q:20} recall@{k}={r:.2f}  RR={rr:.3f}')

print(f'\nmean recall@{k} : {sum(recalls)/len(recalls):.3f}')
print(f'MRR            : {sum(rrs)/len(rrs):.3f}')

restart checkout     recall@3=1.00  RR=0.500
where are invoices   recall@3=1.00  RR=1.000
clear the queue      recall@3=0.50  RR=0.333
rollback a deploy    recall@3=0.00  RR=0.000

mean recall@3 : 0.625
MRR            : 0.458

Read the failures, not just the average. The “clear the queue” query found only one of its two relevant chunks in the top-3 (recall 0.5), and “rollback a deploy” missed entirely (recall 0, RR 0) — chunk 9 was never retrieved, so no re-ranker or prompt could have saved that answer. The mean recall@3 of 0.625 and MRR of 0.458 are the numbers you track over time; a drop in either flags a retrieval regression. Building a labeled set is the unglamorous work that makes every later decision measurable.

The answer side is harder to score with arithmetic, because “is this claim supported?” needs judgement. The standard production trick is LLM-as-judge: you give a model the question, the retrieved context, and the answer, and ask it to rate faithfulness — is every statement grounded in the context? This needs a real model, so the code below is complete but does not run in the sandbox:

llm_judge.py# Faithfulness via LLM-as-judge. Complete and correct; needs YOUR API key to run.
# pip install anthropic ; export ANTHROPIC_API_KEY=sk-...
from anthropic import Anthropic          # runs in your own environment, not the sandbox

client = Anthropic()

JUDGE = '''You are grading FAITHFULNESS. Given the CONTEXT and an ANSWER,
reply with a single number 0.0-1.0: the fraction of the answer's claims that
are directly supported by the context. Do not use outside knowledge.

CONTEXT:
{context}

ANSWER:
{answer}

Score:'''

def faithfulness(context, answer):
    msg = client.messages.create(
        model='claude-opus-4-8',           # any current model id
        max_tokens=8,
        messages=[{'role': 'user',
                   'content': JUDGE.format(context=context, answer=answer)}],
    )
    return float(msg.content[0].text.strip())

# score = faithfulness(context, answer)   ->  e.g. 0.5 if half the claims are invented

llm_judge.py# Faithfulness via LLM-as-judge. Complete and correct; needs YOUR API key to run.
# pip install openai ; export OPENAI_API_KEY=sk-...
from openai import OpenAI                 # runs in your own environment, not the sandbox

client = OpenAI()

JUDGE = '''You are grading FAITHFULNESS. Given the CONTEXT and an ANSWER,
reply with a single number 0.0-1.0: the fraction of the answer's claims that
are directly supported by the context. Do not use outside knowledge.

CONTEXT:
{context}

ANSWER:
{answer}

Score:'''

def faithfulness(context, answer):
    resp = client.responses.create(
        model='gpt-5.5',                   # any current model id
        max_output_tokens=8,
        input=JUDGE.format(context=context, answer=answer),
    )
    return float(resp.output_text.strip())

# score = faithfulness(context, answer)   ->  e.g. 0.5 if half the claims are invented

3 · Evals in CI — gating deploys

A metric you run once and admire is a photograph; a metric you run automatically on every change is a smoke detector. The difference is everything. You can build the loveliest eval set in section 2, score your system at 0.9 faithfulness, feel great — and then watch it rot, because that number was a one-time snapshot and nobody is checking it against the next fifty commits. Quality doesn't collapse in one dramatic bad deploy; it erodes one "small, obviously safe" change at a time, and by the time a human notices, you've lost the trail of which change did it.

The fix is to treat evaluation exactly like you treat unit tests. You'd never let a teammate merge code that breaks the test suite; a red build blocks the merge automatically, no debate. A regression gate is that same idea aimed at answer quality: the eval set runs in your continuous-integration pipeline on every proposed change, and if the numbers drop below your baseline, the build fails and the change can't ship. It converts "I think this prompt is better" — an opinion you'd have to defend in review — into "the numbers say it's at least as good," a fact the machine checks for you.

Why RAG needs this more than ordinary software is the sneaky part, and it's the mistake that catches even experienced teams: in normal code, a change usually breaks something visibly — a test fails, a page errors. In RAG the blast radius is huge and silent. A change to how you split documents, a swap to a newer embedding model, a two-word edit to the system prompt — each looks harmless, touches a different part of the stack, and can quietly tank answer quality with no error anywhere. The whole point of gating in CI is to make that invisible damage visible before it reaches a user, by measuring it the moment the change is proposed.

Metrics you compute once and forget don't protect you. The professional move is to run the eval set in CI, exactly like a test suite: any change that touches retrieval quality must clear the bar before it ships. RAG has an unusually large blast radius — three very different changes can all silently wreck answers:

4 · Guardrails — keeping answers honest

Your eval set can say the system is 95% faithful and the customer in front of you can still be reading the 5% that isn't. Averages comfort the team; they do nothing for the one person getting the wrong answer right now. That's the hard truth guardrails exist to face: evaluation is about the population of answers over time, but a real user only ever experiences a single answer — and they don't care about your mean, they care whether this one is right.

The distinction is like the difference between a factory's quality-control statistics and the seatbelt in the car you're actually sitting in. QC sampling tells the manufacturer that 99.9% of cars are safe — genuinely useful for improving the production line. But it does nothing for you in a crash; that's the seatbelt's job, and the seatbelt acts on your car, in your moment, regardless of the fleet average. Evals are your QC statistics. Guardrails are the seatbelt: cheap checks that fire on each individual response, at request time, to stop a bad one from reaching the user even when the overall system is healthy.

The mistake here is assuming good evaluation makes guardrails redundant — "our faithfulness score is high, so we don't need runtime checks." But a high average guarantees there's a tail of bad answers; that's just what an average is. The rare hallucination, the citation to a chunk that doesn't exist, the retrieved document that happens to contain someone else's private data — none of those show up in an aggregate score, and each of them is a specific, catchable event at the moment the answer is built. Guardrails are how you catch them one at a time. The four below are the standard set, and the beauty is they're mostly cheap, deterministic checks you can afford to run on every request. Evals tell you how the system does on average. Guardrails protect the individual answer at request time. Four are standard in production RAG:

The grounding check is the one you can build in pure Python right now. The idea: split the answer into sentences and flag any sentence that no retrieved chunk supports. Here we approximate “supported” with content-word overlap; a real system swaps in an entailment model or the LLM judge, but the shape is identical:

grounding.py# Grounding check: flag any ANSWER sentence not supported by a retrieved chunk.
# 'Supported' = the sentence shares enough content words with some chunk.
# Pure Python, deterministic. A real system would use an entailment model here.
import re

def content_words(s):
    stop = {'the','a','an','to','and','or','is','are','of','in','on','it','then','up','by'}
    return {w for w in re.findall(r'[a-z]+', s.lower()) if w not in stop}

def sentences(text):
    return [s.strip() for s in re.split(r'(?<=[.!?])\s+', text.strip()) if s.strip()]

def supported(sentence, chunks, thresh=0.5):
    """A sentence is grounded if some chunk covers >= thresh of its content words."""
    sw = content_words(sentence)
    if not sw:
        return True                      # nothing to ground (e.g. 'OK.')
    for c in chunks:
        cov = len(sw & content_words(c)) / len(sw)
        if cov >= thresh:
            return True
    return False

chunks = [
    'To restart checkout, drain the queue then scale the consumer group.',
    'Never restart the database to fix a queue backlog.',
]
answer = ('Drain the queue and scale the consumer group to restart checkout. '
          'You should also email the CEO for approval first.')   # 2nd sentence is invented

flagged = []
for s in sentences(answer):
    ok = supported(s, chunks)
    print(f'[{"GROUNDED" if ok else "UNSUPPORTED"}] {s}')
    if not ok:
        flagged.append(s)

print(f'\n{len(flagged)} unsupported sentence(s) flagged for review.')

[GROUNDED] Drain the queue and scale the consumer group to restart checkout.
[UNSUPPORTED] You should also email the CEO for approval first.

1 unsupported sentence(s) flagged for review.

The first sentence overlaps the retrieved steps and passes; the invented “email the CEO” sentence shares no supporting chunk and is flagged. In production you decide what to do with a flag — block the answer, strip the unsupported sentence, or surface a warning — but first you have to detect it, and that is a cheap check you can run on every response. Combined with a retrieval-score threshold (refuse when the best chunk is too weak, from 3.1) and citation verification, this is the honesty layer.

5 · Failure modes & mitigations

The scariest thing about a broken RAG system is not that it fails — it's that it fails while sounding completely certain. A traditional service that breaks throws an error you can grep for; a RAG system that breaks hands the user a fluent, confident paragraph that happens to be wrong, and everyone downstream believes it. There's no stack trace for "the model answered from the wrong chunk." So the first survival skill isn't clever engineering — it's recognition: knowing the handful of ways this specific kind of system goes wrong, so that when a weird report lands on your desk you can name it in seconds instead of debugging from zero.

Here's the reassuring part, and the reason a plain catalogue is worth memorizing: RAG failures are not an infinite space of unique mysteries. They're a small, recurring cast of characters. It's the same logic a doctor uses — the vast majority of a patient's symptoms map to a short list of common conditions, so an experienced clinician pattern-matches quickly and only reaches for exotic explanations when the common ones are ruled out. When a user says "the answer was confidently wrong," you shouldn't start from a blank page; you should run down a mental checklist: was the right chunk even retrieved? Did two chunks contradict each other? Is the index stale? Was the "document" secretly an attacker's instructions? Cost blowing up?

The common mistake is treating each incident as brand new and reinventing the diagnosis every time — which is slow, and worse, means you never build the reflex to defend against the failure before it happens. Each mode below comes paired with its standard mitigation precisely so you can install the defense proactively, not scramble for it during an outage. Pay special attention to the injection row: it's the one that doesn't feel like a failure mode at all until you realize the "documents" you're pasting into your prompt are text you didn't write. Production RAG fails in specific, recurring ways. Knowing the catalogue is half the battle — most outages are one of these, not a novel mystery:

6 · Cost & latency budgets

Almost every engineer's first instinct for "make it faster" is to optimise the part they most recently touched — and that instinct is usually wrong. You spent a week tuning the vector search, so when latency complaints arrive you go shave milliseconds off the search. Meanwhile the LLM call sitting quietly at the end of the pipeline is eating the overwhelming majority of both the time and the bill, untouched, because it's not the part you were thinking about. This is the single most common way RAG teams waste optimisation effort: fixing the stage that's convenient rather than the stage that's expensive.

A budget is the discipline that fixes this, and it's exactly what it sounds like — the same thing a household budget does. If you're trying to save money, you don't agonize over the price of coffee while ignoring the rent; you write down where every dollar actually goes, and then you attack the biggest line item first. A per-query cost/latency budget is that ledger for a single request: you account for the time and money each stage spends, and the ledger tells you — often surprisingly — that one stage dwarfs the rest. Optimise that one, and ignore the rest until it matters.

For RAG, the ledger almost always names the same culprit: generation, the LLM call, is usually the largest single cost, because generating tokens is far slower and pricier than embedding a query or searching an index. That's why the durable lesson of this section is a ranking, not a set of numbers. The mistake is to memorize specific latency figures — they change with every model release and pricing update, so any number here is a placeholder. What doesn't change is the shape: measure your own pipeline end to end, find the dominant stage (it's probably generation), and spend your optimisation budget there. Every production RAG query spends time and money across four stages. Thinking in an explicit per-query budget tells you where to optimise. The numbers below are illustrative placeholders to show the shape of the reasoning — plug in current pricing and measure your own; the relative ordering (generation usually dominates) is the durable lesson, not the digits:

7 · Operations — monitor, alert, improve

The most dangerous day for a RAG system is the day you stop watching it, because that's the day it starts getting worse without telling you. A launched system feels finished — the code is written, the demo impressed leadership, the eval set passed. But a RAG system isn't a bridge you build once and walk away from; it's a garden. The world it draws on keeps changing underneath it: new documents get added, old ones go stale, the questions users ask this month aren't the ones you tested on last month, and one day someone swaps the underlying model. Left alone, a garden doesn't stay the same — it degrades. So does retrieval quality.

The reason this decay is so insidious is that it's gradual. Nothing crashes. Each week the answers are a fraction less useful than the week before, far too slowly for any single user to file a bug, until one day the system is visibly bad and you're shocked because "nothing changed." Something did change — the world did — and you weren't measuring the drift. This is why operations is a loop, not a launch: serve answers, watch what happens, notice when the numbers slip, learn from the failures, improve, and serve again. Each lap makes the system a little more fit for the reality it's actually operating in.

The mistake teams make is imagining operations as passive babysitting — "we'll check the dashboards sometimes." The powerful version is active and closes back on everything you've already built: real user failures (the thumbs-down, the escalated ticket) get funneled straight into the eval set from section 2, which makes the CI gate from section 3 catch that exact failure forever after. That's the whole architecture eating its own tail productively — production surfaces the failures your test set didn't imagine, and each one becomes a permanent test. A mature RAG system isn't the one with the cleverest retrieval; it's the one whose feedback loop is tightest. A production RAG system is never “done” — the corpus grows, queries drift, and models get swapped. The operations loop keeps quality from silently rotting: