Build a RAG System From Scratch

What RAG is, and when to use it intermediate

Here's a moment you've almost certainly lived through. You ask a chatbot about your own company's refund policy, and it answers with total confidence — polished, fluent, and completely wrong. It has never seen your policy; it's pattern-matching on the thousand generic policies it read during training. The model isn't lying on purpose. It simply has no way to know what it doesn't know, so it fills the gap with something plausible. That single failure — a confident answer built on absent knowledge — is the exact problem RAG exists to solve.

Think of a bright new hire on their first day. Ask them anything about your internal systems and they'll either guess or, if they're wise, say "let me look it up." RAG is that "let me look it up" reflex, bolted onto the model. Instead of forcing the LLM to answer from memory, you first go fetch the relevant pages from your documents, hand them over, and say "answer using only these." The model's fluent-writing skill is still doing the talking — but the facts now come from a source you control and can point to.

The name spells out the recipe: Retrieval-Augmented Generation. You retrieve relevant snippets, augment the prompt with them, then let the model generate the answer. The common beginner mistake is to think RAG is a fancier model or a training technique — it is neither. It's a plumbing pattern that wraps an ordinary model: nothing about the model changes, you just change what you put in front of it at question time. That's why RAG updates instantly when your docs change (no retraining) and why it can cite its sources (it knows exactly which snippets it used).

So when do you actually reach for it? The table below is the decision you'll make on nearly every project. The honest headline: RAG wins when knowledge is large, changes often, or must be attributable — and its whole game becomes getting the right snippets in front of the model. Which is why, as the danger box warns, a "hallucinating" RAG system is usually a retrieval bug wearing a generation costume.

An LLM only knows what was in its training data. RAG lets it answer questions about your private/current documents by retrieving relevant snippets at query time and putting them in the prompt. The model then answers from that supplied context.

The two-phase pipeline intermediate

Picture a good research librarian. Long before you ever walk in, they've done the slow work: read the collection, catalogued it, and shelved everything so any book can be found in seconds. Then you arrive with a question, and the fast work happens — they figure out what you're really asking, pull the three most relevant books, and hand you the exact pages. Two completely different jobs, done at completely different times. RAG is built the same way, and the single most common source of confusion for beginners is smashing those two jobs into one mental blob.

So hold them apart deliberately. The first phase is offline / indexing: you take your documents, cut them into chunks, turn each chunk into a vector, and store them — once, ahead of time. This is the librarian shelving the collection. It can be slow and expensive because it happens on your schedule, not the user's, and you reuse the result across millions of questions. The second phase is online / query time: a question arrives, you embed it, retrieve the nearest chunks, optionally re-rank them, paste the winners into the prompt, and generate a cited answer. This is the librarian answering you — and it has to be fast, because a human is waiting.

Why does the split matter so much in practice? Because the two phases fail differently and cost differently. If retrieval quality is bad, the fix is almost always in the offline phase — your chunking or your embedding model — even though the symptom shows up online as a wrong answer. And the offline/online boundary is where your engineering budget lives: heavy computation (re-embedding a million documents) belongs offline where nobody is watching the clock; the online path should do as little work as possible. The classic mistake is treating RAG as one linear script and then wondering why it's both slow at query time and hard to debug — you've entangled the shelving with the answering. The diagram below draws the two timelines explicitly, with a dashed line marking exactly where "built once" meets "run per question."

Lab 3.1 · Chunking intermediate

Imagine handing someone a 300-page manual and asking "where's the warranty clause?" — but they're only allowed to photocopy a few pages to hand back. If they copy whole chapters, the warranty clause is in there somewhere but buried in noise. If they copy single sentences, they might snip the clause in half and miss the condition that makes it apply. Getting the size of those photocopies right is the entire skill of chunking, and it quietly decides whether your RAG system works before a single embedding is computed.

Here's why it's so pivotal. Your model never searches whole documents — it searches chunks, and it can only ever answer from the chunks retrieval hands it. A chunk is the unit of everything downstream: the thing you embed, the thing you rank, the thing you cite. So a chunk that's badly cut is a fact your system can never retrieve correctly, no matter how good your embeddings or your prompt. That's why experienced RAG builders say chunking is the highest-leverage lever on the whole board: it's upstream of every other component, so its mistakes propagate everywhere.

The tension is a genuine trade-off with no free answer. Chunk too big and each passage carries several ideas at once; its embedding becomes a blurry average of all of them, so it matches everything weakly and nothing strongly — and even when it's retrieved, the real answer is diluted by surrounding irrelevant text that wastes tokens. Chunk too small and you get razor-sharp matches, but you routinely split a single fact across a boundary: the "how" lands in chunk 4 and the "but only if" lands in chunk 5, and retrieval brings back just one of them. The most common rookie error is cutting on a blind fixed character count (say, "every 500 characters") which slices mid-sentence and mid-table, shredding meaning at every boundary.

The fix that the lab below implements has three moves. First, cut on semantic boundaries — paragraphs and headings — so each chunk is a coherent thought, not an arbitrary slice. Second, keep a small overlap between neighbours so a fact sitting right on a cut survives whole in at least one chunk. Third, attach metadata (source file, an id) to every chunk so you can cite it and filter it later. Read the code with those three goals in mind: paragraphs packed to a target size, the last paragraph carried forward as overlap, and each chunk wrapped in a dictionary that remembers where it came from.

Models retrieve chunks, not whole documents. Good chunking is the highest-leverage decision in RAG. Too big → irrelevant text dilutes the answer and wastes tokens. Too small → facts get split across chunks.

chunk.pyimport os, glob, re

def chunk_text(text, source, target_words=120, overlap=25):
    """Split on paragraphs, then pack into ~target_words chunks with overlap."""
    paras = [p.strip() for p in re.split(r"\n\s*\n", text) if p.strip()]
    chunks, buf = [], []
    for para in paras:
        buf.append(para)
        if sum(len(p.split()) for p in buf) >= target_words:
            chunks.append(" ".join(buf))
            buf = buf[-1:] if overlap else []   # carry last para as overlap
    if buf: chunks.append(" ".join(buf))
    return [{"text": c, "source": source, "id": f"{source}#{i}"}
            for i, c in enumerate(chunks)]

def load_corpus(folder="docs"):
    out = []
    for path in glob.glob(f"{folder}/*"):
        with open(path) as f:
            out += chunk_text(f.read(), os.path.basename(path))
    return out

if __name__ == "__main__":
    c = load_corpus()
    print(f"{len(c)} chunks"); print(c[0])

Lab 3.2 · Embed & store intermediate

Think about how you'd organise a giant music library by "vibe." You can't sort songs alphabetically and expect similar moods to end up near each other — "Yesterday" and "Let It Be" would be aisles apart. What you'd really want is a map where meaning is the coordinate: mellow acoustic ballads cluster in one region, thrash metal in another, and songs that feel alike sit close together regardless of their titles. That map, for text, is exactly what an embedding gives you — and it's the trick that lets a computer find documents by what they mean rather than which words they happen to share.

Concretely, an embedding is a function that turns a piece of text into a fixed-length list of numbers — a vector, a point in high-dimensional space. It's trained so that texts with similar meaning land at nearby points and unrelated texts land far apart. The magic property is that this survives paraphrase: "how do I change my password" and "steps to update my login credentials" share almost no words, yet their vectors sit right next to each other because they mean the same thing. Keyword search would score them as unrelated; embeddings see straight through the vocabulary to the intent. That's the leap from matching strings to matching meaning.

Here is the mistake that silently ruins more RAG systems than any other, so burn it in now: you must embed your documents and your queries with the exact same model. Two different embedding models produce two incompatible coordinate systems — like one person giving directions in miles and another in kilometres without saying so. The numbers still line up dimension-for-dimension, so nothing crashes; the code runs clean and returns confident nonsense, because "nearby" in one model's space is meaningless in the other's. This is why the lab wraps embedding behind a single embed() function used everywhere: one function, one model, no chance of the query and the corpus drifting apart.

Two smaller ideas make the store click. Once you have vectors, "find the most similar chunk" becomes "find the nearest point," measured by cosine similarity — the cosine of the angle between two vectors, which you'll unpack in Lab 3.3. And a tidy shortcut: if you normalise every vector to length 1 (as the code does with normalize_embeddings=True), that cosine collapses into a plain dot product, so scoring a query against all chunks becomes one fast matrix multiply. The whole VectorStore below is just: embed every chunk once, keep the matrix, and at query time embed the question and multiply — the beating heart that every production vector database (Pinecone, FAISS, pgvector) dresses up with persistence and scale.

An embedding turns text into a vector of numbers where similar meanings sit close together. We store each chunk's vector; at query time we embed the question and find the nearest chunks by cosine similarity.

store.pyimport numpy as np

# --- swap this body for your embeddings provider; keep the signature ---
from sentence_transformers import SentenceTransformer
_model = SentenceTransformer("all-MiniLM-L6-v2")
def embed(texts: list[str]) -> np.ndarray:
    return np.asarray(_model.encode(texts, normalize_embeddings=True))
# -----------------------------------------------------------------------

class VectorStore:
    def __init__(self):
        self.vecs = None; self.chunks = []
    def add(self, chunks):
        self.chunks = chunks
        self.vecs = embed([c["text"] for c in chunks])   # (N, d), normalized
    def search(self, query, k=5):
        q = embed([query])[0]                        # (d,)
        sims = self.vecs @ q                        # cosine (vectors are normalized)
        top = np.argsort(-sims)[:k]
        return [(self.chunks[i], float(sims[i])) for i in top]

Lab 3.3 · Retrieve advanced

You already do this every day without thinking about it. When a friend describes a movie — "the one where the guy relives the same day over and over" — you don't scan your memory word by word for a title match. You match on the shape of the idea, and Groundhog Day surfaces instantly. Retrieval in RAG is that same reflex made mechanical: the question and every stored chunk have already been turned into vectors (Lab 3.2), and now we simply ask "which chunks point in the same direction as the question?"

The measuring stick is cosine similarity, and the geometry is worth holding in your head because it demystifies the whole thing. Picture each vector as an arrow shot out from the origin. Two arrows pointing the same way — a small angle between them — mean similar meaning; arrows at right angles mean unrelated. Cosine similarity is literally the cosine of that angle: 1.0 when they point identically, sliding toward 0 as they diverge. The reason we care about angle and not raw distance is subtle but important — it makes the score about direction (what the text is about) rather than magnitude (how long or emphatic it is), so a short chunk and a long chunk about the same topic still score as close.

And here's the payoff you saw promised back in Lab 3.2: because we normalised every vector to length 1, that cosine reduces to a plain dot product. So scoring the query against the entire corpus is a single matrix-times-vector operation — self.vecs @ q in the code — that hands back one similarity number per chunk in one shot. Then we keep only the best few with top-k (a job for a heap, from the data-structures track). Nothing more mystical is happening: turn text into arrows, keep the arrows that lean most like the question's.

The discipline this section is really trying to teach isn't the math — it's a habit. Retrieval is the part of RAG that fails silently: if the answer-bearing chunk never makes the top-k, no amount of clever prompting downstream can conjure it back. So the professional move, and the single most valuable takeaway in this chapter, is to test retrieval on its own, before you write one line of generation code. Fire a dozen real questions at search() and eyeball whether the right chunk appears near the top. When people say a RAG system "hallucinates," this quiet, upstream check is almost always the thing they skipped.

retrieve.pyfrom chunk import load_corpus
from store import VectorStore

store = VectorStore()
store.add(load_corpus())

for chunk, score in store.search("how do I reset my password?", k=3):
    print(f"[{score:.3f}] {chunk['source']}: {chunk['text'][:80]}...")

[0.612] faq.md: To reset your password, go to Settings > Security and click...
[0.341] faq.md: Two-factor authentication can be enabled from the same...
[0.208] policy.md: Account security is governed by...

Lab 3.4 · Generate a grounded, cited answer advanced

Think about the difference between an open-book exam and a closed-book one. In a closed-book exam you write whatever you can dredge up from memory — and under pressure, people confidently misremember. In an open-book exam the rules flip: you answer from the pages in front of you, and "I couldn't find it" is a legitimate, honest answer. Everything you've built so far — chunking, embedding, retrieval — exists to lay the right pages open on the desk. This final step is where you tell the model, in no uncertain terms, that it is now sitting an open-book exam and cheating from memory is forbidden.

That instruction is the whole ballgame, because an LLM's default behaviour is to answer from its training-data memory. Retrieval alone doesn't stop it — you can hand the model perfect context and it will still happily blend in half-remembered facts from elsewhere unless you explicitly forbid it. So the generation step has two jobs, and they're both in the prompt, not the code. First, augment: paste the retrieved chunks into the prompt as numbered context, so the model has the facts and a way to reference each one. Second, constrain: a system prompt that says "answer strictly from the numbered context, cite the sources you use as [n], and if the answer isn't there, say so."

Those numbered citations are not decoration — they're the feature that makes RAG trustworthy in the real world. A cited answer is auditable: a user (or a compliance reviewer) can click [1], read the source chunk, and confirm the claim is real. That single property is why regulated industries — legal, medical, finance — can use RAG at all, and it's the thing fine-tuning fundamentally cannot give you: a fine-tuned model has digested its facts into weights and can't point back at a source. Grounding plus citation turns a fluent guesser into an accountable assistant.

The most important line to get right is the escape hatch: the explicit permission to say "I don't have that information." New builders leave this out because it feels like admitting weakness, but it's the opposite — it's the difference between a system that fails loudly and safely and one that fails silently and dangerously. Without it, a question your docs don't cover gets a confident, invented answer; with it, you get an honest miss you can act on. Together those two rules — answer only from context and admit when it's missing — are the entire anti-hallucination recipe, and the way to prove they work is to ask something your docs/ folder definitely doesn't contain and watch the model decline. Read the lab with that lens: the code is almost trivial; the SYSTEM prompt is where the safety lives.

Now assemble the retrieved chunks into a prompt that forces the model to answer only from context and cite sources.

rag.pyfrom anthropic import Anthropic
from store import VectorStore
from chunk import load_corpus
client = Anthropic()

store = VectorStore(); store.add(load_corpus())

SYSTEM = (
    "You answer strictly from the numbered context. "
    "Cite the sources you use inline as [n]. "
    "If the context does not contain the answer, say: "
    "'I don't have that information.' Never use outside knowledge."
)

def answer(question, k=4):
    hits = store.search(question, k=k)
    context = "\n\n".join(
        f"[{i}] (source: {c['source']})\n{c['text']}"
        for i, (c, _) in enumerate(hits, 1)
    )
    resp = client.messages.create(
        model="claude-opus-4-8", max_tokens=600, system=SYSTEM,
        messages=[{"role":"user",
                   "content": f"Context:\n{context}\n\nQuestion: {question}"}],
    )
    text = next(b.text for b in resp.content if b.type=="text")
    return text, hits

ans, sources = answer("How do I reset my password?")
print(ans)

Go to Settings > Security and click "Reset password"; a link is emailed
to your registered address [1]. You can also enable 2FA there [1].

rag.pyfrom openai import OpenAI
from store import VectorStore
from chunk import load_corpus
client = OpenAI()

store = VectorStore(); store.add(load_corpus())

SYSTEM = (
    "You answer strictly from the numbered context. "
    "Cite the sources you use inline as [n]. "
    "If the context does not contain the answer, say: "
    "'I don't have that information.' Never use outside knowledge."
)

def answer(question, k=4):
    hits = store.search(question, k=k)
    context = "\n\n".join(
        f"[{i}] (source: {c['source']})\n{c['text']}"
        for i, (c, _) in enumerate(hits, 1)
    )
    resp = client.responses.create(
        model="gpt-5.5", max_output_tokens=600, instructions=SYSTEM,
        input=f"Context:\n{context}\n\nQuestion: {question}",
    )
    text = resp.output_text
    return text, hits

ans, sources = answer("How do I reset my password?")
print(ans)

Go to Settings > Security and click "Reset password"; a link is emailed
to your registered address [1]. You can also enable 2FA there [1].

Lab 3.5 · Hybrid search + re-ranking advanced

Here's a failure that will blindside you in your first real deployment. Your semantic search is working beautifully — paraphrased questions find the right passages, meaning matching feels like magic. Then a user searches for error code ERR_4021, or the product name "Zephyr-7B," or a specific invoice number, and your slick vector search returns... vaguely related fluff. The exact string is sitting right there in a document, and your system can't find it. That's not a bug you introduced; it's the built-in blind spot of pure vector search, and discovering it in production is a rite of passage.

The reason is instructive. Embeddings are brilliant at meaning and deliberately fuzzy about surface form — that's the whole point, it's why they see through paraphrase. But that same fuzziness smears exact tokens: an error code carries almost no semantic meaning to blur toward, so it dissolves into the vector soup. The old-school tool that excels at exact terms is keyword search — specifically BM25, which scores documents by how many of the query's rare words they contain. BM25 is everything embeddings aren't: literal, term-precise, and blind to paraphrase. Vector search and BM25 are mirror-image strengths and weaknesses.

So the professional answer isn't to pick one — it's hybrid search: run both, then fuse their rankings so a chunk that either method loves rises to the top. The elegant part is how you fuse them. You might expect to normalise and weight two different score scales (a fiddly, brittle tuning job). Instead the lab uses Reciprocal Rank Fusion, which throws away the raw scores entirely and looks only at rank position: each list awards a chunk points based on where it placed (1/(k+rank)), points are summed, and everything re-sorts. No weights to tune, no incompatible scales — just "did either method rank you highly?" It's almost embarrassingly simple and remarkably robust.

The final move is re-ranking, and it embodies a pattern worth naming: the funnel. Fusion casts a wide, cheap net — grab the top 10–30 candidates fast. But wide nets drag in near-misses, and you don't want to burn tokens (or dilute the answer) on junk. So you follow the cheap-and-wide fetch with an expensive-and-narrow filter: hand those candidates to a strong model and ask it to pick the genuine best 3–5. Cheap recall first, precise cut second. This funnel — over-fetch with hybrid, then re-rank down — is, in the experience of most practitioners, the single biggest quality jump a RAG system ever makes, which is exactly why it's the last thing this chapter builds before you measure.

Vector search understands meaning but misses exact terms (error codes, product names, acronyms). Keyword search (BM25) nails exact terms but misses paraphrase. Combine both, then re-rank. This is the biggest quality jump you'll make.

Measuring your RAG system expert

Ask yourself an uncomfortable question: how would you actually know if a change made your RAG system better? You tweak the chunk size, ask it a few questions, the answers "seem good," and you ship. Three weeks later a category of questions is quietly failing and you have no idea when it started or which change caused it. This is how most RAG projects rot — not through one big bug, but through a hundred unmeasured tweaks, each judged by vibes. The cure is boring and non-negotiable: you can't improve what you don't measure.

The key insight — and it's the one that separates people who can debug RAG from people who just poke at it — is that RAG has two failure modes that live in two different places, and you must measure them separately. Retrieval can fail (the right chunk never made it into the context), or generation can fail (the right chunk was there and the model still answered badly). Lumping them into a single "was the answer good?" score is like a doctor recording only "patient feels unwell" — true, but useless for treatment. When you split the two, a wrong answer immediately tells you where to look: bad retrieval points you back to chunking and embedding; bad generation despite good context points you at your prompt.

So measure each stage with metrics that answer a plain-English question. On the retrieval side, Recall@k asks "did the chunk that actually contains the answer make it into the top-k?" — the single most important number in RAG, because if it's low nothing downstream can save you. Context precision asks "of what we retrieved, how much was actually relevant?" — low precision means you're padding the prompt with noise. On the generation side, groundedness asks "is every claim in the answer supported by the retrieved context?" (this is your hallucination detector), and answer relevance asks "did it actually address what was asked?" The table below lines these up.

Don't let the word "metrics" make you think you need a heavyweight framework on day one — that's the trap that stops beginners from measuring anything at all. The humble, high-value starting point is a spreadsheet of 15–20 real questions, each paired with the chunk you expect retrieval to surface. After any change, re-run them and hand-check Recall@k. That golden-question set is the closest thing RAG has to a regression test, and it will catch more real problems than any dashboard. In Chapter 5 you'll automate exactly these measurements into a proper harness — but the discipline starts with the spreadsheet.

You can't improve what you don't measure. Track retrieval and generation separately:

In Chapter 5 you'll build an automated harness for exactly these metrics. For now, keep a spreadsheet of 15–20 questions with the chunk you expect to be retrieved, and check Recall@k by hand after any change.

Common pitfalls expert

Every debugging story in RAG starts the same way: with a symptom that lies about its cause. The system "makes things up," or gives "vague" answers, or "can't find" an obvious document — and the instinct is always to reach for the nearest, most visible knob, usually the prompt. You rewrite the system prompt, add "be accurate," maybe swap to a bigger model, and the problem stubbornly stays. The reason is that in RAG the symptom almost never sits where the cause does, and the table below is really a map from one to the other.

The single most important lesson in this whole chapter is hiding in the first row. "It makes things up" feels like a generation problem — the model is clearly the thing producing the wrong words — so that's where people spend their time. But nine times out of ten the answer-bearing chunk was never retrieved, and the model did its job faithfully: it answered from the (wrong or empty) context you gave it. No prompt on Earth can quote a fact that isn't in the context window. This is why we drilled "test retrieval alone, first" so hard — it's the diagnostic that instantly tells you whether you're chasing a retrieval bug or a generation bug, and it saves you from tuning the innocent half of the system.

A few of the other rows are worth internalising before they bite you, because they're the ones that hurt in production rather than in a demo. Leaking other tenants' data is the scary one: if you run semantic search across everyone's documents and only then think about access control, you've already surfaced another customer's private text into a prompt — the filter has to happen before retrieval, not after. Embedding query and docs with different models is the silent killer from Lab 3.2, failing with clean code and confident garbage. And vague answers usually trace to oversized chunks diluting the signal — a chunking problem masquerading as a model problem. Read this table as a habit, not a lookup: when RAG misbehaves, resist the visible knob and trace the symptom back to its true home in the pipeline.

Exercises expert

hits = 0
for q, expected in testset:
    ids = [c["id"] for c in hybrid.search(q, k=5)[:5]]
    hits += expected in ids
print("recall@5:", hits/len(testset))

import numpy as np

def unit(v):
    v = np.asarray(v, dtype=float)
    return v / np.linalg.norm(v)

a = unit([1.0, 2.0, 2.0])
b = unit([2.0, 4.0, 4.0])   # same direction
c = unit([2.0, 0.0, 0.0])
print(round(float(a @ b), 4))   # 1.0 -- identical direction
print(round(float(a @ c), 4))   # < 1.0 -- different direction

import re

def chunk_text(text, source, target_words=20, overlap=True):
    paras = [p.strip() for p in re.split(r"\n\s*\n", text) if p.strip()]
    chunks, buf = [], []
    for para in paras:
        buf.append(para)
        if sum(len(p.split()) for p in buf) >= target_words:
            chunks.append(" ".join(buf))
            buf = buf[-1:] if overlap else []     # carry last para as overlap
    if buf:
        chunks.append(" ".join(buf))
    return [{"text": c, "source": source, "id": f"{source}#{i}"}
            for i, c in enumerate(chunks)]

doc = "Reset your password in Settings then Security.\n\n" \
      "Enable two-factor authentication in the same place.\n\n" \
      "Contact support if you are locked out for good."
out = chunk_text(doc, "faq.md")
print(len(out), "chunks")
for c in out:
    print(c["id"], "::", c["text"][:45])

import numpy as np

def embed(texts):                         # toy stand-in; same signature as real one
    vs = []
    for t in texts:
        v = np.zeros(16)
        for w in t.lower().split():
            v[hash(w) % 16] += 1.0
        n = np.linalg.norm(v) or 1.0
        vs.append(v / n)                  # normalize -> cosine is a dot product
    return np.asarray(vs)

class VectorStore:
    def __init__(self):
        self.vecs = None; self.chunks = []
    def add(self, chunks):
        self.chunks = chunks
        self.vecs = embed([c["text"] for c in chunks])
    def search(self, query, k=3):
        q = embed([query])[0]
        sims = self.vecs @ q
        top = np.argsort(-sims)[:k]
        return [(self.chunks[i], float(sims[i])) for i in top]

docs = [{"text":"reset your password in settings security", "id":"a"},
        {"text":"enable two factor authentication", "id":"b"},
        {"text":"billing invoices and refunds", "id":"c"}]
s = VectorStore(); s.add(docs)
for c, score in s.search("how do I reset my password", k=2):
    print(round(score, 3), c["id"])

def rrf(*lists, k=60):
    scores = {}
    for lst in lists:
        for rank, cid in enumerate(lst):
            scores[cid] = scores.get(cid, 0) + 1 / (k + rank)
    return sorted(scores, key=lambda c: -scores[c])

dense  = ["a", "b", "c", "d"]   # vector search order
sparse = ["c", "a", "e", "f"]   # BM25 order (exact-term match)
fused = rrf(dense, sparse)
print(fused)
# 'a' is top of dense and 2nd in sparse -> wins; 'c' (top sparse, 3rd dense) close behind
assert fused[0] == "a"
assert "e" in fused and "f" in fused    # sparse-only items still included

def recall_at_k(search_fn, testset, k=5):
    hits = 0
    for q, expected in testset:
        ids = [c["id"] for c in search_fn(q, k=k)[:k]]
        hits += expected in ids
    return hits / len(testset)

CORPUS = [
    {"id":"t1#0", "tenant":"t1", "text":"reset password"},
    {"id":"t1#1", "tenant":"t1", "text":"enable 2fa"},
    {"id":"t2#0", "tenant":"t2", "text":"reset password"},   # other tenant, same words
]

def search(query, k=5, tenant=None):
    pool = [c for c in CORPUS if tenant is None or c["tenant"] == tenant]  # FILTER FIRST
    ranked = [c for c in pool if any(w in c["text"] for w in query.split())]
    return ranked[:k]

# tenant isolation: t1 must never see t2's chunk, even on an identical query
ids = [c["id"] for c in search("reset password", tenant="t1")]
print("t1 sees:", ids)
assert all(i.startswith("t1") for i in ids), "tenant leak!"

testset = [("reset password", "t1#0"), ("enable 2fa", "t1#1")]
print("recall@5:", recall_at_k(lambda q, k=5: search(q, k=k, tenant="t1"), testset))

SYSTEM = (
    "You answer strictly from the numbered context. "
    "Cite the sources you use inline as [n]. "
    "If the context does not contain the answer, say: "
    "'I don't have that information.' Never use outside knowledge."
)

def build_context(hits):
    """hits: list of (chunk_dict, score) -> numbered, citable context."""
    return "\n\n".join(
        f"[{i}] (source: {c['source']})\n{c['text']}"
        for i, (c, _) in enumerate(hits, 1)
    )

hits = [({"source":"runbook.md", "text":"Rotate the API key via the admin console."}, 0.71),
        ({"source":"runbook.md", "text":"Keys expire every 90 days."}, 0.55)]
ctx = build_context(hits)
print(ctx)
assert ctx.startswith("[1]") and "[2]" in ctx   # numbered so the model can cite

# Real generation — same assembly, either SDK:
# Claude (needs ANTHROPIC_API_KEY):
# resp = client.messages.create(model="claude-opus-4-8", max_tokens=600, system=SYSTEM,
#   messages=[{"role":"user","content":f"Context:\n{ctx}\n\nQuestion: {q}"}])
# OpenAI (needs OPENAI_API_KEY):
# resp = client.responses.create(model="gpt-5.5", max_output_tokens=600,
#   instructions=SYSTEM, input=f"Context:\n{ctx}\n\nQuestion: {q}")

Knowledge check check yourself