Foundations

1 · Why RAG exists

Picture asking a brilliant new hire about your company on their first morning. They're genuinely smart — they'll answer anything confidently — but they've never seen your wiki, your last incident, or the contract signed yesterday. So when you ask "how do we restart the checkout service?", they don't say "I don't know." They guess, fluently, and their guess sounds exactly as authoritative as a correct answer would. That is a bare LLM. It's not lying — it simply has no way to tell the difference between what it knows and what it's confabulating, and no access to anything that happened after its training was frozen.

Here's the reframe that makes RAG click, and it's worth sitting with: the model's real skill isn't knowing facts — it's reading and reasoning over text you give it. Those are two very different jobs, and we've been leaning on the wrong one. Asking a model to recall your internal runbook from memory is like asking that new hire to recite a document they've never opened. But hand them the document and ask them to read it and answer — that they're excellent at. RAG is simply the plumbing that fetches the right document and puts it on their desk before they answer.

Why does this deserve to be the first idea in the whole track? Because it inverts where you put your effort. The tempting instinct is to make the model "know more" — bigger model, more training. RAG says: leave the model's knowledge alone and get better at finding and delivering the relevant material at question time. That single shift is what lets a small, cheap model answer accurately about a million private documents it never saw in training.

The mistake almost everyone makes first is trusting a confident answer because it reads well. Fluency is not accuracy. A model with no grounding produces beautifully-worded wrong answers, and there's no wobble in its tone to warn you — the sentence about a runbook step that doesn't exist looks identical to the sentence about one that does. Grounding the answer in retrieved text, with citations, is what turns "trust me" into "here's the passage I used, check it yourself." Keep that distinction — fluent versus correct — in mind as you read the three concrete failures below; every row is a different face of the same root problem.

A large language model knows only what it saw during training. That creates three concrete problems the moment you point it at real work:

The insight behind RAG is simple: don't ask the model to remember — give it the material and ask it to read. Instead of baking knowledge into the weights, you keep knowledge in a searchable store and fetch just the relevant slice for each question.

2 · The retrieve → augment → generate loop

Every RAG system you will ever build or debug is the same three-step recipe — and once it's burned into your head, a broken RAG stops being a mystery and becomes a question of "which of the three steps failed?" That's the whole payoff of this section: not just what the loop is, but a diagnostic map you'll reach for every time an answer comes back wrong.

Think of it like a research assistant answering a question for you. First they go to the shelf and pull the handful of books that look relevant (retrieve). Then they lay the open books on the desk next to your question, with a sticky note saying "answer using only these" (augment). Only then do they read those pages and write you an answer, footnoting which book each claim came from (generate). Notice the strict order: the assistant can only write from books they actually pulled. If they grabbed the wrong shelf, no amount of careful reading saves the answer — they'll write a confident report from irrelevant material.

That ordering is why the loop matters so much for reasoning about failures. Each stage can only work with what the previous stage handed it, so errors don't cancel out — they cascade. A missed chunk in retrieve is invisible to augment and generate; they never learn the fact existed. This is the opposite of how beginners debug: they stare at the prompt wording or blame the model, tuning the last two stages, when the damage was almost always done in the first one. The single most useful habit in all of RAG is to print the retrieved chunks and look at them before touching anything else.

One more distinction that trips people up: this loop is orthogonal to how fancy your system is. A 30-line script and a platform serving millions run the exact same three verbs — the platform just makes each verb more sophisticated (better search in retrieve, smarter prompt assembly in augment, a stronger model in generate). Learn the skeleton now and every advanced technique later slots cleanly into one of these three boxes. Trace it once in the diagram below, then read the table — the "gets wrong when…" column is the debugging checklist you'll use for years.

Every RAG system, from a 30-line script to a platform serving millions, is this same loop:

3 · Embeddings — turning meaning into numbers

Here's the problem the whole retrieve step has to solve. A user types "my car won't start" and the answer they need is filed under "vehicle fails to turn over." Keyword search is useless here — those two phrases share zero words. Yet a human instantly sees they mean the same thing. So how do you get a computer to match on meaning rather than on the literal characters? That question is what embeddings exist to answer, and it's the hinge the entire chapter turns on.

The trick is to give every piece of text an address in "meaning-space." Imagine a giant map where texts about similar things sit near each other: all the car-trouble sentences cluster in one neighborhood, all the baking recipes cluster in another, far away. An embedding is just the coordinates of a piece of text on that map — a list of numbers (a vector) that pins it to a spot. "Car won't start" and "vehicle fails to turn over" get placed as next-door neighbors even though they spell nothing alike, because a model trained on mountains of text learned they're used the same way. Real maps have 2 coordinates; these have hundreds or thousands, but the intuition survives: close on the map = close in meaning.

Once meaning is coordinates, "how similar are these two texts?" becomes "how close are these two points?" — a geometry question a computer answers instantly. The specific measure we use is the angle between the two vectors rather than the straight-line distance, and that choice matters: it means we care about the direction a text points in meaning-space, not how long its vector happens to be. A one-word note and a full paragraph on the same topic point the same way, so they count as similar even though one is "bigger." That's exactly what you want.

The mistake to inoculate yourself against right now: the tiny hand-written vectors in the code below are a teaching prop, not how it really works. Nobody types embeddings by hand, and the fake bag-of-words embedder you'll meet in section 5 can't actually match "car" to "vehicle" — it only knows word overlap. Real embeddings come from a trained model that has read enough text to place synonyms near each other. We use fakes so the whole pipeline runs offline in your browser; just hold the thought that the numbers are illustrative and the real magic lives in the model that produces them. With that caveat set, here's the core idea in runnable form.

To “search by meaning,” we need a way to measure how similar two pieces of text are. An embedding is a list of numbers (a vector) that represents the meaning of a piece of text. Texts with similar meaning get vectors that point in similar directions — even if they share no words. “car won't start” and “vehicle fails to turn over” land near each other; “banana bread recipe” lands far away.

We compare two vectors with cosine similarity: the cosine of the angle between them, from -1 (opposite) through 0 (unrelated) to 1 (identical direction). Here is the whole idea in runnable Python — no library needed:

cosine.py# Cosine similarity from scratch — the heart of semantic search.
import math

def dot(a, b):
    return sum(x * y for x, y in zip(a, b))

def norm(a):
    return math.sqrt(sum(x * x for x in a))

def cosine(a, b):
    return dot(a, b) / (norm(a) * norm(b) + 1e-9)   # +tiny to avoid /0

# Pretend these are 4-number embeddings of three sentences.
car_wont_start   = [0.9, 0.1, 0.0, 0.2]
vehicle_no_crank = [0.8, 0.2, 0.1, 0.1]   # same meaning, different words
banana_bread     = [0.0, 0.1, 0.9, 0.8]   # unrelated topic

print('car vs vehicle :', round(cosine(car_wont_start, vehicle_no_crank), 3))
print('car vs banana  :', round(cosine(car_wont_start, banana_bread), 3))

car vs vehicle : 0.965
car vs banana  : 0.229

The two car sentences score high (~0.97) despite sharing no words; the unrelated one scores low. That is semantic search in one function. In a real system you don't hand-write these vectors — an embedding model produces them. The rule to remember: embed the documents once, embed each question at query time, and rank documents by cosine similarity to the question.

4 · Chunking — deciding what can be found

Chunking is the least glamorous decision in RAG and quietly the one that sinks the most projects. Teams pour weeks into picking an embedding model and tuning prompts, then wonder why the right answer never surfaces — when the real culprit was how they sliced the documents in the very first step. If a fact isn't sitting inside a chunk that scores well for the question, it does not exist as far as your system is concerned. No model, no clever prompt, no re-ranker can retrieve a fact that got chopped in half. That's why this earns a full section right after embeddings.

Think of it like scanning a textbook to answer an exam question. If you photocopy one sentence per page, each page is razor-sharp about a single idea — but half of them read "it must be restarted immediately," and you've lost what "it" refers to; the subject was on the previous sentence. If instead you photocopy a whole chapter per page, every page holds the full story, but it also covers ten unrelated topics, so no single page is strongly about your one question — it's a blurry average of everything. The sweet spot is a photocopy of about one coherent paragraph: enough context to stand on its own, focused enough to match one idea sharply. Chunk size is that photocopy setting, and it trades precision against context.

The subtle killer is the boundary fact — a fact that straddles where you happened to cut. "To recover, drain the queue" ends one chunk and "and scale the consumer group" begins the next, so neither chunk holds the complete instruction and both match the question weakly. The fix is overlap: let each chunk repeat the last sentence or two of the one before it, like overlapping roof shingles, so any fact near a seam lands whole in at least one chunk. It costs a little duplication and buys you a lot of recall — a trade almost always worth making.

The mistake to avoid: hunting for a magic universal chunk size. There isn't one — dense legal contracts, chatty support threads, and code all want different settings, and the only honest way to find yours is to try a size, look at what comes back, and adjust. Treat the numbers below (~100–300 words, small overlap) as a sturdy starting point, not a law. With that framing, here's how chunk size shapes what your system can and can't find.

You don't embed a whole 40-page document as one vector — that would blur every topic together. You split it into chunks (say, a few sentences or a paragraph each), embed each chunk, and retrieve chunks. Chunking is quietly one of the most important decisions in RAG, because a fact can only be retrieved if it sits inside a chunk that scores well for the question.

A common, sturdy default: split on paragraphs, then pack ~200 words per chunk with a small overlap (repeat the last sentence or two into the next chunk) so a fact spanning a boundary still lands whole in at least one chunk. Here's a minimal chunker:

chunk.pydef chunk_words(text, size=60, overlap=15):
    """Split text into ~size-word chunks that overlap by `overlap` words.
    Overlap keeps a fact whole even if it straddles a boundary."""
    words = text.split()
    chunks, i = [], 0
    step = size - overlap
    while i < len(words):
        chunks.append(' '.join(words[i:i + size]))
        i += step
    return chunks

doc = ('The checkout service reads orders from the queue. '
       'If the queue backs up, latency climbs and the pod restarts. '
       'To recover, drain the queue and scale the consumer group. '
       'Never restart the database to fix a queue problem.') * 1

for i, c in enumerate(chunk_words(doc, size=12, overlap=3)):
    print(f'chunk {i}: {c}')

chunk 0: The checkout service reads orders from the queue. If the queue backs up,
chunk 1: queue backs up, latency climbs and the pod restarts. To recover, drain
chunk 2: recover, drain the queue and scale the consumer group. Never restart the
chunk 3: restart the database to fix a queue problem.

Notice how the overlap repeats a few words between chunks — so “drain the queue” appears in two chunks and can be found from either side of the boundary.

5 · Build a minimal RAG pipeline

This is the section where RAG stops being a diagram and becomes something you can run. Everything so far — the three-verb loop, embeddings, chunking — has been building toward one small, honest program that ties it all together. And it's genuinely small: a working RAG is closer to forty lines than four hundred. The reason production systems balloon isn't the core idea; it's the scaling, the monitoring, and the edge cases layered on top. Strip those away and what's left is exactly what you're about to read.

The one deliberate cheat is the fake embedder, and it's worth understanding why we cheat here. A real embedding model needs a download or an API key, which would stop this from running in your browser — so we substitute a deterministic bag-of-words stand-in that turns text into a vector by counting words. It has a real limitation you should feel rather than just be told: it matches only on shared words, so it will happily fail the "car" vs "vehicle" test from section 3. That's not a bug to fix here — it's the whole point. Seeing the fake embedder's blind spot with your own eyes is what makes you appreciate what a trained model buys you, and it's Exercise 5 in the ladder below.

As you read the code, watch for the shape rather than the specific tricks, because the shape is what survives contact with production. You index once (embed every document and keep the vectors), then per question you embed the query, score it against the stored vectors, take the top-k, and paste those into a grounded prompt. That skeleton — index → embed query → score → top-k → prompt → generate — is identical whether your store holds three sentences or three billion. When you later swap in a real embedder and a real vector database, you are replacing the insides of these functions, not their arrangement. Read it top to bottom; each numbered comment marks one stage of the loop you already know.

Now assemble the loop end to end. We use a fake but deterministic embedder (a bag-of-words hash) so this runs offline and you can see every moving part; then we show exactly where a real embedder and a real LLM plug in.

mini_rag.pyimport math, re
from collections import Counter

# ---- 1. a FAKE embedder (deterministic bag-of-words). Swap for a real model later. ----
VOCAB = {}
def embed(text):
    counts = Counter(re.findall(r'[a-z]+', text.lower()))
    for w in counts:
        VOCAB.setdefault(w, len(VOCAB))
    vec = [0.0] * len(VOCAB)
    for w, c in counts.items():
        vec[VOCAB[w]] = float(c)
    return vec

def cosine(a, b):
    n = max(len(a), len(b)); a = a + [0.0]*(n-len(a)); b = b + [0.0]*(n-len(b))
    dot = sum(x*y for x, y in zip(a, b))
    na = math.sqrt(sum(x*x for x in a)); nb = math.sqrt(sum(x*x for x in b))
    return dot / (na*nb + 1e-9)

# ---- 2. index the documents (embed once, keep the vectors) ----
docs = [
    'To restart the checkout service, drain the queue then scale the consumer group.',
    'The billing service stores invoices in Postgres and retries failed charges.',
    'Never restart the database to fix a queue backlog; it makes recovery slower.',
]
index = [(d, embed(d)) for d in docs]

# ---- 3. retrieve: top-k chunks by cosine similarity to the question ----
def retrieve(question, k=2):
    q = embed(question)
    scored = sorted(index, key=lambda di: cosine(q, di[1]), reverse=True)
    return [d for d, _ in scored[:k]]

# ---- 4. augment: build a grounded prompt ----
def build_prompt(question, chunks):
    context = '\n'.join(f'[{i+1}] {c}' for i, c in enumerate(chunks))
    return (f'Answer ONLY from the context. Cite sources like [1].\n\n'
            f'Context:\n{context}\n\nQuestion: {question}\nAnswer:')

q = 'how do I restart checkout?'
chunks = retrieve(q)
print('RETRIEVED:')
for c in chunks: print('  -', c)
print('\nPROMPT SENT TO THE LLM:\n' + build_prompt(q, chunks))

RETRIEVED:
  - To restart the checkout service, drain the queue then scale the consumer group.
  - Never restart the database to fix a queue backlog; it makes recovery slower.

PROMPT SENT TO THE LLM:
Answer ONLY from the context. Cite sources like [1].

Context:
[1] To restart the checkout service, drain the queue then scale the consumer group.
[2] Never restart the database to fix a queue backlog; it makes recovery slower.

Question: how do I restart checkout?
Answer:

That is a complete RAG retrieve+augment. The right two chunks came back and the prompt grounds the model in them. The last step — generate — sends that prompt to an LLM:

generate.py# The generation step. Complete and correct; needs YOUR API key to actually run.
# pip install anthropic ; export ANTHROPIC_API_KEY=sk-...
from anthropic import Anthropic          # runs in your own environment, not the sandbox

client = Anthropic()
def generate(prompt):
    msg = client.messages.create(
        model='claude-opus-4-8',           # any current model id
        max_tokens=300,
        messages=[{'role': 'user', 'content': prompt}],
    )
    return msg.content[0].text

# answer = generate(build_prompt(q, retrieve(q)))
# print(answer)  ->  'Drain the queue, then scale the consumer group. [1]'

generate.py# The generation step. Complete and correct; needs YOUR API key to actually run.
# pip install openai ; export OPENAI_API_KEY=sk-...
from openai import OpenAI                 # runs in your own environment, not the sandbox

client = OpenAI()
def generate(prompt):
    resp = client.responses.create(
        model='gpt-5.5',                   # any current model id
        max_output_tokens=300,
        input=prompt,
    )
    return resp.output_text

# answer = generate(build_prompt(q, retrieve(q)))
# print(answer)  ->  'Drain the queue, then scale the consumer group. [1]'

6 · RAG vs fine-tuning vs a bigger context window

This is the question that separates people who use these tools from people who just heard of them. Pick the wrong approach and you'll spend weeks and a fortune solving a problem the right tool handles in an afternoon — retraining a model every time a document changes, or paying to paste a 300-page manual into every single API call. Getting this choice right is one of the highest-leverage judgments in the whole field, which is why we close the chapter on it.

The cleanest way to hold the three apart is an analogy: imagine getting an employee ready for a task. RAG is handing them a reference binder to consult while they work — the binder can be updated any morning, and they cite the page they used. Fine-tuning is sending them on a training course that reshapes how they habitually work — it changes their style and skills, but you can't send them back to school just to teach them one new fact. A bigger context window is stapling the whole document to the task sheet every single time — fine for a one-page memo, absurd for an encyclopedia. Notice these aren't rivals so much as answers to different questions: what should they know, how should they behave, and how much can they hold at once.

That framing dissolves the beginner confusion neatly. The core rule is RAG for knowledge, fine-tuning for behavior. Facts that change, are private, or are large belong in a retrieval store you can update without touching the model — that's RAG. A consistent voice, an output format, a specialized skill the model should apply every time — that's fine-tuning. And a single small document you always need in full? Just paste it in; don't build a pipeline for a problem you can solve with copy-paste.

The most common mistake here is treating this as a fork in the road. It isn't — the three compose, and mature systems routinely use two or all three together: fine-tune a model to always answer in your JSON schema (behavior), feed it retrieved facts from RAG (knowledge), and paste in a short static policy header every call (context). The skill isn't picking the "winner"; it's matching each kind of requirement to the tool built for it. The table lays out where each shines and, just as importantly, where each quietly falls down.

Newcomers often ask “why not just fine-tune the model, or paste the whole document in?” Each has a place: