RAG at scale

1 · The scaling wall

Here's the moment that catches almost every RAG team off guard. Your prototype is a triumph: 5,000 chunks in a Python list, retrieval feels instant, demos land, everyone's happy. Then you point it at the real corpus — a few million documents — and the exact same code that answered in 20 milliseconds now takes several seconds per query, and the box falls over the moment more than a handful of users show up at once. Nothing about the logic changed. The corpus did. This section is about seeing that wall coming before you hit it.

The reason is the one idea from complexity theory that matters most in production: how does the work grow as the input grows? Brute-force search compares your query against every single stored vector — so if you double the corpus, you double the work on every query. Picture searching a library for a book by pulling every volume off the shelf and reading its spine one at a time. With a hundred books, fine. With ten million, you'll die of old age before you find it — and you have to start over from scratch for the next person who asks. That "start over every time, touch everything" pattern is exactly what O(N·d) per query means: N vectors, each of dimension d, compared afresh for each incoming query.

The mistake that hides the problem is testing only at small scale and trusting the clock. On a laptop with a few thousand vectors, brute-force is genuinely the right choice — it's simple, it's exact, and it's fast enough that any cleverness would be premature. So it passes every test you write, ships, and detonates in production where N is three orders of magnitude larger. The honest way to see the danger early is not to time it (clocks are noisy, machine-dependent, and lie about what will happen at 100× the data) but to count the operations and watch how that count scales. If the count grows one-for-one with the corpus, you know — with certainty, on day one — that the design has a ceiling. That is what the lab below makes visible: a counter that ticks once per vector comparison, so you can read the growth curve straight off the numbers.

Every retriever so far scored the query against every stored vector and kept the best. That is brute-force (or “flat”) search, and it is exactly right at small scale: it is simple, it is exact (it truly finds the nearest neighbours), and for a few thousand chunks it is instant. The problem is the cost curve. Comparing the query against N vectors of dimension d costs O(N·d) per query: double the corpus, double the work, on every single query. Fine at thousands; a wall at millions.

Rather than time it (clocks are noisy and non-deterministic), we count operations. Each full vector comparison bumps a counter, so the growth is exact and reproducible. Watch comparisons track N one-for-one:

scaling_wall.py# Brute-force search cost, measured with an OPERATION COUNTER (not a clock).
# Every vector comparison bumps the counter, so cost is deterministic and reproducible.

def make_index(n, dims=8):
    """Deterministic fake vectors: value depends only on (row, col)."""
    return [[((r * 7 + c * 13) % 17) / 17.0 for c in range(dims)] for r in range(n)]

comparisons = 0
def cosine(a, b):
    global comparisons
    comparisons += 1                      # one full vector compared = one op
    dot = sum(x * y for x, y in zip(a, b))
    na = sum(x * x for x in a) ** 0.5
    nb = sum(x * x for x in b) ** 0.5
    return dot / (na * nb + 1e-9)

def brute_force(query, index):
    return max(range(len(index)), key=lambda i: cosine(query, index[i]))

query = [0.5] * 8
for n in (1000, 2000, 4000, 8000):
    index = make_index(n)
    comparisons = 0
    brute_force(query, index)
    print(f'N={n:>5}  comparisons={comparisons:>5}  (== N, grows linearly)')

N= 1000  comparisons= 1000  (== N, grows linearly)
N= 2000  comparisons= 2000  (== N, grows linearly)
N= 4000  comparisons= 4000  (== N, grows linearly)
N= 8000  comparisons= 8000  (== N, grows linearly)

The comparison count equals N every time — that is the O(N) part, and each comparison itself costs O(d). At 10k vectors nobody notices; at 10 million, a single query touches 10 million vectors, and if you serve hundreds of queries per second the machine melts. You cannot brute-force your way to scale — you need a data structure that finds near vectors without scanning them all.

2 · Approximate nearest neighbours (ANN)

The way past the wall is a trade almost nobody makes willingly at first — and then can't live without. You give up the guarantee of finding the exact nearest neighbours, and in return you touch a tiny fraction of the corpus. It sounds reckless: aren't wrong answers bad? But sit with the arithmetic for a second. Would you rather find the perfect top-5 by inspecting all ten million vectors on every query (and melt the machine), or find the almost-perfect top-5 — say, missing one true result out of a hundred queries — by inspecting a few thousand? At scale, "provably close, on a budget" beats "perfect, but impossible."

The mental model that makes this click is how you actually find a restaurant in a strange city. You do not compute the distance from where you're standing to every restaurant in the metro area and pick the minimum — that's brute force. You glance at the neighbourhood you're in, notice which few blocks look promising, and walk toward those, ignoring the other 99% of the map entirely. You might occasionally miss a slightly-closer spot one block over. You almost never care. Approximate nearest neighbour (ANN) search is that instinct written as a data structure: it organizes the vectors so a query can head straight for the promising region and skip the rest.

The dial that controls this is recall — the fraction of the true nearest neighbours your approximate search actually returns. Turn the dial toward high recall and you inspect more vectors (safer, slower, dearer); turn it down and you inspect fewer (cheaper, faster, more misses). In practice you can hold recall very high — 0.95 to 0.99 is a common operating range (illustrative) — while cutting the vectors inspected by orders of magnitude, because near-duplicate vectors cluster together and you rarely need to leave the promising region. The common mistake is to treat "approximate" as a synonym for "unreliable" and either avoid ANN out of fear or crank recall to 1.0 (which is just brute force wearing a costume). Recall is a knob you set deliberately and measure against a brute-force ground truth — not a defect you tolerate. Two families of index dominate, and it's worth carrying both as pictures: a graph you walk (HNSW) and clusters you probe (IVF).

The escape is to stop insisting on the exact nearest neighbours. Approximate nearest neighbour search accepts that it will occasionally miss a true top result, in exchange for touching a tiny fraction of the vectors. The knob is recall — the fraction of true neighbours the approximate search still finds — and in practice you can hold recall very high (say 0.95–0.99, illustrative) while inspecting orders of magnitude fewer vectors. That trade is the entire reason production RAG can exist at scale. Two families dominate, and it is worth holding both as mental pictures:

We will not build a real HNSW graph — that is a substantial system and the whole point of using a library is that someone already did it correctly. But the IVF idea is small enough to simulate, and it makes the recall trade concrete. Below we cluster 600 vectors into 4 buckets, then search only the nprobe nearest buckets, counting how many vectors we inspect versus brute force:

ivf_sim.py# IVF idea: cluster vectors, then search ONLY the nearest cluster(s).
# We count how many vectors each strategy inspects. Fully deterministic.

def make_vectors(n, dims=4):
    # spread points across the space so all clusters get populated
    return [[((r * 2 + c * 7) % 13) / 13.0 for c in range(dims)] for r in range(n)]

def dist2(a, b):
    return sum((x - y) ** 2 for x, y in zip(a, b))

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    na = sum(x * x for x in a) ** 0.5
    nb = sum(x * x for x in b) ** 0.5
    return dot / (na * nb + 1e-9)

# ---- build: assign each vector to its nearest of C fixed centroids (IVF 'inverted lists') ----
vectors = make_vectors(600, dims=4)
centroids = [[0.15, 0.25, 0.75, 0.35], [0.75, 0.65, 0.2, 0.25],
             [0.25, 0.85, 0.35, 0.7], [0.85, 0.2, 0.55, 0.8]]
lists = {c: [] for c in range(len(centroids))}
for i, v in enumerate(vectors):
    nearest_c = min(range(len(centroids)), key=lambda c: dist2(v, centroids[c]))
    lists[nearest_c].append(i)

query = [0.2, 0.3, 0.7, 0.4]

# ---- brute force: inspect every vector, exact answer ----
bf_inspected = len(vectors)
bf_best = max(range(len(vectors)), key=lambda i: cosine(query, vectors[i]))

# ---- IVF: probe only the nprobe nearest clusters ----
def ivf_search(query, nprobe):
    order = sorted(range(len(centroids)), key=lambda c: dist2(query, centroids[c]))
    probed = order[:nprobe]
    candidates = [i for c in probed for i in lists[c]]
    inspected = len(candidates)
    best = max(candidates, key=lambda i: cosine(query, vectors[i]))
    return best, inspected

for c in range(len(centroids)):
    print(f'cluster {c}: {len(lists[c])} vectors')
print(f'\nbrute force : inspected {bf_inspected}, best index {bf_best}')
for nprobe in (1, 2, 3):
    best, inspected = ivf_search(query, nprobe)
    hit = 'HIT ' if best == bf_best else 'MISS'
    print(f'IVF nprobe={nprobe}: inspected {inspected:>3}, best index {best}  ({hit} vs brute force)')

cluster 0: 184 vectors
cluster 1: 92 vectors
cluster 2: 232 vectors
cluster 3: 92 vectors

brute force : inspected 600, best index 12
IVF nprobe=1: inspected 184, best index 11  (MISS vs brute force)
IVF nprobe=2: inspected 416, best index 11  (MISS vs brute force)
IVF nprobe=3: inspected 508, best index 12  (HIT  vs brute force)

This is the recall trade in one run. At nprobe=1 we inspected only 184 of 600 vectors — a big saving — but the true best (index 12) sat in a cluster we didn't probe, so we returned a near-but-wrong neighbour (index 11): a recall miss. Probing more clusters inspects more vectors and, by nprobe=3, recovers the exact answer. That is the entire ANN dial: more probing → higher recall → more work. A real IVF over millions of vectors would use many more clusters so even a small nprobe skips the vast majority of the corpus, and HNSW reaches similar recall by graph-walking instead of clustering.

3 · Vector databases — library vs managed

An ANN index is an engine, not a car. It finds near vectors fast — and that's all it does. The trap teams fall into is mistaking the engine for the whole vehicle: they wire a raw ANN library into a prototype, it screams along, and then reality arrives with a list of questions the index never promised to answer. What happens when the process restarts — are the vectors still there, or did they evaporate with the RAM? How do you return "only documents this user is allowed to see"? How do you add a million new vectors while queries are still flowing? Who gets paged at 3am when a node dies? None of those are search problems. They're systems problems, and answering them is precisely what a vector database is for.

Think of it like the difference between an SQLite file and a full Postgres deployment. Both store rows and run queries; the gap between them is everything around the query — durability, concurrent writers, backups, access control, replication, operations. A bare ANN library is the SQLite of vector search: perfect when the data fits in one process and you're happy owning every operational detail yourself. A managed or server vector database is the Postgres: it hands you persistence, metadata filtering, sharding, and replication as built-in features, at the cost of running (or paying for) more machinery.

The honest framing — and the reason this section is a table, not a verdict — is that neither is "better." The right choice falls out of your constraints: corpus size, whether vectors must survive a restart, whether you need to filter by metadata in the same query, how much QPS you serve, and how much operational plumbing your team wants to own. A common and expensive mistake is reaching for a heavyweight managed system while still prototyping (paying in complexity and cost for scale you don't have), or the opposite — riding a raw in-memory library straight into production and discovering on the first restart that "durability" was your job all along. Read the comparison below as a decision aid, not a leaderboard: match the shape to your situation, and remember the query interface barely changes either way.

An ANN index is just the search structure. A real system needs much more around it, and that is what a vector database provides. You have two broad ways to get there, and the honest framing is a trade-off, not a winner — the right choice depends on your team, scale, and constraints:

Whichever you pick, the query interface is the one you already know from 3.1–3.3: embed the query, ask the index for the top-k, optionally with a metadata where filter and hybrid keyword fusion. The database changes where the vectors live and who operates it, not the retrieve → augment → generate shape.

4 · Sharding & distribution

Every machine has a ceiling, and one day your index hits it. A single node has a fixed amount of RAM, and an ANN index has to live in memory to be fast — so past some corpus size, the index simply won't fit. Or the vectors fit fine but the query rate climbs until one machine can't answer fast enough. Either way you've run out of "one bigger box," and the answer is to spread the work across several boxes. That's sharding: cut the corpus into slices and give each node one slice to hold and search.

The picture is a library reference desk that got too busy for one librarian. Instead of one person who knows the whole collection, you split the collection by section — one librarian owns Art, another History, another Science — and when a question comes in you ask all of them at once, then combine their best answers into one. That's the scatter-gather pattern: scatter the query to every shard, each searches its own slice and returns its local top-k, and a coordinator gathers those partial results and merges them into the true global top-k. Because every shard is smaller, each search is faster, and because they run in parallel, total throughput climbs with the number of nodes.

Two subtleties are worth pinning down, because they're where sharding bites. First, correctness depends on each shard returning enough candidates: if you want the global top-10 you can't have each shard return only its top-3, or a shard holding four of the true winners will drop one on the floor before the merge ever sees it. Second — and this is the mistake that costs teams weeks — sharding is not free scaling, it's a tax you pay in complexity. You now wait for the slowest shard on every query (tail latency), you need a coordinator to build and operate, and shards drift uneven over time and need rebalancing. The professional instinct is therefore counterintuitive: don't shard until you're forced to. A single well-provisioned node with a good ANN index and a read replica handles far more than beginners assume. Reach for the distributed design only when one node genuinely can't hold the index or keep up with traffic — and separate replication (copies of a shard for throughput and surviving a dead node) from sharding (splitting the data itself) in your head, because they solve different problems.

A single node holds only so many vectors in memory and serves only so many QPS. When one machine isn't enough, you shard: split the index across several nodes so each holds a slice of the corpus. A query then fans out to all shards, each returns its local top-k, and a coordinator merges them — the classic scatter-gather pattern.

5 · Ingestion at scale & freshness

The demo lied to you in one quiet way: it treated the corpus as a photograph. You embedded a fixed pile of documents once, built the index, and asked questions of a frozen snapshot. Real corpora are not photographs — they're rivers. Documents arrive, get edited, get deleted, all day, every day. A support knowledge base changes with every new ticket; a news index is stale within minutes; a policy wiki shifts every time legal touches it. So the real question isn't "how do I build the index" — it's "how do I keep the index honest while the world underneath it keeps moving." That property has a name: freshness, the lag between a document changing and its new version becoming searchable.

There are two rhythms for feeding the pipeline, and the analogy is the post office. Batch ingestion is the daily mail run: you gather up everything that changed, and once a night (or once an hour) you re-embed and rebuild in one efficient sweep. It's simple and cheap per document, but between runs your answers describe yesterday's world. Streaming (incremental) ingestion is the courier: each document is embedded and upserted the moment it arrives, so answers are near-real-time — at the cost of more moving parts and per-item overhead, and a hard requirement that upserts be idempotent so a retry doesn't create a duplicate vector. Neither is correct in the abstract; you pick by how stale your users can tolerate answers being.

Three realities ambush teams specifically at scale, and they're worth naming so you recognize them before they cost you. The first and nastiest: swapping the embedding model is a full migration, not a config change. Vectors from two different models live in two different, incomparable coordinate systems — a query embedded with the new model scored against documents embedded with the old one produces confident garbage. You cannot mix them; you must re-embed the entire corpus (potentially millions of expensive calls), build a fresh versioned index, verify it, and cut over atomically. The second: deletes are sneaky. Remove a document from your source and its vector often lingers in the index, cheerfully retrievable — so you keep serving content you thought you'd taken down. Treat every update as delete-then-insert, propagate deletes promptly, and know that some indexes only tombstone (mark-as-dead) and reclaim the space on a later rebuild. The third is freshness lag itself — the delay you must design for, choosing batch for stable corpora and streaming for fast-moving ones. The mistake underneath all three is assuming ingestion is a one-time setup step; at scale it's a living system with its own failure modes.

The index isn't built once and frozen — documents are added, changed, and deleted continuously, and every change means embedding text and updating the index. How you feed that pipeline decides how fresh your answers are:

Three ingestion realities bite specifically at scale:

6 · Caching — embeddings & query results

Here's a free lunch that's actually free — as long as you don't poison it. Real user traffic is nothing like the uniform random queries you test with. It's wildly skewed: a small handful of questions ("reset password," "refund policy") and a small handful of documents account for a huge share of every request that hits your system. Which means you are, right now, paying to compute the exact same answer over and over. Caching is simply refusing to do that: remember the expensive result the first time, hand back the copy every time after. On skewed traffic the payoff is enormous, because the popular queries — the ones you'd otherwise recompute thousands of times a day — get computed once.

Two layers earn their keep, and they cut work at different points in the pipeline. An embedding cache maps text to its vector, so a piece of text is never sent through the embedding model twice — this saves the single priciest step in both ingestion and query. A query-result cache maps a whole query to its retrieved chunks (or even the final generated answer), so a repeated question skips retrieval and generation entirely — the biggest latency-and-cost win available, because it short-circuits the whole expensive chain. The lab below makes both concrete with a counter on a pretend-expensive embed(), so you can watch repeats cost nothing.

But there's a reason "cache invalidation" is the industry's favourite hard-problem joke, and at scale it's the part that will actually hurt you. A cache is a promise that the stored answer is still correct — and that promise quietly expires the moment the world changes underneath it. A query-result cache is only valid until the indexed documents change: edit the underlying doc and the cache will now serve a confidently wrong, out-of-date answer to your most popular queries — the worst possible ones to get wrong. An embedding cache is safe only as long as the same model produced it; the instant you re-embed with a new model (section 5), every cached vector is stale and must be dropped. The mistake that bites everyone is keying a cache on the text alone. The fix is to key it on text plus model version and plus index version, and to expire result entries on a TTL or bust them when their documents update — so that "the world changed" automatically means "the stale entry no longer matches." A cache that can't tell it's out of date isn't a speed-up; it's a bug that scales.

Real traffic is skewed: a small set of popular queries and documents accounts for a large share of requests. Caching exploits that. Two layers pay off, and both cut the expensive embedding calls:

Here is both ideas in one runnable simulation. We count how many times the pretend-expensive embed() actually runs, and how many requests are served straight from the result cache:

caching.py# Caching: skip re-embedding text we've seen, and skip re-retrieving repeated queries.
# We count how many times the (expensive) embed function actually runs. Deterministic.

embed_calls = 0
_embed_cache = {}

def embed(text):
    """Pretend-expensive embedder. The cache makes repeats free."""
    global embed_calls
    if text in _embed_cache:
        return _embed_cache[text]          # cache HIT: no work
    embed_calls += 1                       # cache MISS: pay the cost once
    vec = [(sum(ord(ch) for ch in text) + i) % 97 / 97.0 for i in range(4)]
    _embed_cache[text] = vec
    return vec

# A stream of queries where some repeat (real traffic is very skewed toward popular ones).
queries = ['reset password', 'refund policy', 'reset password',
           'reset password', 'refund policy', 'shipping time']

query_cache = {}
query_cache_hits = 0
for q in queries:
    if q in query_cache:
        query_cache_hits += 1              # served straight from the result cache
        continue
    query_cache[q] = embed(q)              # miss: embed + (would retrieve) once

print(f'requests            : {len(queries)}')
print(f'distinct queries    : {len(set(queries))}')
print(f'embed() calls made  : {embed_calls}   (== distinct queries, repeats were free)')
print(f'query-cache hits     : {query_cache_hits}')
print(f'work avoided         : {query_cache_hits}/{len(queries)} requests served from cache')

requests            : 6
distinct queries    : 3
embed() calls made  : 3   (== distinct queries, repeats were free)
query-cache hits     : 3
work avoided         : 3/6 requests served from cache

Six requests, but only three distinct queries — so embed() ran three times and half the requests were served from the result cache with zero retrieval work. On skewed real traffic the hit rate is often far higher. The catch is invalidation: a cached result is only correct until the underlying documents change.

7 · Thinking in a cost & latency budget

At small scale, "make it work" is the whole job. At scale, every design choice is a line item on a bill — measured in both milliseconds a user waits and cents you pay per query. The engineer who thinks in features ships something that works and quietly costs a fortune or feels sluggish; the engineer who thinks in budgets knows, per query, how long the user will wait and how much money left the account — and, crucially, which stage spent it. This section is about that shift in mindset: from "does it return an answer" to "where does the time and money actually go."

The mental model is a household budget, and it comes with the same discipline. You have a fixed allowance — say, "answer in under a second" and "under a fraction of a cent per query" — and four categories competing for it: embedding the query, searching the vectors, optionally re-ranking the candidates, and generating the answer with the LLM. Just like money, latency and cost aren't spread evenly across those categories; a few stages dominate and the rest are rounding error. The whole art is spending your optimization effort where the money and milliseconds actually are.

And this is where good engineers waste the most time, so it's the mistake worth branding on your memory: optimizing the stage that feels clever instead of the stage that's expensive. Vector search is the fun part — HNSW parameters, recall dials, sharding diagrams — so it's tempting to spend a week shaving milliseconds off retrieval. But in a great many RAG systems the LLM generation call dwarfs everything else in both latency and cost, sometimes by an order of magnitude. Shaving 5ms off a search while a bloated 2-second generation call sits untouched is optimizing the rounding error and ignoring the bill. The rule that saves you: profile a real query end to end first, find the genuine bottleneck, and spend your effort there. Every figure in the table below is an illustrative placeholder to show the shape of the reasoning — the actual numbers come from measuring your system, because your models, data, and infrastructure decide where the money really goes.

At scale, every design choice is a spend. It helps to think of a per-query budget — a latency budget (how long a user waits) and a cost budget (money per query) — and to know which stage spends what. The numbers below are illustrative placeholders to show the shape of the reasoning, not measurements — your real figures depend on your models, data, and infrastructure, so measure them: