Vector Databases, Embeddings & Frameworks

The retrieval problem motivation

Your RAG demo dazzled everyone on 200 documents. Then someone loaded the real corpus — 8 million support tickets — and now every question takes nine seconds, half the answers cite the wrong customer, and the bill for re-scanning everything on each query keeps climbing. Nothing about your logic broke. What broke is that "compare the question to every document" is a promise you can only keep at small scale. This whole page is about keeping that promise when the pile gets huge.

Here's the shape of the problem in one sentence: for any incoming query, you must answer "which of my N documents are most relevant?" — and you must answer it fast, over possibly millions of items, ideally while respecting hard filters ("only this customer's docs, only PDFs from 2026"). Think of it like a library. On a tiny personal shelf you just read every spine. In a city library you cannot — you need a catalogue, an index, and a way to narrow to the right room before you start scanning titles. Retrieval at scale is that catalogue, built for meaning instead of alphabetical order.

The common mistake is treating this as a single knob ("get a vector database and it'll be fast"). It isn't — it's a stack of decisions, each covered below: how you turn text into vectors (embeddings), how you find near ones without scanning them all (ANN indexes), what infrastructure holds and filters them (vector DBs), how you combine meaning-search with keyword-search (hybrid + re-ranking), and whether to wire it yourself or lean on a framework. The Ch 3 RAG build does the small-scale version by hand so the pieces are familiar; this page is how each one scales — and, just as important, when you actually need the heavier machinery versus when a plain array scan is still the right, boring answer.

1 · Embeddings — meaning as a vector intermediate

Computers can't compare meanings — they can only compare numbers. So how does a machine know that "how do I cancel my subscription?" and "steps to end my plan" are asking the same thing, when they share almost no words? A keyword search sees two unrelated strings. The trick that makes modern retrieval work is to first translate meaning itself into numbers, so that "close in meaning" becomes "close in space" — something arithmetic can actually measure.

That translation is an embedding. Picture a giant map where every possible sentence gets a pin. Sentences about refunds cluster in one region; sentences about passwords cluster in another; the two clusters sit far apart. An embedding model is the cartographer: you hand it text and it hands back that pin's coordinates — a fixed-length list of floats (typically 384 to 3072 of them). Two texts that mean similar things get pinned near each other, even if they used entirely different words. Retrieval then becomes almost embarrassingly simple: embed the query to get its pin, and find the document pins nearest to it.

"Nearest" needs a definition, and here's the one that matters: we measure the angle between two vectors (cosine similarity, from A4), not the straight-line distance. Why the angle? Because it captures direction — "what is this about" — while ignoring magnitude, which for text often just reflects length or word-count quirks you don't care about. Two vectors pointing the same way score near 1 (same meaning); at right angles, near 0 (unrelated); opposite, near -1.

The mistake that quietly poisons a retrieval system: embedding your queries with one model and your documents with another, or swapping models without re-embedding the whole corpus. The coordinates only mean anything relative to the same map — mix two cartographers and every distance you compute is garbage, yet nothing crashes, so the bug hides as "retrieval just feels a bit off." Keep query and document embeddings from the identical model, and normalize them so cosine reduces to a plain dot product.

python# sentence-transformers runs a small embedding model locally:
#   from sentence_transformers import SentenceTransformer
#   model = SentenceTransformer("all-MiniLM-L6-v2")   # 384-dim
#   vecs = model.encode(["how to scale a deployment", "kubectl scale docs"])

import numpy as np
def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

# stand-in vectors; real ones come from the model above
q  = np.random.rand(384).astype(np.float32)
d1 = np.random.rand(384).astype(np.float32)
print(round(cosine(q, d1), 3))    # similarity in [-1, 1]; higher = more related

2 · Exact search — and why it stops scaling intermediate

Before you reach for anything clever, know that the dumb way is often the right way. The simplest possible search is: compare the query to every single document, keep the best few. No index, no approximation, no infrastructure. It is trivially correct, it never misses a true match, and — this is the part people skip past — for most projects it is entirely good enough. The instinct to grab a vector database on day one is the most common over-engineering mistake in this whole topic.

This brute-force approach is called exact nearest-neighbor search, and it's the exact A4 pattern: score the query against every vector, take the top-k. Think of it like finding the closest star to Earth by measuring the distance to all of them and sorting — flawless, and fine when there are a few thousand stars. The reason it eventually breaks is pure arithmetic: it costs O(N·d) per query, meaning the work grows in lockstep with both the number of documents N and the vector dimension d. Double the corpus, double the time. That's a straight line, and lines look harmless until N gets large.

Here's where the line bites. At ten thousand documents, an exact scan is a blink. At ten million 768-dimension vectors, a single query touches billions of floating-point numbers — that's gigabytes read from memory on every query, and now you're at hundreds of milliseconds per lookup and a compute bill that scales with your traffic. The failure is gradual and sneaky: it works in the demo, works in staging, and only melts once real data and real query volume arrive together. The honest rule of thumb — good to roughly a hundred thousand vectors you can live with exact search; past that, the sections below (ANN indexes, then vector databases) exist precisely to break that linear wall by not looking at every vector.

pythonimport numpy as np

def exact_topk(corpus, query, k=5):     # corpus: (N, d) normalized
    scores = corpus @ query                # (N,) — O(N*d)
    idx = np.argpartition(-scores, k)[:k]  # top-k, O(N) (D4 heap idea)
    return idx[np.argsort(-scores[idx])]

# Perfect recall, but O(N*d) every query -> fine to ~100k, painful at 10M+

3 · Approximate nearest neighbor (ANN) — trade recall for speed expert advanced

Here's a bargain that sounds too good, and mostly isn't. What if you gave up being perfectly right — accepting that once in a while you'd miss the true best match — and in exchange your search got a hundred or a thousand times faster? For retrieval, that's an easy trade, because the query embedding is already a fuzzy approximation of intent, the top-10 results are all "pretty relevant" anyway, and a language model reading them rarely cares whether the genuinely-8th-best chunk sneaks in at rank 9. You were never chasing a provably optimal answer — so paying for one is waste.

That bargain is approximate nearest neighbor (ANN) search: indexes that find almost the best matches in sub-linear time — they stop scanning every vector. The key concept to hold onto is recall: the fraction of the true nearest neighbors an approximate search actually recovers. Recall is a dial, not a fixed cost — you can turn it toward "more accurate but slower" or "faster but sloppier," and the entire craft of running ANN is choosing where that dial sits and measuring what you gave up. The cardinal sin here is adopting an ANN config without ever measuring its recall against exact search; "it's fast now" tells you nothing about whether it quietly started returning junk.

The beautiful part, for a data-structures course, is that ANN indexes aren't magic — they're the tree and graph structures you already met, pointed at similarity. The two dominant families below both work by the same instinct a human uses: don't check everything, narrow to a promising neighborhood first. HNSW does it with a navigable graph you hop across; IVF does it by pre-clustering vectors and only searching the nearest clusters; product quantization shrinks each vector so more of them fit in memory. Each row of the table is a different answer to "how do I avoid looking at all N vectors?"

python# pip install faiss-cpu
#   import faiss, numpy as np
#   d = 384
#   index = faiss.IndexHNSWFlat(d, 32)      # HNSW, 32 links/node
#   index.add(corpus)                        # corpus: (N, d) float32
#   scores, ids = index.search(query[None], k=5)   # ANN top-5 — sub-linear
#
# For exact search as a baseline: faiss.IndexFlatIP(d)  (inner product)
# Tune recall vs speed via HNSW efSearch / IVF nprobe.
print("FAISS: local, in-process ANN — great for embedding into an app")

4 · Vector databases advanced

An ANN index alone is like an engine with no car around it. FAISS will find you the nearest vectors blazingly fast — but where do those vectors live when your process restarts? How do you delete a document a customer asked you to forget? How do you say "search only this tenant's data"? How do you add a thousand new docs at 2am without rebuilding everything? A raw index answers none of those. In real systems, "find nearest vectors" is maybe 20% of the job; the other 80% is the plumbing around it.

A vector database is that plumbing, wrapped around an ANN index so you get a running service instead of a bare data structure. It adds the things a production system can't live without: persistence (vectors survive a restart), metadata filtering (attach tags like tenant/date/source and restrict search to a scope), CRUD and upserts (add, update, delete individual records without a full rebuild), and scaling and sharding (spread the index across machines when it outgrows one). The clean mental split: FAISS is a library — you call it in-process and you own storage, lifecycle, and ops; a vector DB is a service — it owns all of that for you, at the cost of running (or paying for) another system.

The mistake to sidestep is picking the heaviest option reflexively. There's a real spectrum, laid out in the table below — from a library you embed in one app, through a lightweight embedded store for prototypes, to self-hosted and fully-managed services for production scale, plus the "keep vectors next to your existing relational data" option (pgvector). The right choice falls out of honest questions: How many vectors? Do you need filtering and multi-tenancy? Are you willing to run a server, or do you want someone else to? Choosing a distributed cluster for a 50k-document internal tool is as much an error as bolting brute-force onto 50 million records.

python# pip install chromadb
#   import chromadb
#   client = chromadb.Client()
#   col = client.create_collection("runbooks")
#   col.add(
#       ids=["r1", "r2"],
#       documents=["how to scale prod", "rollback a bad deploy"],
#       metadatas=[{"env": "prod"}, {"env": "prod"}],   # <-- filterable
#   )
#   res = col.query(query_texts=["scale up"], n_results=3,
#                   where={"env": "prod"})   # metadata filter + vector search
print("Chroma auto-embeds docs and combines vector search with metadata filters")

5 · Hybrid search & re-ranking expert expert

Ask an embedding search for error code "SG-4471" and it may shrug. Semantic search is brilliant at meaning and terrible at literals — exact IDs, product SKUs, function names, rare surnames, that one acronym. It smooths everything into "close in meaning," which is exactly wrong when the user needs that precise token and nothing near it. Meanwhile old-fashioned keyword search nails the literal but has no idea that "cancel my plan" relates to "end subscription." Each is strong precisely where the other is weak — so the professional move is to stop choosing.

Hybrid search runs both at once: dense (embedding) retrieval for meaning and sparse lexical retrieval — classically BM25, which scores by term frequency — for exact matches. Now you have two ranked lists that disagree, and here's the trap: their scores aren't comparable. A cosine of 0.82 and a BM25 score of 14.3 live on different planets; averaging them is meaningless. The fix, Reciprocal Rank Fusion (RRF), sidesteps the whole problem by throwing the raw scores away and fusing on rank position alone — a document earns points for finishing near the top of either list, and the biggest points for finishing high in both. Agreement across two different methods is a strong relevance signal, and RRF needs zero calibration to exploit it.

One more stage sharpens the result. The retrieval so far is fast but coarse — it's good at recall (fetching a wide net that probably contains the right answers) and only okay at precision (putting the single best one first). So the pattern is retrieve wide, then re-rank narrow: pull the top ~20 candidates cheaply, then hand just those to a re-ranker — a cross-encoder that reads the query and each candidate together (not as separate pre-computed vectors) and scores true relevance far more accurately. It's too slow to run over millions, which is the whole point: you only ever run it on the shortlist. Fuse broadly to catch everything, then re-rank to sharpen the top.

pythondef rrf(rankings, k=60):
    """Fuse several ranked ID lists into one. rankings: list of [id, id, ...]."""
    scores = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)   # fused order

dense = ["d3", "d1", "d7"]        # from vector search
sparse = ["d1", "d9", "d3"]       # from BM25
print(rrf([dense, sparse]))       # d1 & d3 rise (they appear in both)
# Then rerank the top ~20 with a cross-encoder for final precision.

6 · LangChain vs LlamaIndex — what they are intermediate

You've now hand-built every piece: embed, index, retrieve, fuse, re-rank. That's a lot of glue code — connectors for every file type, adapters for every vector store, plumbing for every model provider. Rewriting all of it for each project is tedious, and this is the itch frameworks scratch. LangChain and LlamaIndex are Python libraries that ship those pieces pre-assembled, so a RAG pipeline that took you a page of code becomes three lines. The catch — and the reason this section is worth reading carefully — is that they don't add new ideas; they package the ones you already own, and packaging always hides something.

Think of them like two kitchen setups. LlamaIndex is the specialist's data kitchen: its whole gravity is ingestion and retrieval — loading heterogeneous documents, chunking them, building indexes, and querying them well. LangChain is the generalist's orchestration kitchen: chains, agents, tools, and memory glued together, spanning many providers and multi-step workflows. They overlap heavily (both can do RAG; both can call tools), so the useful question is never "which is better" but "which one's center of gravity matches my problem." The table below lines up their focus, strengths, and core abstractions so you can match a project to the right tool rather than defaulting to whichever you heard of first.

7 · How the frameworks are built advanced

A framework feels like magic right up until it breaks — and then it feels like a wall. Retrieval quality drops, or latency spikes, or the bill triples, and if the framework is a black box to you, you're stuck poking at config flags hoping one helps. The cure is to look under the hood once, and discover there's no engine you haven't seen. Every framework abstraction is just a formalized version of a primitive this course already built by hand. Learn the mapping and the black box turns into a labelled diagram you can debug.

Take LlamaIndex first, because its vocabulary is almost a mirror of the RAG pipeline. A Document is your raw source; it gets split into Nodes (the chunks from A5); each Node is embedded and loaded into an Index (vector, keyword, or graph — the structures from this page and D4/D5); a Retriever pulls the relevant Nodes; and a Query Engine wraps "retrieve + synthesize into an answer." Read that chain again: it is exactly embed → index → retrieve → prompt, just with proper nouns. Nothing new, only named.

LangChain organizes the same reality around composition. Its core idea is the Runnable — any step with an .invoke() — and its LCEL syntax pipes them with | like a Unix shell (prompt | model | parser), so data flows left to right through stages you can swap. Around that sit Tools (the registry pattern from A1), Agents (the reason-act loop you built in Ch 4), and Memory (the message buffer that keeps conversation state). So when you can name which primitive a call maps to — "this is the retriever," "this is the chunker" — you can reason about where a failure lives instead of guessing, which is the entire payoff of having built these by hand first.

python# LlamaIndex — RAG in a few lines:
#   from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
#   docs = SimpleDirectoryReader("runbooks/").load_data()
#   index = VectorStoreIndex.from_documents(docs)   # chunk+embed+index
#   answer = index.as_query_engine().query("how do we roll back?")
#
# LangChain LCEL — compose steps with | (each is a "Runnable"):
#   chain = prompt | model | output_parser
#   chain.invoke({"question": "..."})
#
# Both wrap: chunking (A5) + embeddings + vector index (this page) +
# the agent loop (Ch 4) + structured output (A6). No magic — just glue.
print("Frameworks = the course's concepts, pre-assembled")

8 · Build direct vs use a framework advanced

This is the decision that starts more team arguments than any other in RAG work — and both camps are usually arguing past each other. "Frameworks are bloated abstraction soup." "Hand-rolling everything is reinventing wheels." The reason it turns into a religious war is that people treat it as a permanent identity choice rather than what it actually is: a per-project engineering trade-off with honest costs on both sides, and often a sequence rather than a single pick.

The trade is straightforward once you name it. A framework buys you breadth and speed-to-ship — dozens of document loaders, vector-store adapters, and provider integrations already written and battle-tested — at the cost of an abstraction tax: indirection to debug, opinions you may not share, and some lock-in. Building direct (SDK plus your own code, the way this course does) buys you control and transparency — you see and tune every prompt, every millisecond of latency, every dollar of cost — at the cost of writing and maintaining the plumbing yourself. Neither is virtuous; they're suited to different situations.

So resist the two mirror-image mistakes: adopting a heavy framework for a small focused app you could hand-build in an afternoon, and hand-rolling connectors for ten data sources when a framework already ships all ten. A common, sane path is to prototype on a framework to move fast, then drop to the direct SDK on the hot paths where control actually pays off. The callouts below give the honest decision criteria — and, more importantly, the reminder that whichever you pick, the fundamentals from this whole track (chunking, embeddings, indexes, the loop, structured output, resiliency) are what you're really running; the framework just labels them.

Exercises expert

🎯 Interview practice interview

The interview questions this topic gets asked — worked, with code. For the full pattern catalog see A9 · Big Tech AI-engineering patterns.

python# OFFLINE:  docs -> chunk (overlap) -> embed -> ANN index (HNSW/IVF) + metadata
# ONLINE:   query -> embed -> retrieve top-k -> rerank -> prompt w/ context -> cite
# SCALE:    shard the index; cache hot queries; hybrid dense+BM25 (RRF)
# FAILURE:  stale index, retrieval miss -> hallucination; measure recall@k + answer evals

pythonimport numpy as np
def top_k(corpus, query, k=5):
    c = corpus / np.linalg.norm(corpus, axis=1, keepdims=True)
    q = query / np.linalg.norm(query)
    scores = c @ q
    idx = np.argpartition(-scores, k)[:k]
    return idx[np.argsort(-scores[idx])]

Checkpoint expert

import math

def cosine(a, b):
    dot  = sum(x*y for x, y in zip(a, b))
    na   = math.sqrt(sum(x*x for x in a))
    nb   = math.sqrt(sum(y*y for y in b))
    return dot / (na * nb) if na and nb else 0.0

query = [1.0, 0.0, 1.0]
docs  = {"d1": [1.0, 0.0, 0.9], "d2": [0.0, 1.0, 0.0], "d3": [0.5, 0.1, 0.6]}
ranked = sorted(docs, key=lambda k: cosine(query, docs[k]), reverse=True)
print(ranked)                          # ['d1', 'd3', 'd2']
print(round(cosine(query, docs["d1"]), 3))   # 0.986

import math

def cosine(a, b):
    dot = sum(x*y for x, y in zip(a, b))
    na, nb = math.sqrt(sum(x*x for x in a)), math.sqrt(sum(y*y for y in b))
    return dot / (na*nb) if na and nb else 0.0

def topk(query, corpus, k=2):
    scored = [(cid, cosine(query, v)) for cid, v in corpus.items()]
    return sorted(scored, key=lambda t: t[1], reverse=True)[:k]

corpus = {f"d{i}": [i%3, (i+1)%3, (i+2)%3] for i in range(6)}
print(topk([1,0,1], corpus, k=2))

# cost: O(N * dim) per query. At N=10M vectors this is ~10M dot products
# EVERY query -> too slow. That's the wall exact search hits.

import math
def cosine(a,b):
    d=sum(x*y for x,y in zip(a,b)); na=math.sqrt(sum(x*x for x in a)); nb=math.sqrt(sum(y*y for y in b))
    return d/(na*nb) if na and nb else 0.0

corpus = {f"d{i}": [(i*7)%5, (i*3)%5, (i)%5] for i in range(12)}
q = [2,1,0]

def exact_topk(k):
    return [c for c,_ in sorted(corpus.items(), key=lambda t: cosine(q,t[1]), reverse=True)[:k]]

def ann_topk(k, probe=6):                       # only scan a subset of candidates
    subset = list(corpus.items())[:probe]       # stand-in for an index's shortlist
    return [c for c,_ in sorted(subset, key=lambda t: cosine(q,t[1]), reverse=True)[:k]]

def recall_at_k(k):
    truth = set(exact_topk(k)); got = set(ann_topk(k))
    return len(truth & got) / len(truth)

print("exact:", exact_topk(3))
print("ann:  ", ann_topk(3))
print(f"recall@3: {recall_at_k(3):.0%}")

def rrf(rankings, k=60):
    scores = {}
    for ranked in rankings:
        for rank, doc in enumerate(ranked):     # 0-based rank
            scores[doc] = scores.get(doc, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=lambda d: scores[d], reverse=True)

dense  = ["d2", "d1", "d5"]   # by semantic similarity
sparse = ["d5", "d3", "d2"]   # by keyword/BM25 match
print(rrf([dense, sparse]))
# d2 first: ranks high in dense AND appears in sparse -> fused to the top

def first_stage(query, corpus):                 # cheap: raw keyword frequency
    q = query.lower().split()
    scored = [(d, sum(t.lower().split().count(w) for w in q)) for d,t in corpus]
    return [d for d,_ in sorted(scored, key=lambda x:x[1], reverse=True)]

def cross_encoder(query, text):                 # richer: fraction of query terms covered
    q, t = set(query.lower().split()), set(text.lower().split())
    return len(q & t) / len(q)

corpus = [
    ("d1", "refund refund refund refund policy"),      # spams 'refund', misses 'window'
    ("d2", "how to enable two factor authentication"),
    ("d3", "the refund window is five business days"),  # actually answers the query
]
wide = first_stage("refund window", corpus)
texts = dict(corpus)
reranked = sorted(wide, key=lambda d: cross_encoder("refund window", texts[d]), reverse=True)
print("first-stage:", wide)      # d1 inflated by repetition
print("reranked   :", reranked)  # d3 promoted — covers both query terms

def choose(need_fine_control, standard_pipeline, team_small, care_about_lockin):
    if need_fine_control and care_about_lockin:
        return "BUILD DIRECT — you own the retriever/index/rerank; no abstraction tax"
    if standard_pipeline and team_small:
        return "FRAMEWORK — LlamaIndex for doc-QA/ingestion; LangChain for chains/agents"
    if standard_pipeline:
        return "FRAMEWORK to prototype, then peel back hot paths you need to control"
    return "BUILD DIRECT — non-standard pipeline; a framework fights you"

print(choose(False, True,  True,  False))   # FRAMEWORK
print(choose(True,  False, False, True))    # BUILD DIRECT

Knowledge check check yourself