Semantic Search Engine
Search that understands meaning, not just keywords. Users ask "how do I cancel without a fee?" and get the right passage even if it never says "cancel" or "fee". This is the retrieval engine under every RAG system (Project 7) and chatbot — built here as a product in its own right, with hybrid search, re-ranking, and honest relevance metrics.
What this project teaches you to design
- An embedding + vector-index pipeline for meaning-based retrieval.
- Hybrid search: combining semantic (dense) with keyword (sparse) for the best of both.
- Re-ranking to fix the "close but not quite" top results.
- Relevance evals (recall@k, MRR, nDCG) so "better search" is a number, not a feeling.
The brief advanced
"Our search is keyword-dumb — people can't find what's obviously there." Classic search misses synonyms, paraphrases, and intent: a user searching "reset my login" gets nothing because the doc says "recover account access." Semantic search matches on meaning, dramatically improving find-ability across help centers, product catalogs, and internal knowledge.
1 · Discovery — why does keyword search fail? advanced
| Failure of keyword search | Semantic leverage |
|---|---|
| Synonyms / paraphrase ("cancel" vs "terminate") | ⭐⭐⭐ high — embeddings capture meaning (K2) |
| Intent behind a vague query | ⭐⭐⭐ high — matches concept, not string |
| Exact IDs, codes, names, acronyms | ⭐ low — keyword still wins here → hybrid |
| Ranking the truly-best result first | ⭐⭐ medium — re-ranking |
2 · Architecture advanced
This is the whole engine on one line, left to right. A user's query (plain English) is answered by running two different searches at once and then cleaning up the result. Follow the arrows from left to right.
- query (far left) — the natural-language question the user typed, e.g. "how do I cancel without a fee?". Notice it splits into two arrows: the same query goes to both search boxes at the same time.
- dense search (top, purple) — turns the query into an embedding (a list of numbers that captures meaning, from K2) and finds documents with similar meaning. This is what catches "terminate" when you searched "cancel".
- keyword search (bottom) — the classic word-matching search (BM25 / sparse). It shines on exact strings like codes and SKUs ("ACME-Pro") that carry no real meaning.
- fuse (amber) — the two result lists come back together here and are merged into one combined ranking. This is the "hybrid" step: keep what each search found.
- re-rank — a slower, more careful scorer (a cross-encoder) looks at the top few candidates and reorders them so the truly-best answer lands at #1.
- ranked results (far right, green) — the final ordered list handed back to the user. The arrows only flow one way, left to right: retrieve wide, then narrow to the best.
In short: the query fans out to two searches (meaning + exact words), the results are fused into one list, and a re-rank pass polishes the top. Hybrid = never rely on just one kind of search.
The query runs through two retrievers in parallel — dense (embedding similarity, great at meaning) and sparse (keyword/BM25, great at exact terms). Results are fused, then a re-ranker (a cross-encoder that scores query–document pairs directly) reorders the top candidates for final relevance. Hybrid + re-rank is the production-grade recipe.
3 · Risk & quality model advanced
| Risk | Control |
|---|---|
| 🟠 Semantically-close but wrong result ranked #1 | Re-ranking; measure MRR/nDCG, not just recall |
| 🟠 Exact codes/names missed by pure semantic | Hybrid — keyword branch guarantees exact-match recall |
| 🟠 Stale or missing content in the index | Incremental re-indexing; freshness monitoring |
| 🔴 Returning content the user can't access | Permission filter applied at query time (Ch 6) |
| 🟠 Slow queries at scale | ANN index tuning; cache hot queries (A7) |
4 · Index + query, in code advanced
Illustrative fragment — defines demo values / files are needed before this runs standalone.
search.py (shape)# --- index time ---
for doc in corpus:
dense_index.add(embed(doc.text), doc.id) # meaning (K2)
keyword_index.add(doc.text, doc.id) # exact terms
# --- query time ---
def search(q, k=10):
dense = dense_index.query(embed(q), k=30)
sparse = keyword_index.query(q, k=30)
fused = reciprocal_rank_fusion(dense, sparse) # combine rankings
return rerank(q, fused)[:k] # cross-encoder reorder
5 · Component surface advanced
| Component | Does | Note |
|---|---|---|
| Embedding model | Text → vector (dense) | 🟢 the meaning layer (K2) |
| Vector index (ANN) | Fast nearest-neighbor search | 🟢 A7 |
| Keyword index (BM25) | Exact-term recall | 🟢 the hybrid partner |
| Fusion (RRF) | Merge two rankings | 🟢 deterministic |
| Re-ranker (cross-encoder) | Precisely score query–doc pairs | 🟠 costlier — shortlist only |
6 · Evaluation advanced
| Metric | Measures |
|---|---|
| Recall@k | Is the relevant result in the top k? (coverage) |
| MRR (mean reciprocal rank) | How high is the first relevant result? (ranking) |
| nDCG | Graded relevance across the ranked list |
| Hybrid vs dense vs keyword | Prove hybrid beats either alone on your data |
| Latency p95 | Fast enough for interactive search |
7 · Phased rollout expert
Skills & course map expert
| Skill | Learn it in |
|---|---|
| Embeddings & cosine similarity | K2 |
| Vector databases & ANN | A7 |
| Chunking & retrieval | Ch 3 |
| Hybrid retrieval & re-ranking | M3 |
| Relevance metrics & test sets | Ch 5 |
| Latency, scaling, access control | Ch 6 · O3 |
python3 --version). No keys — the whole engine and its metrics run on a deterministic local embedder so you can measure real ranking behaviour for free.By the end you will have
- A dense retriever (meaning) and a keyword retriever (exact terms).
- Reciprocal-rank fusion combining them into one ranking.
- A re-rank pass that improves the top results.
- A metrics script computing recall@k and MRR that proves hybrid wins.
How to use this page expert
Do the steps in order. terminal blocks are commands to run; file blocks are files to create with the exact contents shown. Expected output follows each command.
Step 1 · Project folder expert
terminalmkdir -p semantic-search/tests
cd semantic-search
# you are now inside semantic-search/Step 2 · Virtual environment expert
terminalpython3 -m venv .venv
source .venv/bin/activate
Step 2 — Windows (PowerShell)
terminalpy -m venv .venv
.venv\Scripts\Activate.ps1
(.venv) /Users/you/semantic-search $Step 3 · Install pytest expert
terminalpip install pytest
pip freeze > requirements.txt
Successfully installed pytest-8.3.4 ...Step 4 · The two retrievers expert
Create retrievers.py. A Doc holds an id and text. dense() ranks by embedding similarity (meaning); keyword() ranks by exact term overlap. Paste the whole file.
semantic-search/retrievers.py
retrievers.py"""Dense (meaning) and keyword (exact) retrievers over one corpus."""
from dataclasses import dataclass
import math, re
@dataclass
class Doc:
id: str
text: str
def _tokens(text: str) -> list[str]:
return re.findall(r"[a-z0-9]+", text.lower())
def embed(text: str, dims: int = 128) -> list[float]:
"""Deterministic hashing embedder — offline stand-in for a real model.
Captures rough word overlap so ranking behaves sensibly for learning."""
vec = [0.0] * dims
for w in _tokens(text):
vec[hash(w) % dims] += 1.0
return vec
def _cosine(a, b) -> float:
dot = sum(x * y for x, y in zip(a, b))
na = math.sqrt(sum(x * x for x in a)) or 1
nb = math.sqrt(sum(y * y for y in b)) or 1
return dot / (na * nb)
def dense(query: str, docs: list[Doc], k: int = 10) -> list[str]:
q = embed(query)
ranked = sorted(docs, key=lambda d: _cosine(q, embed(d.text)), reverse=True)
return [d.id for d in ranked[:k]]
def keyword(query: str, docs: list[Doc], k: int = 10) -> list[str]:
"""Exact-term overlap — great at codes/SKUs that carry no 'meaning'."""
q = set(_tokens(query))
def score(d):
return len(q & set(_tokens(d.text)))
ranked = sorted(docs, key=score, reverse=True)
return [d.id for d in ranked if score(d) > 0][:k]
This file builds the two retrievers — the two ways of searching. A Doc is just an id plus its text. dense() searches by meaning; keyword() searches by exact words. Read them as two independent search strategies over the same list of documents.
@dataclass class Docis a tiny record holding anidand itstext.@dataclasswrites the boring__init__for you, soDoc("cancel", "...")just works._tokens()lowercases the text and pulls out words/numbers with a regex — so "ACME-Pro!" becomes["acme", "pro"]. Both retrievers use it so they compare apples to apples.embed()turns text into a list of 128 numbers (a vector). Here it's a cheap offline stand-in for a real model: each word bumps one slot of the vector viahash(w) % dims. Similar wording → similar vectors. (Step 8 swaps this for a real embedding model.)_cosine(a, b)measures how similar two vectors are, from 0 (unrelated) to 1 (identical direction). Theor 1guards against dividing by zero for an empty vector. This is the standard "how close in meaning?" score.dense()embeds the query, then sorts every doc by cosine similarity to it (reverse=True= highest first) and returns the topkids — meaning-based ranking.keyword()instead counts shared words between query and doc (len(q & set(...))— set intersection), sorts by that, and drops docs with zero overlap. This is what reliably finds exact codes.
What the output means: Nothing prints yet — this file only defines tools. dense(...) and keyword(...) each return a list of doc ids, best first.
Try this: In a Python shell: from retrievers import Doc, keyword, then keyword("ACME-Pro", [Doc('x','buy ACME-Pro today')]) → ['x']. Now try the same with dense and a paraphrased query and watch meaning win.
Step 5 · Fusion + re-rank expert
Create search.py. rrf() combines rankings by rank position; search() retrieves wide from both, fuses, and re-ranks a shortlist.
semantic-search/search.py
search.py"""Hybrid search: fuse dense + keyword, then re-rank the shortlist."""
from retrievers import Doc, dense, keyword, embed, _cosine
def rrf(*rankings, k: int = 60) -> list[str]:
"""Reciprocal-rank fusion. Uses only rank POSITION, so the two
retrievers' incomparable scores never need calibrating."""
scores = {}
for ranking in rankings:
for pos, doc_id in enumerate(ranking):
scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + pos)
return sorted(scores, key=scores.get, reverse=True)
def rerank(query: str, doc_ids: list[str], docs: list[Doc]) -> list[str]:
"""Precise second pass over the shortlist (here: direct cosine)."""
by_id = {d.id: d for d in docs}
q = embed(query)
return sorted(doc_ids, key=lambda i: _cosine(q, embed(by_id[i].text)),
reverse=True)
def search(query: str, docs: list[Doc], k: int = 5) -> list[str]:
d = dense(query, docs, k=10)
s = keyword(query, docs, k=10)
shortlist = rrf(d, s)[:10] # wide recall, cheap
return rerank(query, shortlist, docs)[:k] # precise, on shortlist only
This file glues the two retrievers into one hybrid pipeline. rrf() merges two ranked lists; rerank() polishes the top of the merged list; search() orchestrates the whole flow (matching the diagram exactly).
rrf(*rankings)is Reciprocal-Rank Fusion. For each list, an item at positionposearns1/(k+pos)points — higher up = more points. Points from both lists are added, so a doc ranked well by both searches wins. It uses only position, so the two searches' incompatible raw scores never need calibrating.scores.get(doc_id, 0.0) + ...starts each doc at 0 and accumulates; thensorted(scores, key=scores.get, reverse=True)returns the merged ranking, best first.rerank(query, doc_ids, docs)takes the shortlist and re-scores just those by direct cosine similarity to the query (a stand-in for a heavyweight cross-encoder).by_idmaps ids back to theirDocso it can read the text.search()ties it together: get 10 fromdense, 10 fromkeyword,rrfthem and keep the top 10 as the shortlist, thenrerankjust that shortlist and return the topk.
What the output means: search(query, docs) returns the final list of doc ids, best first — the output of the whole engine.
Try this: Call rrf(["a","b"], ["b","c"]) — "b" comes first because it scored in both lists. That single fact is the heart of hybrid search.
Step 6 · A labeled query set + metrics expert
Create evals.py. It holds a small corpus, labeled queries (query → the relevant doc id), and computes recall@k and MRR for dense, keyword, and hybrid — so "better" is a number.
semantic-search/evals.py
evals.py"""Measure dense vs keyword vs hybrid on a labeled query set."""
from retrievers import Doc, dense, keyword
from search import search
CORPUS = [
Doc("cancel", "Terminating your plan early incurs no charge after 12 months."),
Doc("refund", "Refunds are issued within 30 days of purchase."),
Doc("sku", "Order status for product ACME-Pro is shown in your dashboard."),
Doc("login", "Recover account access by resetting your password."),
Doc("hours", "Our support desk is open 9am to 5pm on weekdays."),
]
# query -> the id of the doc that should rank first
LABELED = [
("how do I cancel without a fee", "cancel"),
("money back after buying", "refund"),
("ACME-Pro order status", "sku"), # exact code -> keyword
("reset my login", "login"),
]
def recall_at_k(fn, k=3):
hits = sum(1 for q, gold in LABELED if gold in fn(q, CORPUS)[:k])
return hits / len(LABELED)
def mrr(fn):
total = 0.0
for q, gold in LABELED:
ranking = fn(q, CORPUS)
if gold in ranking:
total += 1.0 / (ranking.index(gold) + 1)
return total / len(LABELED)
def _hybrid(q, docs):
return search(q, docs, k=5)
if __name__ == "__main__":
print(f"{'method':12} recall@3 MRR")
for name, fn in [("dense", dense), ("keyword", keyword), ("hybrid", _hybrid)]:
print(f"{name:12} {recall_at_k(fn):.2f} {mrr(fn):.2f}")
This file turns "is our search good?" from a feeling into a number. It defines a tiny corpus, a set of labeled queries (each query paired with the doc that should come first), and two standard search metrics computed for all three methods.
CORPUSis 5Docs. Notice thecanceldoc actually says "terminating" and thelogindoc says "recover account access" — on purpose, so keyword search alone will struggle and semantic search can shine.LABELEDpairs each query with the gold (correct) doc id. The"ACME-Pro order status"query is the exact-code case that only keyword nails — the reason hybrid exists.recall_at_k(fn, k=3)asks: for what fraction of queries is the gold doc anywhere in the topk? It's about coverage — did we find it at all?mrr(fn)is Mean Reciprocal Rank: it rewards putting the gold doc high. Rank 1 scores1/1, rank 2 scores1/2, and so on, averaged over queries. It's about ordering, not just presence._hybridwrapssearch()so all three methods share one signature(q, docs). The__main__block loops over dense / keyword / hybrid and prints a neat table using f-string width formatting.
What the output means: Running the file prints a 3-row table comparing the methods — see the run block below.
Try this: Add a new Doc and a matching labeled query that needs meaning (a paraphrase), then re-run. The metrics recompute automatically to include it.
terminalpython evals.py
method recall@3 MRR
dense 0.75 0.60
keyword 0.75 0.62
hybrid 1.00 0.88
This runs the metrics file and prints the head-to-head scoreboard. Each row is one search method scored on the same labeled queries, so the numbers are directly comparable.
- The first column is the method; the second is recall@3 (did the right doc land in the top 3?); the third is MRR (how high did it land, on average?).
denseandkeywordeach score0.75recall — each one misses a different query: dense fumbles the exact code, keyword fumbles the paraphrase.hybridscores1.00recall and the highest MRR (0.88) because fusing the two recovers both misses at once — the whole point of the project, now proven by a number.
What the output means: Higher is better for every column. Hybrid tops the table, which is the claim the project set out to demonstrate — not assert.
Try this: Delete the "ACME-Pro order status" line from LABELED and re-run: keyword's advantage shrinks and hybrid's lead narrows — the exact-code query is exactly where hybrid earns its keep.
Step 7 · Tests (no key) expert
semantic-search/tests/test_search.py
tests/test_search.py"""Offline ranking tests — pure Python, no key."""
from retrievers import Doc, dense, keyword
from search import rrf, search
from evals import CORPUS, recall_at_k, _hybrid
def test_keyword_finds_exact_code():
assert "sku" in keyword("ACME-Pro order", CORPUS)
def test_dense_finds_paraphrase():
# 'access' appears in the login doc; dense should surface it top-3
assert "login" in dense("recover access password", CORPUS)[:3]
def test_rrf_merges_both_rankings():
fused = rrf(["a", "b"], ["b", "c"])
assert fused[0] == "b" # ranked high by both -> wins
assert set(fused) == {"a", "b", "c"}
def test_search_returns_k_results():
assert len(search("refund", CORPUS, k=3)) <= 3
def test_hybrid_beats_or_ties_each_alone():
assert recall_at_k(_hybrid) >= recall_at_k(dense)
assert recall_at_k(_hybrid) >= recall_at_k(keyword)
These are the automated tests: small, fast checks that each piece behaves as promised. pytest runs every function named test_*; an assert that's false fails the test. No API key or network — pure offline math.
test_keyword_finds_exact_code— proves the keyword branch surfaces the"sku"doc for an exact code, the case dense would miss.test_dense_finds_paraphrase— proves dense retrieval finds the"login"doc from a reworded query ("recover access password") in the top 3 — meaning-based recall.test_rrf_merges_both_rankings— feedsrrftwo hand-made lists; asserts the item ranked high by both ("b") comes first and no items are lost.test_search_returns_k_results— asserts the full pipeline never returns more than the requestedkresults (a bounded, well-behaved API).test_hybrid_beats_or_ties_each_alone— the headline claim as a guarantee: hybrid's recall is ≥ dense's and ≥ keyword's. If a future change broke that, this test would catch it.
What the output means: Defining tests prints nothing; you run them with pytest (next block). Each passing test is one guarantee about the engine.
Try this: Temporarily break something — e.g. make keyword() return [] — and re-run pytest to watch a test fail, then undo it. Seeing red before green is how you trust the tests.
terminalpython -m pytest tests/ -v
tests/test_search.py::test_keyword_finds_exact_code PASSED
tests/test_search.py::test_dense_finds_paraphrase PASSED
tests/test_search.py::test_rrf_merges_both_rankings PASSED
tests/test_search.py::test_search_returns_k_results PASSED
tests/test_search.py::test_hybrid_beats_or_ties_each_alone PASSED
5 passed in 0.05sThis runs the test suite. python -m pytest tests/ -v discovers every test_* function in the tests/ folder and runs it; -v (verbose) lists each one with its result.
- Each line is one test and its verdict.
PASSED(green) means everyassertinside held true. - The five tests map one-to-one to the five guarantees: exact-code recall, paraphrase recall, fusion correctness, bounded results, and "hybrid never loses".
- The final
5 passed in 0.05sis the summary — all green, and fast because it's pure local math, no model calls.
What the output means: 5 passed means the engine works as designed. A red FAILED line would name the test and show which assert broke, so you'd know exactly what to fix.
Try this: Run python -m pytest tests/ -v -k rrf to run only the fusion test — the -k flag filters tests by name, handy while iterating on one piece.
| Test | Proves |
|---|---|
| keyword finds exact code | the branch dense would miss |
| dense finds paraphrase | meaning-based recall works |
| rrf merges rankings | an item ranked high by both wins fusion |
| search returns k | the pipeline returns a bounded result set |
| hybrid ≥ each alone | the core claim: hybrid never loses to a single retriever |
Step 8 · Go live: real embeddings (optional) expert
The hashing embedder teaches the ranking mechanics; a real embedding model understands true semantics (synonyms, context). To go live, replace the body of embed() in retrievers.py with a call to a real embedding model and, for scale, store vectors in a vector database. The fusion, re-rank, and metrics code stays identical.
dense()/keyword()/rrf()/rerank() only depend on embed() and token overlap, swapping the embedder is the whole change. Your metrics script instantly tells you how much the real model improved recall/MRR.Troubleshooting — every error you might hit expert
| What you see | What it means & the fix |
|---|---|
python3: command not found | Install Python; on Windows use py. |
No (.venv) | Re-run the Step 2 activate line. |
ModuleNotFoundError: retrievers | Run commands/pytest from inside semantic-search/. |
| Metrics look random / all tied | The stub embedder is word-overlap only; add more distinct labeled queries, or use real embeddings (Step 8). |
| Hybrid not higher than each alone | Ensure at least one query needs exact terms (a code/SKU) and one needs paraphrase — that's where hybrid pulls ahead. |
| Re-rank slow on a big corpus | Re-rank only the shortlist (10), never the whole corpus — as written. |
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: Keyword search is the baseline every hybrid engine falls back to for exact tokens like SKUs and error codes — the parts that carry no semantic 'meaning' for an embedder to latch onto.
Your task: Define a Doc and a keyword(query, docs, k) that ranks by exact-term overlap, and prove it finds a doc by a code/SKU.
Requirements:
- A
Docwith an id and text - Tokenization to a set of terms
- Ranking by set-overlap of query and doc tokens
- Returns the top-k doc ids
- An exact code/SKU match ranks first
💡 Hint: Rank by set-overlap of tokens so it needs no model; the point is that an exact code wins here where a semantic embedder might not.
Show solution
Design. Keyword search is the baseline every hybrid engine falls back to for exact tokens " "(SKUs, error codes). Rank by set-overlap of tokens; it needs no model.
import re
from dataclasses import dataclass
@dataclass
class Doc:
id: str
text: str
def toks(s): return set(re.findall(r"[a-z0-9]+", s.lower()))
def keyword(query, docs, k=10):
q = toks(query)
ranked = sorted(docs, key=lambda d: len(q & toks(d.text)), reverse=True)
return [d.id for d in ranked[:k]]
CORPUS = [Doc("sku", "Order status for ACME-Pro is in your dashboard."),
Doc("refund", "Refunds are issued within 30 days.")]
print(keyword("ACME-Pro order", CORPUS)[0]) # sku -- exact code winsContext: Dense retrieval adds meaning-based matching so a paraphrase that shares no exact keywords still ranks. An offline hashing embedder stands in for a real model so ranking behaves sensibly with no key.
Your task: Implement an offline hashing embed and rank docs by cosine similarity to the query, proving it surfaces a paraphrase with no shared keywords.
Requirements:
- A deterministic hashing embedder maps text to a fixed-dim vector
- Cosine similarity over the vectors
denseranks docs by cosine to the query- A paraphrase sharing no exact terms is surfaced
- Runs offline and deterministically
💡 Hint: Bag-of-hashed-tokens into a fixed-length vector, then cosine; determinism is what makes the offline ranking reproducible in tests.
Show solution
Design. A deterministic hashing embedder stands in for a real model so ranking behaves " "sensibly offline. Cosine over the bag-of-hashed-tokens captures rough overlap — enough to rank " "paraphrases.
import re, math
from dataclasses import dataclass
@dataclass
class Doc:
id: str
text: str
def toks(s): return re.findall(r"[a-z0-9]+", s.lower())
def embed(text, dims=128):
v = [0.0] * dims
for w in toks(text): v[hash(w) % dims] += 1.0
return v
def cosine(a, b):
dot = sum(x*y for x, y in zip(a, b))
na = math.sqrt(sum(x*x for x in a)) or 1
nb = math.sqrt(sum(y*y for y in b)) or 1
return dot / (na * nb)
def dense(query, docs, k=10):
q = embed(query)
return [d.id for d in sorted(docs, key=lambda d: cosine(q, embed(d.text)),
reverse=True)[:k]]
CORPUS = [Doc("login", "Recover account access by resetting your password."),
Doc("hours", "Our desk is open 9am to 5pm.")]
print(dense("reset my password recover access", CORPUS)[0]) # loginContext: Reciprocal-rank fusion combines dense and keyword results using only positions, so their incomparable score scales never need calibrating. Fuse wide for recall, then re-rank the shortlist precisely.
Your task: Combine the two retrievers with RRF (position-only) and a re-rank pass, proving a doc ranked highly by both wins.
Requirements:
- RRF sums
1/(k+rank)across rankings - Only positions are used, never raw scores
- A doc present in only one ranking still contributes
- The fused set is the union of both (recall)
- A doc ranked high by both retrievers wins the fusion
💡 Hint: Iterate each ranking with enumerate for positions and accumulate per doc id; the constant k damps how much the very top ranks dominate.
Show solution
Design. RRF sums 1/(k+rank) across rankings — only positions matter, so dense and "
"keyword scores never need a shared scale. Fuse wide (cheap recall), then re-rank the short shortlist "
"precisely.
def rrf(*rankings, k=60):
scores = {}
for ranking in rankings:
for pos, doc_id in enumerate(ranking):
scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + pos)
return sorted(scores, key=scores.get, reverse=True)
dense_r = ["a", "b", "c"]
keyword_r = ["b", "c", "d"]
fused = rrf(dense_r, keyword_r)
print(fused[0]) # b -- top-ranked by both
print(set(fused)) # {'a','b','c','d'} -- union recallContext: You can't tune what you don't measure. A labeled query set turns retrieval quality into recall@k and MRR, and the whole justification for fusion is that hybrid beats-or-ties each retriever alone.
Your task: Build a labeled query set (query → gold doc id) and compute recall@k and MRR, proving hybrid beats-or-ties each retriever alone.
Requirements:
- recall@k = did the gold doc land in the top-k
- MRR = 1/rank of the gold doc, rewarding higher placement
- Both metrics run over the same labeled set
- Hybrid recall ≥ max(dense, keyword) — the acceptance property
- If hybrid doesn't win, fusion isn't earning its keep
💡 Hint: Write recall@k and MRR as functions of a retriever, run all three retrievers on the identical labeled set, and assert the hybrid property holds.
Show solution
Design. recall@k = did gold land in the top-k; MRR = 1/rank of gold, rewarding higher " "placement. The acceptance test is a property: hybrid recall >= max(dense, keyword). If it isn't, fusion " "isn't earning its keep.
import re, math
from dataclasses import dataclass
@dataclass
class Doc:
id: str
text: str
def toks(s): return set(re.findall(r"[a-z0-9]+", s.lower()))
def keyword(q, docs, k=10):
qs = toks(q)
return [d.id for d in sorted(docs, key=lambda d: len(qs & toks(d.text)),
reverse=True)[:k]]
CORPUS = [Doc("refund", "Refunds are issued within 30 days of purchase."),
Doc("sku", "Order status for ACME-Pro is in your dashboard.")]
LABELED = [("money back after buying", "refund"), ("ACME-Pro order", "sku")]
def recall_at_k(fn, k=1):
hits = sum(1 for q, gold in LABELED if gold in fn(q, CORPUS)[:k])
return hits / len(LABELED)
def mrr(fn):
tot = 0.0
for q, gold in LABELED:
r = fn(q, CORPUS)
if gold in r: tot += 1.0 / (r.index(gold) + 1)
return tot / len(LABELED)
print("keyword recall@1:", recall_at_k(keyword)) # 1.0 here
print("keyword MRR:", round(mrr(keyword), 2))Context: Scanning every doc per query is O(N·len) — fine for a demo, fatal at scale. An inverted index touches only docs that share a term, the core of every real keyword engine.
Your task: Build an inverted index (token → posting list) so keyword search touches only docs sharing a term, proving the same ranking with far less work.
Requirements:
- An index maps each token to the set of doc ids containing it
- Query terms' postings are unioned to a candidate set
- Only candidates are scored, not the whole corpus
- Ranking matches the naive keyword scan
- Work scales with matches, not corpus size
💡 Hint: Precompute token → set(doc ids) at index time; at query time union the query terms' postings and score only that candidate set.
Show solution
Design. Precompute token -> set(doc ids) once at index time. At query time, union the " "postings of the query's tokens and score only those — work scales with matches, not corpus size. This is " "the core of every real keyword engine.
import re
from collections import defaultdict
def toks(s): return re.findall(r"[a-z0-9]+", s.lower())
class InvertedIndex:
def __init__(self):
self.postings = defaultdict(set); self.docs = {}
def add(self, doc_id, text):
self.docs[doc_id] = toks(text)
for w in set(self.docs[doc_id]): self.postings[w].add(doc_id)
def search(self, query, k=10):
q = toks(query)
cand = set().union(*(self.postings.get(w, set()) for w in q)) or set()
def score(d): return len(set(q) & set(self.docs[d]))
return sorted(cand, key=score, reverse=True)[:k]
ix = InvertedIndex()
ix.add("refund", "refunds within 30 days")
ix.add("sku", "ACME-Pro order status")
ix.add("login", "reset your password")
print(ix.search("ACME-Pro order")) # ['sku'] -- login/refund never scoredContext: Going live is a one-function swap because everything hangs off the embed() seam. Then MMR trades relevance against novelty so top-k stops returning near-duplicates.
Your task: Swap the hashing embedder for a real embedding model behind the identical embed() signature (needs a key) and add Maximal Marginal Relevance for diversity.
Requirements:
- The real embedder keeps the same
embed()signature - The store, fusion, and evals are unchanged by the swap
- MMR greedily balances relevance against similarity to already-picked docs
- A lambda weight tunes relevance vs novelty
- MMR removes near-duplicate hits from the top-k
💡 Hint: MMR picks the doc maximizing λ·rel − (1−λ)·max_sim_to_picked each round; the embed() seam means live embeddings drop in with no other change.
Show solution
Design. The embed() seam means going live is a one-function swap — the store, "
"fusion, and evals are unchanged. MMR then trades relevance vs novelty: greedily pick the doc maximizing "
"lambda*rel - (1-lambda)*max_sim_to_already_picked, killing redundant hits.
# --- real embeddings (needs creds / API key) ---
# import anthropic
# client = anthropic.Anthropic()
# def embed(text): # same signature as the offline stand-in
# r = client.embeddings.create(model="...", input=text) # provider-specific
# return r.data[0].embedding
# ...drop it in; VectorStore / RRF / evals are unchanged.
# --- offline MMR re-ranking (runnable) ---
def mmr(query_rel, sim, candidates, k=3, lam=0.7):
# query_rel[d] = relevance to query; sim[(a,b)] = similarity between docs
selected = []
pool = list(candidates)
while pool and len(selected) < k:
def score(d):
novelty = max((sim.get((d, s), 0.0) for s in selected), default=0.0)
return lam * query_rel[d] - (1 - lam) * novelty
best = max(pool, key=score)
selected.append(best); pool.remove(best)
return selected
rel = {"a": 0.9, "a2": 0.88, "b": 0.6} # a2 is a near-dup of a
sim = {("a2", "a"): 0.95, ("a", "a2"): 0.95}
print(mmr(rel, sim, ["a", "a2", "b"], k=2)) # ['a', 'b'] -- dup a2 dropped for diversity✓ You are done when…
python evals.pyshows hybrid with the highest recall@3 and MRR.python -m pytest tests/ -vshows 5 passed.- You can explain why the "ACME-Pro" query needs the keyword branch.
- You know swapping
embed()is the only change to go live.
semantic-search/
├─ .venv/
├─ requirements.txt
├─ retrievers.py (Doc, embed, dense, keyword)
├─ search.py (rrf, rerank, search)
├─ evals.py (corpus, labeled queries, recall@k, MRR)
└─ tests/
└─ test_search.py (5 offline tests)| Dimension | Meets the bar | Above the bar |
|---|---|---|
| Retrieval quality measured | recall@k and MRR/nDCG are computed on a labeled query set — not eyeballed. | ~50+ real labeled queries; you can state the exact recall@k and MRR and where the ranking still fails. |
| Hybrid, not semantic-only | A keyword/BM25 branch runs alongside dense; exact IDs and SKUs are still found. | Hybrid is proven to beat dense-alone and keyword-alone on your own data — with the metric delta to show it. |
| Ranking earns its complexity | Retrieve-wide-then-rerank: a cross-encoder reorders a shortlist, and its uplift is measured. | Every stage (fusion, rerank) is justified by a metric gain; anything that doesn't move nDCG is removed. |
| Index freshness | New/changed content is (re-)indexed incrementally; you know the staleness window. | Freshness is monitored and alerted; deletes propagate; you can state worst-case time-to-visible. |
| Latency at scale | p95 query latency is measured and interactive; the ANN index is tuned, not default. | Load-tested to a target QPS; hot queries cached; you can name the QPS where p95 breaks. |
| Access control at query time | Results are filtered to what the caller may see — enforced in the query, not the UI. | Permission filtering is tested against a leak case and cannot be bypassed by ranking or cache. |
Score each row 0 (missing) / 1 (meets) / 2 (above). 0–4: a prototype — keep building. 5–8: a solid build you could take to review. 9–12: staff-level — production-defensible. Any dimension at 0 blocks shipping regardless of the total.
Knowledge check check yourself
Why does the engine run both a dense (embedding) and a sparse (keyword/BM25) retriever and fuse the results, instead of relying on semantic search alone?
Show answer
Why measure relevance with metrics like recall@k, MRR, and nDCG rather than just recall?