Document Intelligence with RAG

This is a sketch, not runnable code — it shows the shape of an ingestion pipeline so the real files later make sense. Read it as the four jobs every document has to go through before it can be searched: read it, understand it, cut it up, and file it away.

1. for doc in corpus: means "do all of the following once for every document in the pile". corpus is just your whole set of documents.
2. text = parse(doc) turns the raw file (a PDF, a scan) into plain text you can work with. meta = extract_metadata(text) pulls out the structured facts — vendor, type, dates — reusing the Project 4 extraction idea.
3. for chunk in chunk_by_section(text): cuts the document into pieces along its structure (sections, clauses) rather than every 500 characters — so a piece is a whole thought, not a sentence sliced in half.
4. index.add(...) files each piece away with three things: its embedding (the meaning-as-numbers), its text, and its metadata — crucially including doc_id and page so every piece can be traced back to where it came from.

Try this: Notice {**meta, "doc_id": ..., "page": ...} — the **meta copies in all the extracted fields, then adds the source location on top. That source location is what makes citations possible later.

This search method makes one decision that matters for safety: it narrows the candidates by metadata before it ever ranks by similarity. Order is the whole point — a chunk a user isn't allowed to see is removed from the pool before scoring, so it can never surface in results.

1. pool = [c for c in self.chunks if not where or all(...)] keeps only chunks that match the where filter (e.g. {"owner": "alice"}). If where is empty, every chunk stays.
2. q = embed(query) turns the question into a vector, then sorted(pool, key=lambda c: cosine(q, c.embedding), reverse=True) orders the survivors best-match-first.
3. The final line returns the top k but only those with similarity above 0 — so a query about something not in the corpus returns an empty list instead of the least-bad guess.

Try this: This is the access-control point the risk table warned about, enforced in real code. Compare it to filtering after ranking: if you ranked first, a forbidden document could top the list and only then get hidden — one bug away from a leak.

This file is the engine room of the search: it turns text into numbers, measures how alike two pieces of text are, and stores everything. It runs fully offline so you can learn RAG without paying for anything or setting up an account.

1. embed(text) turns a string into a list of 64 numbers (a vector). It lower-cases the text, splits it into words, and for each word bumps up one slot in the list. Same words → similar lists. It's deliberately simple — Step 9 swaps it for a real, smarter embedder.
2. cosine(a, b) scores how similar two vectors are, from 0 (nothing in common) to 1 (identical direction). This is the standard way to compare meaning-vectors; the if na == 0 or nb == 0 guard just avoids dividing by zero for empty text.
3. @dataclass class Chunk is a tidy record for one stored piece: its text, which document it came from (doc_id), the page, free-form meta, and its embedding. field(default_factory=dict) just gives each Chunk its own empty dict/list instead of accidentally sharing one.
4. Store.add embeds a chunk (if it isn't already) and keeps it in a list. Store.search is the heart: it first builds pool by filtering on metadata, then embeds the query, sorts the pool by cosine similarity (highest first), and returns the top k — dropping any with zero similarity so an off-topic question comes back empty.

What the output means: Nothing prints — this file only defines the tools. Other files import Store, Chunk and embed and put them to work.

Try this: Look at the very last line: it re-checks cosine(q, c.embedding) > 0 so results with no word overlap are thrown away. That one filter is why the out-of-corpus question later can honestly answer "I don't have that."

Now the real ingest file. Its job is to take one document's text and hand the store a series of clean, self-contained pieces — each tagged with a page number so it can be cited later.

1. chunk_sections(text) splits the document on blank lines (text.split("\n\n")) — a simple stand-in for "cut on structure, not mid-sentence". Each non-empty block becomes one chunk.
2. yield block, page hands back one section at a time (a generator), and page += 1 gives each section a rising page number so we always know where it sat in the document.
3. ingest(store, doc_id, text, meta) loops over those sections and calls store.add(Chunk(...)) for each — stamping every chunk with the same doc_id and meta, plus its own page.
4. It returns n, the number of chunks added — a small confirmation you can print or assert on.

What the output means: Nothing on its own; called by other code. For a two-section document it adds 2 chunks and returns 2.

Try this: This is where the earlier tip "chunk on structure, not character count" becomes real. Try feeding it text with no blank lines — you'll get one big chunk, and you can feel why good chunk boundaries matter for retrieval.

This system prompt is how you keep the model honest in live mode. It's a set of firm instructions the model must obey before it sees the question.

1. "Answer ONLY from the numbered sources" forbids the model from using its own background knowledge — it may only use the chunks you retrieved and pasted in.
2. "Cite the source id in [brackets] after each claim" forces every statement to point at a specific numbered chunk, so a human can verify it.
3. The last clause tells it to reply with exactly the ABSTAIN string when the sources don't contain the answer — the same honest refusal the offline path produces, now enforced by instruction.
4. client.messages.create(...) sends it all: the system rules, and a user message containing the numbered Sources: followed by the Question:. The reply text is read from msg.content[0].text.

Try this: Grounding lives in two places at once: the prompt tells the model to stay on-source, and retrieval controls which sources it even sees. Weakening either one lets hallucinations back in.

This is the payoff file: given a question, it retrieves the best chunks, then either writes a grounded answer (real mode) or echoes the top chunk (offline mock mode) — and it refuses to answer when retrieval finds nothing. It works with no API key by default.

1. _context(hits) formats the retrieved chunks into a numbered list like [0] (contract_acme.txt p1) Payment terms.... Those numbers are the citation labels the answer will point back to.
2. _mock_llm is the free offline stand-in: it just returns the top chunk's text with [0] tacked on, so you can watch the retrieve→cite flow without paying for a model. _real_llm is the same idea with a real Claude call, and a strict system prompt (covered in the next box).
3. answer(question, store, where) ties it together: it searches, and if not hits it returns the ABSTAIN string immediately — that early return is the safety valve. Otherwise it picks mock vs real based on the USE_REAL_API environment variable and returns both the text and the list of sources (doc_id + page for each hit).
4. find_docs(store, where) answers "which documents match these fields?" — pure metadata filtering, no search — which is how you do cross-document questions like "which contracts auto-renew?".

What the output means: Imported and called by the demo block at the bottom and by the tests. On its own it defines the answer/abstain/analytics functions.

Try this: Find the line if not hits: return {"text": ABSTAIN, "sources": []}. That single check is the difference between an honest "I don't know" and a confident made-up answer over a real contract.

This is the demo actually running — no API key needed. It ingests the two sample files, asks two questions, and lists the contracts. Read the output as proof that the whole pipeline works end to end.

1. The first answer repeats ACME's payment sentence with a trailing [0] — the citation pointing at retrieved chunk 0. The mock model is just echoing the top hit, but the grounding + citation flow is real.
2. The second question ("capital of France") is not in the documents, so the answer is the exact abstain line — search() found no overlapping chunk and answer() returned early.
3. The last line, contracts: ['contract_acme.txt'], is find_docs answering a cross-document question by metadata filter alone — no search involved.

What the output means: Three things: a cited answer from the corpus, an honest abstention for an out-of-corpus question, and a metadata-filtered document list. That trio is exactly what the tests in Step 8 lock in.

Try this: Add a blank line and a new paragraph to samples/contract_acme.txt, re-run, and the extra section becomes its own citable chunk (p3).

These seven tests prove the important behaviours without any API key — they run on the deterministic offline embedder, so they're fast, free, and give the same answer every time. Each test is a tiny, readable statement of one guarantee.

1. _store() is a shared helper that builds a fresh store with two documents — a contract owned by alice and a policy owned by bob — so every test starts from the same known corpus.
2. The first three tests check retrieval and filtering: the right chunk comes back; a type filter returns only the policy; and an owner filter never leaks another user's document — that's the access-control test.
3. test_out_of_corpus_abstains feeds nonsense ("xyzzy quux nothing") and asserts the answer equals ABSTAIN — pinning the honest-refusal behaviour so a future change can't silently break it.
4. The last three assert every hit carries a string doc_id and int page (so answers are always citable), that find_docs returns exactly the contract, and that the answer text actually contains a [0] citation.

What the output means: Run with python -m pytest tests/ -v you should see all seven lines marked PASSED and 7 passed at the end.

Try this: Break one guarantee on purpose — e.g. delete the > 0 check in store.search — and watch test_out_of_corpus_abstains go red. That's the test doing its job: catching a safety regression.