Project 6 · Design Chapter

Deep Research Agent

Q: How does the code make it impossible for the agent to cite a source it never actually fetched?

Every gathered source id is recorded in a gathered set, and a claim is kept only if its source_id is in that set; invented citations are dropped in code. The citation set is controlled by your tools, not the model, so the model can't fabricate a source.

The pattern behind every "deep research" feature of 2026: given a question, the agent plans the sub-questions, searches multiple sources, verifies claims against evidence, and writes a structured, cited report. The most autonomous project here — a showcase of multi-step planning, tool use, and citation discipline.

🎯 Advanced📈 trending🔬 analysts / knowledge workmulti-step + citations

What this project teaches you to design

A plan → search → read → verify → synthesize loop (the research pattern).
Citation discipline: every claim traceable to a source.
Adversarial verification to fight hallucination in long outputs.
Research-specific evals: coverage, citation accuracy, factual support.

The brief advanced

"Do the hours of reading-and-synthesizing that go into a briefing — with sources." Market scans, competitive analysis, literature reviews, due diligence — all involve gathering many sources and distilling them. An agent that plans the research, gathers and cross-checks sources, and produces a cited draft turns a day of work into minutes, with a human editing the final.

1 · Discovery — where does the time go? advanced

Where time goes	Agent leverage
Searching & skimming many sources	⭐⭐⭐ high — parallel search & read
Cross-checking a claim across sources	⭐⭐⭐ high — verification
Structuring findings into a report	⭐⭐⭐ high — synthesis with citations
Judging source quality & final narrative	⭐ low — human editor decides

Problem statement"Producing a briefing means hours of searching, reading, and synthesizing across sources. If an agent planned the sub-questions, gathered and verified evidence, and wrote a cited first draft, an analyst could edit instead of research — cutting turnaround from a day to minutes, with every claim traceable to a source."

2 · Architecture advanced

🗺️ How to read this diagram

This is the whole agent as a pipeline, read left to right. A research question goes in one end; a cited report comes out the other. Every project on this page is just code that walks these boxes in order.

Question → Plan — the question first hits the purple Plan box, which breaks it into sub-questions (smaller, focused angles to research). This is Step 2 in the build.
The three middle boxes — web search, fetch / read, and internal docs (RAG) are the tools the agent uses to gather evidence for each sub-question. They all feed the next box.
Verify (amber box) — every gathered claim is re-checked against the source it came from (claim ↔ source). Unsupported claims are dropped here. This is the anti-hallucination heart of the design.
enough? → loop or stop — the agent asks whether it has covered the question. The dashed blue arrow curving back to the tools is the loop: if there are gaps, it searches again (bounded by a budget).
Cited report — the final purple box writes the report from verified findings only, with a citation on every claim.

In short: Follow the solid arrows once (Question → Plan → tools → Verify → enough? → Cited report), then the dashed arrow shows the one loop back. Plan, gather, verify, synthesize — that's the entire project.

A genuinely agentic loop (Ch 4): the agent plans sub-questions, gathers evidence from web search/fetch and internal docs (RAG), verifies each claim against its source, decides whether it has enough (looping back to search gaps), and finally synthesizes a cited report. This is the most "Tier 4" of the projects.

3 · Risk & quality model advanced

Research agents don't touch infrastructure — the risks are epistemic: fabricated facts, fake citations, and shallow coverage. Quality controls are the safety model.

Risk	Control
🔴 Hallucinated facts in a long report	Every claim must cite a retrieved source; a verify pass re-checks each claim against its evidence and drops unsupported ones
🔴 Fabricated or misattributed citations	Citations come only from sources actually fetched (tracked by the tool), never invented; the writer can only cite from the gathered set
🟠 Shallow / one-sided coverage	Plan diverse sub-questions; a "completeness critic" checks what's missing and triggers more search
🟠 Low-quality / biased sources	Prefer reputable domains; note source type; a human editor makes the final call
🟠 Prompt injection from a fetched page	Web content is untrusted — instructions embedded in a page must not hijack the agent (Ch 6)
💸 Runaway cost (endless searching)	Cap search rounds / token budget; stop when marginal new info is low

The golden rule of research agentsNo claim without a citation, and no citation without verification. A polished report full of confident, unsourced (or wrong-sourced) assertions is dangerous precisely because it looks authoritative. The verify pass is non-negotiable.

4 · Tool surface advanced

Tool	Does	Risk
`web_search`	Find candidate sources for a sub-question	🟢 read-only
`fetch_url`	Retrieve & extract a page's content	🟢 read-only (untrusted content)
`search_internal`	RAG over your own docs/reports (Ch 3)	🟢 read-only
`note_finding`	Record a claim + its source (builds the citation set)	🟢 scratchpad
`verify_claim`	Re-check a claim against its cited source (can be a sub-agent)	🟢 read-only

Server-side vs. your own toolsMany providers offer built-in web search + fetch tools with citations — use those where available (less to build, citations handled). Add your own search_internal (Ch 3 RAG) so the agent can mix public and proprietary sources. The note_finding scratchpad is how the writer later cites only real, gathered evidence.

5 · The research loop — plan, gather, verify, synthesize advanced

This is the richest agent pattern in the course. It composes several techniques:

Plan — decompose the question into sub-questions (structured output: a list of angles to cover).
Gather — for each sub-question, search + fetch (often in parallel), and note_finding with the source.
Verify — for each noted claim, re-check it against its source (an adversarial "is this actually supported?" pass — the Ch 5 idea). Drop unsupported claims.
Assess coverage — a "what's missing?" critic; if there are gaps, loop back and search more (bounded by a budget).
Synthesize — write the report from verified findings only, citing each claim.

🔗 Reuse from the courseThe loop is Ch 4 at its most autonomous. Verification is the Ch 5 "verify each finding" pattern applied inline. Internal-source RAG + citation formatting is Ch 3. Budget caps & untrusted-content handling are Ch 6. Structured planning uses Pydantic (P4).

6 · Evaluation expert

Eval	Measures
Citation accuracy	Does each cited source actually support the claim it's attached to? (sample & check — the key metric)
Factual support / groundedness	What fraction of report claims are backed by a real source? (LLM-judge + spot human check)
Coverage	Does the report address the key sub-questions a good analyst would? (rubric)
No-fabricated-citation (deterministic)	Every citation URL/source was actually fetched by a tool (cross-check the gathered set)
Cost / rounds	Stayed within the search/token budget

Verification is both a runtime step and an evalThe same "does the source support the claim?" check runs inside the agent (to drop bad claims before writing) and as an eval (to score the finished report). Building it once serves both — a neat efficiency.

7 · Phased rollout expert

Phase 1 · Single-pass with citations — search → answer with sources for focused questions. Prove citation discipline. (Ch 4 + built-in search)

Phase 2 · Multi-step + verify — plan sub-questions, verify claims, drop unsupported ones, assemble a short cited brief. (Ch 5 verification)

Phase 3 · Full reports + internal sources — coverage critic, internal-doc RAG, longer structured reports, budget controls. (Ch 6)

Always — a human edits before anything is published or acted on. The agent drafts; it doesn't decide.

Skills & course map expert

Skill	Learn it in
Multi-step agentic loop (plan/gather/synthesize)	Ch 4
Adversarial verification of claims	Ch 5
Citations, internal-source RAG	Ch 3
Structured plan / findings schema	Python P4
Budget caps, untrusted web content, deploy	Ch 6
Streaming a long report to the user	Ch 1 + P4 async

Build-along plan expert

Single-pass first (Ch 4): a web-search + fetch loop that answers a focused question with citations. Prove it never cites a source it didn't fetch.
Add planning (P4 schema): decompose the question into 3–5 sub-questions before searching.
Add verification (Ch 5): for each claim, a check that the cited source supports it; drop the ones that fail.
Add a coverage critic: "what's missing?" → one more search round, bounded by a budget.
Synthesize: write a structured, cited report from verified findings only; stream it out.
Evals (Ch 5): sample claims and check citation accuracy + coverage on a set of test questions.

The most autonomous project — save it for lastIt combines planning, parallel tool use, verification, and synthesis, so it's the natural finale after the others. Want full build labs + runnable code? Ask and I'll build them like the DevOps capstone.

🎓 You've seen all six projectsSix in-demand agents, one shared blueprint, all built from the course you've already worked through. Pick the one that fits your goals — the gallery has a "which first?" guide — and build it. When you're ready to build one for real, ask me for its full build labs + runnable code, and it'll get the same treatment as the DevOps capstone.

🛠️ Hands-on build — everything below is on this pageThe rest of this page is the complete, self-contained build: set up from an empty folder, paste in every file, run it (with a mock, so no API key is needed), and pass the tests. Follow it top to bottom — no other page required.

Learning objectives

Build the plan → gather → verify → synthesize loop.
Enforce citation integrity in code (no invented sources).
Verify each claim against its source before it reaches the report.
Measure citation integrity and coverage with evals.

What you'll build expert

Ask a research question; the agent plans sub-questions, gathers from a source corpus, verifies each claim against its source, and writes a cited report using only verified findings. A human edits the final — the agent drafts.

Finished code includedAll in llm-course-starter/research-agent/. Design rationale in the design chapter.

Step 1 · A mock source corpus expert

Example code for learning — review, test, and adapt it before running against real or production systems. Commands can create, change, or delete resources. See the Terms & Disclaimer.

Step 1

mock/sources.py is a fake 'web' with search() and fetch() — so the whole loop runs offline and deterministically. It includes an irrelevant "cast iron" source, so evals can check the agent doesn't cite off-topic material. Swap in real web_search/web_fetch later — same interface.

Step 2 · Plan the sub-questions expert

Step 2

Uses the client / objects set up earlier in this lesson, and needs ANTHROPIC_API_KEY set. Run the earlier blocks first.

agent/engine.pydef plan(question):
    p = client.messages.parse(model=MODEL, max_tokens=400,
        system="Break the research question into 3-5 focused sub-questions.",
        messages=[{"role":"user","content":question}],
        output_format=Plan)
    return p.parsed_output.sub_questions

▶ How this works

The first real step of the agent: turn one big, vague question into a short list of focused sub-questions. Instead of asking the model to "research X" in one shot, we make it plan first — the same way a good analyst sketches an outline before reading anything.

plan(question) is a normal function that takes the user's research question as its one input.
client.messages.parse(...) is a model call that returns structured data, not free text. output_format=Plan says "give the answer back shaped like our Plan object" (a Pydantic model defined earlier) — so we get a clean list, not a paragraph to parse by hand.
The system line is the instruction: break the question into 3-5 focused sub-questions. The messages line passes the actual question in as the user turn.
return p.parsed_output.sub_questions hands back just the list of sub-questions — the plan the rest of the loop will work through.

What the output means: You get a Python list of 3-5 short strings, e.g. ["How big is the EV market?", "What are battery cost trends?", ...] — one search target per angle.

Try this: Change the system instruction to "2-3 sub-questions" and the plan gets shorter. This one prompt controls how broad the whole research pass will be.

Step 3 · Gather — with citation integrity expert

Step 3

agent/engine.py# track the source_ids we actually gathered
gathered = {}
for sq in sub_questions:
    for hit in sources.search(sq, k=2):
        gathered[hit["id"]] = {"text": sources.fetch(hit["id"]), ...}

# model extracts 'claim | source_id' lines from ONLY these sources
for line in extracted.splitlines():
    claim, _, sid = line.rpartition("|")
    if sid.strip() in gathered:          # REJECT invented citations
        findings.append(Finding(claim=claim.strip(), source_id=sid.strip()))

▶ How this works

This is the step that makes the agent trustworthy: it gathers evidence and, crucially, remembers exactly which sources it actually pulled — so later nothing can cite a source that was never fetched. Read it as two halves: gather, then extract claims safely.

gathered = {} is a dictionary that will hold every source the agent really retrieved, keyed by its id. This set is the agent's "allowed to cite" list.
The nested for loops walk each sub-question, run sources.search(sq, k=2) to get the top 2 hits, and store each hit's fetched text in gathered under its id. Now every source the agent has seen is recorded.
The model then returns lines shaped claim | source_id. line.rpartition("|") splits each line at the last | into the claim text and the source id it points at.
if sid.strip() in gathered: is the guard — a claim is kept only if its source_id is one we truly gathered. If the model invented a citation, sid won't be in gathered and the line is thrown away. Kept lines become Finding objects.

What the output means: findings ends up as a list of Finding(claim, source_id) objects where every source_id is real — invented citations were silently dropped by the if check.

Try this: Imagine the model returns a line ending in | s99 but no source s99 was ever fetched. Trace it: "s99" in gathered is False, so the line is skipped. That single line is the whole anti-hallucination rule.

The anti-hallucination guaranteeA finding can only cite a source_id that's in gathered — the set of sources actually returned by search/fetch. If the model invents a citation, it's dropped in code. The model can't fabricate a source, because the citation set is controlled by your tools, not the model.

Step 4 · Verify each claim expert

Step 4

Uses the client / objects set up earlier in this lesson, and needs ANTHROPIC_API_KEY set. Run the earlier blocks first.

agent/engine.pydef verify(finding, gathered):
    src = gathered.get(finding.source_id)
    v = client.messages.parse(model=MODEL, max_tokens=300,
        system="Does the source text actually support the claim? Be strict.",
        messages=[{"role":"user","content":
            f"Source: {src['text']}\n\nClaim: {finding.claim}\n\nSupported?"}],
        output_format=Verdict)
    return v.parsed_output.supported

verified = [f for f in findings if verify(f, gathered)]   # drop unsupported

▶ How this works

Gathering a claim with a real source id isn't enough — the source might not actually say what the claim says. This step re-reads each source and asks the model, strictly: does this evidence support this claim? Claims that fail are dropped before they ever reach the report.

verify(finding, gathered) takes one finding and the gathered sources. src = gathered.get(finding.source_id) looks up the exact source text the claim points to.
It makes a model call whose system instruction is deliberately harsh — "Be strict." — so borderline or unsupported claims get rejected rather than waved through.
The user message pastes the source text and the claim together and asks "Supported?". output_format=Verdict forces a clean yes/no answer (a supported boolean), not a wishy-washy paragraph.
verified = [f for f in findings if verify(f, gathered)] is a list comprehension that keeps only the findings whose verify call returned true — a one-line filter that drops every unsupported claim.

What the output means: verified is the subset of findings the source text genuinely backs up. Anything the model couldn't confirm against its own cited source is gone.

Try this: Make the system prompt lenient ("if it's roughly related, say supported") and more weak claims survive; make it stricter and fewer do. This dial is the same idea used as an eval in Ch 5.

Verification: runtime step AND evalThe same "does the source support the claim?" check runs inside the agent (to drop bad claims) and as an eval (to score the finished report). Build it once, use it twice. This is the Ch 5 verification pattern applied inline.

Step 5 · Synthesize from verified findings only expert

Step 5

terminalpython agent/engine.py       # needs key

The global EV market grew strongly in 2025: sales reached 17 million
units, up 25% year over year, with China about 60% of the total [s1].
Battery pack costs fell to $110/kWh [s2], and public charging points
passed 5 million globally [s3]...

Sources:
  [s1] https://ex.com/ev-2025
  [s2] https://ex.com/batteries
  [s3] https://ex.com/charging

The report is written from only the verified findings — every sentence traceable to a real, gathered, verified source. The cast-iron source never appears.

▶ How this works

This is the payoff: the finished report the agent writes from the verified findings only. You ran python agent/engine.py and this text is what it printed. Notice every factual sentence carries a bracketed source tag.

Each claim ends with a citation like [s1], [s2], [s3] — a pointer to the exact source that backs it up. No sentence is left unsourced.
The Sources block at the bottom lists what each tag maps to (a real URL), so a human editor can click through and check any claim in seconds.
Because the writer is only allowed to use verified findings, the off-topic "cast iron" source from the mock corpus never appears — the agent didn't pad the report with irrelevant material.

What the output means: A short, cited briefing where every claim is traceable to a gathered, verified source — exactly the "no claim without a citation" golden rule made real.

Try this: Compare this to the risk table earlier on the page: "hallucinated facts" and "fabricated citations" are the top risks, and this output is the proof the code controls them.

6 · Tests & evals expert

Step 6

terminalpython -m pytest tests/ -v      # no key
python evals.py                 # needs key

# tests
test_search_finds_relevant_sources PASSED
test_fetch_returns_text PASSED
test_citation_integrity_rule PASSED
4 passed

# evals
invalid citations: [] (expect [])
cited irrelevant source: False (expect False)
EV sources cited: 3/3
✅ evals passed

▶ How this works

The final step proves the agent behaves — twice. pytest runs fast, offline tests (no API key), and evals.py runs the live checks that need a key. This is how you turn "it seemed to work" into evidence.

The # tests section is pytest output. Each line ending PASSED is one automated check that ran green — e.g. test_citation_integrity_rule confirms only gathered ids can be cited. 4 passed is the summary.
The # evals section is the live-model quality report. invalid citations: [] means zero fabricated sources slipped through (the list is empty, which is what expect [] wants).
cited irrelevant source: False means the agent didn't cite the off-topic cast-iron source. EV sources cited: 3/3 means it grounded the report in all the relevant sources — good coverage.
✅ evals passed at the end is the go/no-go signal: every hard-fail check (no invalid citations, no irrelevant ones) held.

What the output means: Green across the board: the citation-integrity rule holds offline, and with a real model there are no fabricated or off-topic citations and full coverage of the relevant sources.

Try this: Break the if sid in gathered guard from Step 3 on purpose, re-run, and watch invalid citations stop being empty — the eval is designed to catch exactly that regression.

✅ Test/eval cases

Check	Needs key?	Proves
search finds relevant, excludes cast-iron	No	retrieval is on-topic
citation-integrity rule	No	only gathered ids can be cited
no invalid citations (eval)	Yes	no fabricated sources — hard-fail
no irrelevant source cited (eval)	Yes	doesn't pad with off-topic — hard-fail
coverage: cites the EV sources	Yes	the report is actually grounded

Troubleshooting expert

⚠️ Common issues

Symptom	Fix
Report has uncited claims	Strengthen the synthesize prompt ("only these findings, cite each"); ensure you pass verified findings only
Fabricated citation slips through	Confirm the `if sid in gathered` filter runs; the eval hard-fails on invalid ids
Verify drops good claims	Verifier too strict, or the source text is truncated — pass full source text
Endless searching (real web)	Cap search rounds / token budget (Ch 6); stop when little new info appears
Injected instructions from a page	Treat fetched content as untrusted data (Ch 6); it must not change the agent's task

🪜 Practice ladder beginner → industry

Six graded exercises, easy to real-world. Try each before opening its solution.

Exercise 1 · Scaffold: a mock source corpus + fetchBeginner

Context: Offline research needs a corpus with stable ids, because citations only mean something if the ids are real and fetchable. Keeping search dumb (keyword overlap) lets the whole thing run with no model.

Your task: Build an in-memory corpus keyed by id with search(query, k) (keyword overlap) and fetch(source_id) — everything downstream cites these ids.

Requirements:

Sources are a dict keyed by stable ids
search ranks by token overlap and returns ids
fetch returns a source's text by id
No model or network is used
Ids are the anchor every later citation refers to

💡 Hint: Rank by set-overlap of tokens so it needs no embeddings; the ids you return here are what every citation check validates against later.

Show solution

Design. Offline research needs a corpus with stable ids — citations are only meaningful if " "ids are real and fetchable. Keep search dumb (overlap) so it runs with no model.

import re
def toks(s): return set(re.findall(r"[a-z0-9]+", s.lower()))

SOURCES = {
    "s1": "Churn rose to 8 percent in Q3 driven by onboarding friction.",
    "s2": "Net revenue retention held at 105 percent.",
    "s3": "Support wait times doubled after the March release.",
}
def search(query, k=2):
    q = toks(query)
    scored = sorted(SOURCES, key=lambda i: len(q & toks(SOURCES[i])),
                    reverse=True)
    return [{"id": i} for i in scored[:k]]
def fetch(sid): return SOURCES[sid]

print(search("why did churn rise"))   # [{'id': 's1'}, ...]
print(fetch("s1")[:20])

Exercise 2 · Core feature: gather with citation integrityIntermediate

Context: The heart of the agent is gathering findings while rejecting invented citations. The model may cite only from ids you actually handed it — any other id is dropped before it enters the pipeline.

Your task: Write gather that extracts claim | source_id lines and keeps a finding only if its id is one you actually gathered.

Requirements:

Search and fetch populate a gathered set of ids
Model output is parsed into claim / source-id pairs
A finding whose id isn't in the gathered set is dropped
An invented id (not returned by search) is rejected
This is what stops hallucinated sources entering the pipeline

💡 Hint: Split each line on the last |, then keep the finding only if its id is in the gathered dict — reject anything the model made up.

Show solution

Design. The model is allowed to cite only from the ids you handed it. Any line whose id isn't " "in gathered is dropped — this is what stops hallucinated URLs from entering the pipeline.

from dataclasses import dataclass
@dataclass
class Finding:
    claim: str
    source_id: str

SOURCES = {"s1": "Churn rose to 8 percent in Q3.",
           "s2": "NRR held at 105 percent."}

def gather(sub_questions, search, fetch):
    gathered, findings = {}, []
    for sq in sub_questions:
        for hit in search(sq, k=2):
            gathered[hit["id"]] = fetch(hit["id"])
    # model output (mocked): claim | source_id, incl. one INVENTED id (s9)
    extracted = "Churn rose to 8 percent | s1\nMade up fact | s9"
    for line in extracted.splitlines():
        claim, _, sid = line.rpartition("|")
        if sid.strip() in gathered:              # reject invented citations
            findings.append(Finding(claim.strip(), sid.strip()))
    return findings, gathered

def search(q, k=2): return [{"id": "s1"}]
def fetch(i): return SOURCES[i]
fs, _ = gather(["why churn"], search, fetch)
print([(f.claim, f.source_id) for f in fs])   # only the s1 claim survives

Exercise 3 · Harder variant: verify each claim, drop the unsupportedAdvanced

Context: Beyond a real citation, the source must actually support the claim. The real agent asks a strict LLM judge; offline you approximate with a high content-overlap threshold that rejects sounds-plausible-but-unsupported claims.

Your task: Add a verifier: for each finding, check the source text supports the claim (offline: strong token overlap), keeping only verified findings.

Requirements:

A support score is the fraction of the claim's content words present in the source
A high threshold is required to count as supported
Supported findings are kept; unsupported are dropped
A near-miss that shares some words but isn't supported is rejected
Runs entirely offline

💡 Hint: Use content words (length ≥ 3) and require most of the claim's words to appear in the source; a high threshold is what rejects the plausible-but-wrong claim.

Show solution

Design. The real agent asks a strict LLM judge; offline we approximate with a support score: " "the claim's content words must be largely present in the source. A high threshold rejects " "sounds-plausible-but-unsupported claims.

import re
def content(s): return set(re.findall(r"[a-z0-9]{3,}", s.lower()))

def supports(source, claim, thresh=0.8):
    c = content(claim)
    if not c: return False
    return len(c & content(source)) / len(c) >= thresh

def verify_all(findings, gathered):
    return [f for f in findings
            if supports(gathered[f["id"]], f["claim"])]

gathered = {"s1": "Churn rose to 8 percent in Q3 due to onboarding."}
findings = [
    {"claim": "churn rose to 8 percent", "id": "s1"},   # supported
    {"claim": "churn fell to 2 percent", "id": "s1"},   # NOT supported
]
kept = verify_all(findings, gathered)
print([f["claim"] for f in kept])   # ['churn rose to 8 percent']

Exercise 4 · Subtle correctness: synthesize from verified-only, keep citations attachedExpert

Context: Synthesis must be structurally incapable of citing anything unverified. Every sentence carries its source id, and when there are zero verified findings the agent must refuse rather than fabricate.

Your task: Build a synthesizer that renders each verified finding with its id and refuses on an empty verified set, then assert no unverified claim can appear.

Requirements:

Only the verified list is passed into synthesis
Each rendered claim carries its [id]
An empty verified set returns a refusal, not invented text
Assert every claim in the report is cited
Assert the empty case refuses

💡 Hint: Pass only verified findings in and render claim [id]; short-circuit to an "insufficient evidence" string when the list is empty — the anti-hallucination backstop.

Show solution

Design. Pass only the verified list into synthesis and render each as 'claim [id]'. If the " "verified set is empty, the agent must say it cannot answer rather than invent — the anti-hallucination " "backstop.

def synthesize(verified):
    if not verified:
        return "Insufficient verified evidence to answer."   # refuse, don't fabricate
    lines = [f"- {f['claim']} [{f['source_id']}]" for f in verified]
    return "Report:\n" + "\n".join(lines)

verified = [{"claim": "churn rose to 8 percent", "source_id": "s1"}]
report = synthesize(verified)
print(report)
assert "[s1]" in report                       # every claim is cited
assert synthesize([]).startswith("Insufficient")  # empty -> refuse
print("ok")

Exercise 5 · Production concerns: dedupe sources + bound the research loopProfessional

Context: Real gathers return the same source under many sub-questions and can loop forever. Deduping by id plus two independent caps makes the loop bounded and idempotent — safe to retry.

Your task: Dedupe gathered sources by id and cap the loop by both maximum sub-questions and maximum total fetches so cost is bounded.

Requirements:

Gathered sources are keyed by id, so duplicates collapse
A breadth cap limits how many sub-questions are explored
A total-fetch cap bounds overall work
Both caps are respected simultaneously
The result is the same regardless of duplicate hits (idempotent)

💡 Hint: A dict keyed by id gives free dedupe; enforce a sub-question slice and a running fetch counter, returning early when either cap is reached.

Show solution

Design. Gathered is a dict keyed by id -> natural dedupe. Two independent caps (breadth and " "total fetches) bound the loop; both must be respected. Bounded + idempotent = safe to retry.

def research_loop(sub_questions, search, fetch,
                  max_subs=3, max_fetches=5):
    gathered, fetches = {}, 0
    for sq in sub_questions[:max_subs]:            # breadth cap
        for hit in search(sq, k=3):
            if fetches >= max_fetches:             # total-fetch cap
                return gathered
            if hit["id"] not in gathered:          # dedupe by id
                gathered[hit["id"]] = fetch(hit["id"]); fetches += 1
    return gathered

CORPUS = {f"s{i}": f"source {i}" for i in range(10)}
def search(q, k=3): return [{"id": "s1"}, {"id": "s1"}, {"id": "s2"}]
def fetch(i): return CORPUS[i]
g = research_loop(["a", "b", "c", "d"], search, fetch)
print(sorted(g))          # ['s1', 's2']  -- deduped, capped

Exercise 6 · Real-world: structured planning + verification with ClaudeIndustry scenario

Context: Shipping against a real model means structured outputs remove brittle string parsing: plan and verdict become schemas the model must fill, and a missing key falls back to the offline pipeline.

Your task: Use the Anthropic SDK's structured outputs to plan sub-questions and to get strict yes/no verdicts (needs a key), keeping the offline gather/verify pipeline as the fallback.

Requirements:

Plan and Verdict are typed schemas the model fills
The verify judge is deliberately strict (a boolean supported)
Real calls use the Anthropic SDK
A missing key falls back to the offline path
Offline tests still pass with no key

💡 Hint: Wrap the real calls so an absent key routes to the keyword/overlap pipeline; structured outputs (Pydantic schemas) replace regex parsing of the model's text.

Show solution

Design. Structured outputs remove brittle string parsing: Plan and " "Verdict are Pydantic schemas the model must fill. The verify judge is deliberately strict. " "Wrap real calls so a missing key falls back to the offline path.

# --- real structured calls (needs creds / API key) ---
# from pydantic import BaseModel
# import anthropic
# class Plan(BaseModel):    sub_questions: list[str]
# class Verdict(BaseModel): supported: bool
# client = anthropic.Anthropic(); MODEL = "claude-opus-4-8"
# def plan(question):
#     p = client.messages.parse(model=MODEL, max_tokens=400,
#         system="Break into 3-5 focused sub-questions.",
#         messages=[{"role":"user","content":question}], output_format=Plan)
#     return p.parsed_output.sub_questions
# def verify(finding, gathered):
#     src = gathered[finding.source_id]
#     v = client.messages.parse(model=MODEL, max_tokens=300,
#         system="Does the source support the claim? Be strict.",
#         messages=[{"role":"user",
#            "content":f"Source: {src}\nClaim: {finding.claim}\nSupported?"}],
#         output_format=Verdict)
#     return v.parsed_output.supported

# --- offline fallback (runnable) ---
import os
def plan(question):
    if os.environ.get("ANTHROPIC_API_KEY"):
        raise RuntimeError("wire real_plan here")   # never reached offline
    return [f"{question} - detail {i}" for i in range(1, 4)]  # deterministic

print(plan("why did churn rise?"))   # 3 sub-questions, no key needed

✓ Checkpoint — done when…

The agent plans, gathers, verifies, and synthesizes a cited report.
It can't cite a source it didn't gather (tested with no key).
Unsupported claims are dropped by the verify step.
The eval passes: no invalid/irrelevant citations, good coverage.

🎉 All six projects builtYou now have runnable, tested, mock-first code for every project — DevOps, support, code review, document intelligence, data analyst, and deep research — each with a design chapter, a build lab, and a safety/quality model proven by tests. Pick one, point it at real data, and ship it.

📋 Master rubric — grade your research agent

Dimension	Meets the bar	Above the bar (staff-level)
Citation integrity	Every claim in the report is traceable to a real gathered source; invented sources are impossible because citations are validated in code against what was actually retrieved.	Citations point to the specific passage supporting the claim, and a citation that doesn't support its claim is caught by an eval, not just its existence.
Adversarial verification	Each claim is checked against its source before it reaches the report; unsupported claims are dropped or hedged.	Verification is independent of generation (a separate pass with fresh context) so the agent can't rubber-stamp its own hallucination.
Coverage	The plan gathers enough distinct sources to cover the question; coverage is measured, not assumed from a single search.	Coverage rewards source diversity and cross-checking (a claim confirmed by two independent sources), not just source count.
Factual support	Evals measure what fraction of claims are actually supported by their cited sources, not just that citations exist.	Contradictions between sources are surfaced to the reader rather than silently resolved to whichever the model saw last.
Loop control & cost	The plan→gather→verify→synthesize loop is bounded so research terminates within a token/dollar budget instead of wandering.	The agent plans once and gathers in parallel where possible; depth is spent on the hard sub-questions, not uniformly.
Human-editable output	The draft is structured with claims and sources a human editor can check and finalize, not an unsourced wall of prose.	Confidence and open questions are flagged inline so the editor knows exactly where to spend their scrutiny.

Score each row 0 (missing) / 1 (meets) / 2 (above). A passing research build is 9+/12 with citation integrity at 2 — a report with an invented source, or a claim its own cited source doesn't support, is an automatic fail, because in research a confident unsupported claim is the whole failure mode you were hired to prevent.

Knowledge check check yourself

✓ Knowledge check

What is the deep-research agent's 'golden rule', and which step enforces it?

Show answer

No claim without a citation, and no citation without verification. The verify step re-checks each claim against its cited source and drops unsupported ones before synthesis — a polished report of confident, unsourced assertions is dangerous precisely because it looks authoritative.

✓ Knowledge check

How does the code make it impossible for the agent to cite a source it never actually fetched?

Show answer

Every gathered source id is recorded in a gathered set, and a claim is kept only if its source_id is in that set; invented citations are dropped in code. The citation set is controlled by your tools, not the model, so the model can't fabricate a source.

Educational content, provided as-is and without warranty. Code samples are examples — review, test, and adapt them before using in production. See the Terms of Use & Disclaimer. Use at your own risk.

What this project teaches you to design

The brief advanced

1 · Discovery — where does the time go? advanced

2 · Architecture advanced

3 · Risk & quality model advanced

4 · Tool surface advanced

5 · The research loop — plan, gather, verify, synthesize advanced

6 · Evaluation expert

7 · Phased rollout expert

Skills & course map expert

Build-along plan expert

Learning objectives

What you'll build expert

Step 1 · A mock source corpus expert

Step 2 · Plan the sub-questions expert

Step 3 · Gather — with citation integrity expert

Step 4 · Verify each claim expert

Step 5 · Synthesize from verified findings only expert

6 · Tests & evals expert

Troubleshooting expert

🪜 Practice ladder beginner → industry

✓ Checkpoint — done when…

Knowledge check check yourself

More in Projects