Agentic RAG & GraphRAG for Agents
Classic RAG retrieves once, then answers (Chapter 3). Agentic RAG makes retrieval a decision the agent controls — when to search, how to reformulate, whether to search again. GraphRAG changes what you retrieve from: a knowledge graph instead of flat chunks, so the agent can answer connect-the-dots questions vector search can't.
Learning objectives
- Contrast classic RAG with agentic RAG — retrieval as a tool the agent chooses.
- Build a retrieve→grade→rewrite→re-retrieve loop (self-correcting retrieval).
- Explain what GraphRAG is and the class of questions it answers that vector RAG can't.
- Route between vector, graph, and web retrieval as separate tools.
- Know the cost/latency tradeoffs and when classic RAG is still the right call.
Classic RAG vs agentic RAG intermediate
In classic RAG the pipeline is fixed: every query embeds → retrieves top-k → stuffs context → answers. It has no way to notice bad retrieval or to try again. Agentic RAG promotes retrieval to a tool the agent calls on its own judgment — so it can decide whether to retrieve, reformulate a weak query, retrieve from a different source, or loop until the context is good enough.
This picture contrasts the two shapes of RAG. On the left, classic RAG is a straight line; on the right, agentic RAG is a loop. The whole lesson is about turning that line into a loop the agent controls.
- Left — Classic RAG (fixed): just two boxes,
retrieve→answer. It always retrieves once and answers from whatever came back. There is no way for it to notice the retrieval was bad. - Right — Agentic RAG:
retrieve→grade. The grade step is the new idea: the agent judges whether the retrieved docs actually answer the question. - Follow the curved dashed arrow labelled "bad? rewrite query & retry": if the grade is bad, it loops back to
retrievewith a better query. This is the self-correction. - Only when the grade is good does the flow reach the green
answer (grounded)box at the bottom — "grounded" means the answer is backed by docs the agent judged relevant, not guessed.
In short: Classic RAG retrieves once and hopes; agentic RAG retrieves, checks, and retries until the context is good enough. Same input and output — a smarter middle.
Lab M3.1 · Self-correcting retrieval (retrieve → grade → rewrite) intermediate
The signature agentic-RAG pattern: a LangGraph loop that grades relevance and rewrites the query when retrieval is poor. It's the L4 routing graph with a retrieval twist.
Requires: pip install langgraph
agentic_rag.pyfrom langgraph.graph import StateGraph, START, END
from typing import TypedDict
class S(TypedDict):
question: str
query: str
docs: list
tries: int
def retrieve(s):
return {"docs": retriever.invoke(s["query"]), "tries": s["tries"] + 1}
def grade(s) -> str:
"""Ask the model if the docs actually answer the question."""
verdict = model.invoke(
f"Do these docs answer '{s['question']}'? yes/no.\n{s['docs']}").content.lower()
if "yes" in verdict: return "generate"
if s["tries"] >= 3: return "generate" # give up gracefully — bound the loop
return "rewrite"
def rewrite(s):
return {"query": model.invoke(
f"Rewrite this search query to retrieve better results: {s['query']}").content}
g = StateGraph(S)
g.add_node("retrieve", retrieve); g.add_node("rewrite", rewrite); g.add_node("generate", generate)
g.add_edge(START, "retrieve")
g.add_conditional_edges("retrieve", grade, {"generate": "generate", "rewrite": "rewrite"})
g.add_edge("rewrite", "retrieve") # loop back with the improved query
g.add_edge("generate", END)
app = g.compile()
This builds the self-correcting retrieval loop as a small LangGraph state machine. Three little functions do the work — retrieve, grade, rewrite — and the bottom half wires them into a graph that can loop. "Agentic" here means the agent decides whether the retrieval was good enough and whether to try again.
- The
Sclass is the shared state that flows through the graph: the user'squestion, the current searchquery, the retrieveddocs, and atriescounter. Each function reads this state and returns the fields it wants to update. retrieveruns the search for the current query, stores the results indocs, and bumpstriesby 1 so we can count how many attempts we've made.gradeis the heart of it: it asks the model "Do these docs answer the question? yes/no". If the reply contains"yes", it returns"generate"(go write the answer). Otherwise it returns"rewrite"to try again — unless we've already tried 3 times, in which case it gives up and answers anyway so the loop can't run forever.rewriteasks the model to reword the search query into something likely to retrieve better results, and puts that improved query back into the state.- The bottom block wires the graph: start at
retrieve; thenadd_conditional_edgesusesgrade's return value to branch to eithergenerateorrewrite; andrewriteloops back toretrieve.compile()turns this wiring into a runnableapp.
What the output means: Running app on a question retrieves, grades, and — if the docs are weak — rewrites and retries, up to 3 times, before generating a grounded answer. A good question answers on the first pass; a poorly-matched one visibly loops.
Try this: Trace the path for a bad first retrieval: retrieve → grade says "no" → rewrite → retrieve again. Now imagine removing the tries >= 3 check — the loop could spin forever. That one line is why bounding retries matters.
tries keeps the loop from spinning forever (L5) — always give a self-correcting loop an exit.What GraphRAG is intermediate
Vector RAG retrieves independent chunks by similarity. It's great at "find the passage about X" and poor at "how are X and Y connected across many documents?" — because the answer isn't in any single chunk. GraphRAG first builds a knowledge graph (entities as nodes, relationships as edges) from your corpus, then retrieves by traversing that graph, so the agent can follow connections and reason over structure.
This diagram shows what each kind of retrieval hands back. Vector RAG (left) returns a pile of loose chunks; GraphRAG (right) returns entities joined by relationships. That difference is exactly why GraphRAG can answer connect-the-dots questions.
- Left — Vector RAG: four separate little boxes scattered around, captioned "independent, no links". Each box is a text chunk that looked similar to your query. Nothing tells you how they relate — because vector search only knows about similarity, not connections.
- Right — GraphRAG: four labelled circles (
A,B,C,D) are entities — things like people, services, or documents. - The lines between the circles are relationships (edges). Together the circles-and-lines form a knowledge graph. To answer a question the agent traverses these links — walks from one entity to a connected one.
- So a question like "which services depend on the database team X owns?" becomes a walk across the graph, instead of hoping the answer sits inside a single chunk.
In short: Vector RAG = a bag of similar passages. GraphRAG = a map of how things connect. Use the map when the answer lives in the links between facts, not in any one fact.
| Question type | Vector RAG | GraphRAG |
|---|---|---|
| "What does the doc say about refunds?" | ✅ Great — it's in one chunk | Overkill |
| "Summarize the whole corpus's themes" | ✗ No chunk holds the summary | ✅ Community summaries over the graph |
| "How are A and B connected?" | ✗ Relationship spans documents | ✅ Path through the graph |
| "Which X depend on Y?" | ✗ Requires joining facts | ✅ Graph traversal |
Lab M3.2 · Retrieval routing — many sources as tools advanced
A mature agentic-RAG system treats each retrieval method as a tool and lets the agent pick: vector search for passages, graph query for relationships, web search for fresh facts. This is L1's router / L4's conditional edge, applied to where to look.
Requires: pip install langchain-core
router.pyfrom langchain_core.tools import tool
@tool
def vector_search(query: str) -> str:
"""Find specific passages/facts stated in the docs. Use for 'what does X say' questions."""
return vector_retriever.invoke(query)
@tool
def graph_query(entities: str) -> str:
"""Find relationships BETWEEN entities. Use for 'how are X and Y connected' / 'what depends on Z'."""
return graph_store.query(entities)
@tool
def web_search(query: str) -> str:
"""Find current/external facts not in the internal corpus. Use for recent events."""
return web.search(query)
# the agent chooses the right retriever per question (L3 create_react_agent)
agent = create_react_agent(model, tools=[vector_search, graph_query, web_search])
This lab treats each way of retrieving as a separate tool and lets the agent pick the right one per question. The key trick: the agent chooses based on each tool's docstring (the text in triple-quotes), so those descriptions do the real routing work.
- The
@tooldecorator above each function registers it as something the agent is allowed to call. There are three:vector_search,graph_query, andweb_search. vector_search's docstring says "find specific passages... use for 'what does X say' questions" — that steers plain lookup questions here. It runs the similarity search you built in Chapter 3.graph_query's docstring says "find relationships BETWEEN entities... 'how are X and Y connected'" — that steers relational questions to the knowledge graph (the GraphRAG idea from the diagram above).web_search's docstring says "current/external facts not in the internal corpus" — that steers recent-events questions to the live web.- The last line,
create_react_agent(model, tools=[...]), hands all three tools to one agent. When you ask a question, the agent reads the docstrings and calls whichever tool best fits — noif/elserouting code needed.
What the output means: Ask a passage question and the agent calls vector_search; ask "how are A and B connected" and it calls graph_query; ask about a recent event and it calls web_search — each chosen purely from the descriptions.
Try this: Rewrite graph_query's docstring to be vague (e.g. just "query the graph") and picture what happens: the agent can no longer tell when to use it, so relational questions may wrongly go to vector search. The docstrings are the router.
Grounding & quality still rule everything advanced
Agentic and graph retrieval add power, not a free pass on the fundamentals. Everything from Chapter 3 and Topic T1 still governs whether the answer is trustworthy:
| Concern | Still applies because… |
|---|---|
| Grounding / citations | The agent must answer from retrieved context, not memory — cite sources (Chapter 3) |
| Chunking & rerank quality | A smart loop over bad chunks still returns bad answers (A7) |
| Prompt injection via docs | Retrieved content is untrusted — graph or vector, it can carry instructions (T1) |
| Cost/latency of the loop | Each retry is more calls and tokens — bound the loop and prefer classic RAG when it suffices |
Common pitfalls expert
| Pitfall | Fix |
|---|---|
| Self-correcting loop with no cap | Bound retries; answer gracefully when you give up |
| Reaching for GraphRAG for lookup questions | Use vector RAG — GraphRAG's cost only pays off on relational/global queries |
| Vague retriever tool descriptions | Say which question type each answers so routing works |
| Assuming agentic RAG fixes bad chunks | Fix chunking/rerank first (A7) — the loop can't rescue garbage |
| Trusting retrieved text | Treat all retrieval as untrusted; guard against injection (T1) |
| Stale knowledge graph | Rebuild/update the graph as the corpus changes |
Exercises expert
Exercise M3.1 — Grade-and-rewrite loop
Context: The retrieve → grade → rewrite loop only proves itself when you feed it a question your corpus answers poorly — that is when the rewrite-and-retry and the retry cap both have to work.
Your task: Build the Lab M3.1 grade-and-rewrite graph over your Chapter 3 corpus, ask a question your chunks answer poorly, and watch it rewrite the query and retry.
Requirements:
- Run against your own Chapter 3 chunks, not a toy corpus
- Choose a question the chunks answer poorly so a first retrieval fails the grade
- Observe the query being rewritten and retrieval retried
- Confirm it stops after the retry cap even when no good answer is found
💡 Hint: The interesting case is the ungroundable question: the cap, not a good answer, is what must end the loop.
Exercise M3.2 — Route three sources
Context: Retrieval routing succeeds or fails on how well your tool descriptions distinguish question types. The trace tells you whether each question actually reached the right retriever.
Your task: Build the Lab M3.2 router with vector, mock-graph, and web tools, then ask a passage lookup, a "how are X and Y related" question, and a current-events question and verify from the trace that each was routed correctly.
Requirements:
- Wire up three retrieval tools: vector, a mock graph, and web
- Ask one passage lookup, one relational question, and one current-events question
- Read the trace to confirm each query hit the intended retriever
- If routing is wrong, tighten the tool descriptions until it is correct
💡 Hint: Misroutes (a lookup going to graph, a relational question going to vector) mean the tool descriptions aren't distinguishing the question types clearly enough.
Show what to look for
If the lookup goes to the graph tool or the relational question goes to vector search, your tool descriptions aren't distinguishing the question types clearly enough. Tighten them until routing is correct — that's the whole skill.
Exercise M3.3 — Vector vs graph, same question
Context: One genuinely relational question exposes the gap between semantic similarity and relationship traversal. Reasoning through it is more instructive than building a full graph.
Your task: Take a relational question such as "which components depend on the auth service?", answer it with plain vector RAG and explain why it struggles, then describe how a graph traversal would answer it.
Requirements:
- Pick a genuinely relational, multi-hop question
- Attempt it with plain vector RAG and articulate why similarity search falls short
- Describe how a graph traversal would compose the answer instead
- The written reasoning is the deliverable — no full graph build required
💡 Hint: Vector search finds text that looks similar; it cannot compose facts across edges, which is exactly what the relational question demands.
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: Classic RAG retrieves once and then answers; agentic RAG turns retrieval into a decision the model controls — when to search, whether to re-search, or whether to skip retrieval entirely. Telling the two apart from a flow's shape is the first skill.
Your task: Write rag_kind(flow) that inspects a list of pipeline steps and labels the flow as classic or agentic RAG.
Requirements:
- Treat the input as an ordered list of step names
- Classify as agentic when the flow can loop or choose retrieval (grade, re-retrieve, or a decide/route step)
- Otherwise classify as classic: retrieve once, then answer
- Return a human-readable label that states why (decision vs fixed step)
- Demonstrate both a classic and an agentic flow
💡 Hint: The dividing line is control shape: a fixed retrieve-then-answer sequence is classic; anything that grades, re-retrieves, or routes is agentic.
Show solution
The dividing line is whether retrieval is a fixed step or a decision. Runnable:
def rag_kind(flow):
# flow is a list of steps; agentic if the agent can loop/choose retrieval
has_loop = any(s in flow for s in ("grade", "decide_search", "re_retrieve"))
chooses = "route" in flow
if has_loop or chooses:
return "agentic RAG (retrieval is a decision the agent controls)"
return "classic RAG (retrieve once, then answer)"
print(rag_kind(["retrieve", "answer"]))
print(rag_kind(["decide_search", "retrieve", "grade", "re_retrieve", "answer"]))Classic RAG always retrieves once; agentic RAG lets the model decide when to search, re-search, or skip retrieval entirely.
Context: Self-correcting retrieval grades whether the retrieved chunk actually answers the query; if it doesn't, the agent rewrites the query and tries again. The bound on retries is what keeps this from looping forever on an ungroundable question.
Your task: Model the retrieve → grade → rewrite loop offline with a fake retriever, a grader, and a rewriter, capped at a maximum number of tries.
Requirements:
- A fake retriever returns a chunk for known queries and empty for the rest
- A grader decides whether the chunk supports the query (term overlap is enough)
- On a failed grade, rewrite the query and retry rather than answering
- Bound the loop with a
max_triescap - When the cap is hit with no grounded chunk, give up gracefully (the "I don't know" path)
- Show a hit, a rewrite-then-hit, and an ungroundable give-up
💡 Hint: This is the reflection pattern applied to retrieval; the retry bound is the correctness detail, not an afterthought.
Show solution
The core agentic-RAG loop, bounded to avoid infinite retries. Runnable:
def fake_retrieve(query):
corpus = {
"refund window": "Refunds are accepted within 30 days.",
"reset password": "Reset it in Settings -> Security.",
}
for key, text in corpus.items():
if any(w in query.lower() for w in key.split()):
return text
return "" # nothing relevant found
def grade(query, chunk):
return bool(chunk) and any(w in chunk.lower() for w in query.lower().split())
def rewrite(query):
return query.replace("give me back money", "refund window") # clarify intent
def agentic_rag(query, max_tries=3):
for attempt in range(1, max_tries + 1):
chunk = fake_retrieve(query)
if grade(query, chunk):
return f"answer from: '{chunk}' (attempt {attempt})"
query = rewrite(query) # self-correct and retry
return "give up: no grounded answer (would say 'I don't know')"
print(agentic_rag("what is the refund window"))
print(agentic_rag("give me back money")) # rewritten -> refund window
print(agentic_rag("what is the weather")) # ungroundable -> give upGrading + rewriting is the reflection pattern applied to retrieval; the max_tries bound is the correctness detail that keeps it from looping forever.
Context: Agentic RAG treats vector, graph, and web retrieval as separate tools and routes between them by what the question needs. Because each source has a different cost/latency profile, the router should reach for the cheapest one that can answer.
Your task: Write route_retrieval(query) that picks a retrieval source from the shape of the question and explains the tradeoff behind the choice.
Requirements:
- Route relationship / multi-hop questions to the graph source
- Route freshness / open-domain questions (latest, today, current) to the web source
- Default everything else to vector search over internal docs
- Return both the chosen source and a short cost/latency rationale
- Exercise the router on a policy lookup, a relational question, and a current-events question
💡 Hint: Vector is the cheap default; graph earns its higher latency only on relationships; web is the slowest and costs an external call, so reserve it for freshness.
Show solution
Route on what the question needs; prefer the cheapest source that can answer. Runnable:
def route_retrieval(query):
q = query.lower()
# graph: relationship / multi-hop questions
if any(w in q for w in ("related to", "connected", "who reports to", "path from")):
return ("graph", "multi-hop relationships; higher latency, precise")
# web: fresh / open-domain
if any(w in q for w in ("latest", "today", "current price", "news")):
return ("web", "fresh info; slowest + external cost")
# vector: default semantic lookup over our docs
return ("vector", "semantic lookup over internal docs; cheapest, fastest")
for q in ["what is our refund policy",
"who reports to the VP of Sales",
"latest price of the token"]:
src, why = route_retrieval(q)
print(f"{src:>6} <- {q} ({why})")Each source is a tool with a cost/latency profile; the router's job is to reach for the cheapest one that can actually answer — vector by default, graph for relationships, web only when freshness demands it.
Context: GraphRAG answers the class of questions plain vector search cannot: multi-hop relationship queries that require composing facts rather than matching text. Modelling a tiny knowledge graph and walking it makes the difference concrete.
Your task: Represent facts as (subject, relation, object) edges and write a traversal that follows one relation transitively to answer a 2-hop question.
Requirements:
- Store facts as relation-labelled edges in an adjacency structure
- Follow a single relation type transitively (e.g. a management chain)
- Bound the walk with a
max_hopslimit and avoid revisiting nodes - Use only the standard library
- Return the multi-hop chain and note that vector search cannot assemble it
💡 Hint: Build an adjacency map keyed by subject, then step along the requested relation one hop at a time; the chain you accumulate is the answer vector search can't compose.
Show solution
Represent facts as edges; answer by walking hops. Runnable (stdlib):
from collections import defaultdict, deque
EDGES = [
("Ada", "reports_to", "Grace"),
("Grace", "reports_to", "Linus"),
("Ada", "works_on", "Search"),
]
graph = defaultdict(list)
for s, rel, o in EDGES:
graph[s].append((rel, o))
def hops(start, rel, max_hops=3):
# follow `rel` edges transitively (e.g. the management chain)
chain, cur, seen = [], start, set()
for _ in range(max_hops):
nxt = [o for r, o in graph[cur] if r == rel and o not in seen]
if not nxt:
break
cur = nxt[0]; seen.add(cur); chain.append(cur)
return chain
print("Ada's management chain:", hops("Ada", "reports_to"))
# ['Grace', 'Linus'] -- a 2-hop answer vector search cannot assembleVector search finds semantically similar chunks but can't compose facts; GraphRAG traverses explicit relationships, which is exactly the class of multi-hop questions it wins.
Context: However clever the retrieval, grounding is non-negotiable: cite the sources used, refuse when retrieval comes back empty, and flag answers the context doesn't actually support. This guard runs after generation and before anything is returned.
Your task: Write a guard(answer, retrieved) that enforces the citation + refusal contract on a generated answer against its retrieved context.
Requirements:
- Refuse outright when the retrieved list is empty
- Compare the answer's content words against the context's, ignoring stopwords
- Flag a possible hallucination when overlap falls below a minimum threshold
- On success, return an OK result that carries the cited sources
- Demonstrate the grounded, empty-retrieval, and unsupported-answer cases
💡 Hint: Strip stopwords before comparing so the overlap check reflects real content; empty retrieval short-circuits to a refusal before any support check runs.
Show solution
The guard runs after generation and before returning. Runnable:
STOP = {"is", "are", "the", "a", "an", "of", "in", "to", "and", "our"}
def guard(answer, retrieved, min_overlap=1):
if not retrieved:
return "REFUSE: no sources -> 'I don't have that information.'"
# crude support check on CONTENT words: answer should share terms with context
ctx_terms = set(w for c in retrieved for w in c.lower().split()) - STOP
ans_terms = set(answer.lower().split()) - STOP
overlap = len(ctx_terms & ans_terms)
if overlap < min_overlap:
return "FLAG: answer not supported by context -> possible hallucination"
cites = [c[:30] for c in retrieved]
return f"OK (cite: {cites})"
print(guard("Refunds within 30 days.", ["Refunds are accepted within 30 days."]))
print(guard("The moon is made of cheese.", [])) # refuse
print(guard("Our CEO is Bob.", ["Refund policy is 30 days."])) # unsupported -> FLAGAgentic RAG adds cleverness on top, but grounding is non-negotiable: cite what you used, refuse when you have nothing, and flag answers the context doesn't support.
Context: A production agentic-RAG service chains the pieces together — route, retrieve, grade, rewrite-and-retry or answer, then guard — all bounded and logged for observability. In practice this is a graph; offline you can model the exact same control flow.
Your task: Assemble an offline orchestrate(query) that routes, retrieves, grades, retries within a bound, and returns an answer with citations and a trace — then sketch the equivalent LangGraph shape marked as needing the SDK.
Requirements:
- Run bounded attempts: retrieve, check grounding, and either answer or rewrite and retry
- On a grounded hit, return the answer plus its citations
- Record a per-attempt trace (source and hit/miss) for observability
- On exhausting the bound, return the fixed "I don't have that information" refusal
- Include a commented LangGraph
StateGraphsketch, clearly labelled as needing the SDK - Note that the SDK only adds durable state and streaming on top of this flow
💡 Hint: The offline model is the whole control flow; keep the loop bounded and append to a trace each attempt so the routing and retries are inspectable.
Show solution
The full orchestrator, offline and runnable:
def orchestrate(query, max_tries=3):
log = []
def retrieve(q):
docs = {"refund": "Refunds within 30 days.", "reset": "Reset in Settings."}
return [v for k, v in docs.items() if k in q.lower()]
for attempt in range(1, max_tries + 1):
source = "vector" if "policy" not in query else "vector"
chunks = retrieve(query)
grounded = bool(chunks)
log.append((attempt, source, "hit" if grounded else "miss"))
if grounded:
return {"answer": chunks[0], "cites": chunks, "trace": log}
query = query + " refund" # rewrite/expand and retry
return {"answer": "I don't have that information.", "cites": [], "trace": log}
import json
print(json.dumps(orchestrate("what is the money-back rule"), indent=2))# needs the SDK -- the same flow as a LangGraph graph (documented API):
# from langgraph.graph import StateGraph, END
# g = StateGraph(dict)
# g.add_node("route", route_fn); g.add_node("retrieve", retrieve_fn)
# g.add_node("grade", grade_fn); g.add_node("answer", answer_fn)
# g.add_conditional_edges("grade", lambda s: "answer" if s["ok"] else "retrieve")
# g.set_entry_point("route"); app = g.compile()The offline model captures the whole control flow — route, retrieve, grade, bounded retry, guard, and a trace — which is exactly what the SDK graph encodes; the SDK just adds durable state and streaming.
✓ Checkpoint — you can move on when you can…
- Explain how agentic RAG differs from classic RAG.
- Build a retrieve→grade→rewrite loop with a bounded number of retries.
- Say what GraphRAG is and which questions it answers that vector RAG can't.
- Route between vector, graph, and web retrieval as tools.
- Name the cost, grounding, and injection tradeoffs — and when classic RAG wins.
Knowledge check check yourself
How does agentic RAG differ from classic RAG, and what role does the "grade" node play in the retrieve→grade→rewrite loop?
Show answer
What class of question does GraphRAG answer that vector RAG can't, and what upfront cost does GraphRAG carry?