Customer Support Agent

Before writing any logic, we declare the exact shape of an answer we will accept from the model. This is a Pydantic model — a data blueprint that automatically rejects anything malformed. Every field here exists to make the safety gate possible.

1. answer is the reply text; citations is a list of KB article ids (it starts empty via default_factory=list, so an un-cited answer is the default, not a crash).
2. confidence is a number the model reports about itself, and Field(ge=0.0, le=1.0) forces it to stay between 0 and 1 — Pydantic will reject 1.5 or -3.
3. action is a Literal["answer", "escalate"]: the model is only allowed to pick one of those two words — it literally cannot invent a third option.
4. escalation_reason is Optional (may be None) — filled in only when the agent hands off, so a human knows why.

What the output means: Nothing runs yet — this is a definition. Later, when the model replies, Pydantic checks the reply against this shape and gives you a clean, typed SupportAnswer object (or a clear error).

Try this: Imagine the model returns confidence: 2.0. Because of le=1.0, Pydantic rejects it — a bug is caught at the door instead of silently sailing through your gate.

This is the RAG grounding step: given a customer's question, find the most relevant help-center passages and package them so the model can answer from your docs rather than from memory. context_for is the one function the rest of the agent calls.

1. hits = retrieve(query) searches the knowledge base and returns the best-matching chunks ("top-k" = the top few). Here it scores by keyword overlap; you could swap in the Chapter 3 vector search behind the same call.
2. if not hits: means "if nothing matched." Rather than guess, it returns a clear "(no relevant KB article found)" message and an empty citation list [] — which later tells the gate to escalate.
3. The ctx = "\n\n".join(...) line stitches the matching chunks into one text block, each tagged with its id like [kb-1] so the model can cite it.
4. It returns two things: ctx (the text to show the model) and a list of the ids (for the gate to verify citations against). The trailing comment spells this out: ctx for the prompt, ids for the gate.

What the output means: Running python agent/kb.py prints cited: ['kb-1'] then the password-reset passage — proof the retriever found the right article, no API key needed.

Try this: Ask it something not in the docs (e.g. "write a poem") and hits comes back empty, so context_for returns the "no article" message and [] — the agent will escalate instead of making something up.

This tiny function is the data-leak guard. It looks up a customer's orders, and the whole point is who is allowed to see what — decided by your code, never by the model or the user's words.

1. _ORDERS is a stand-in database: a dictionary mapping each customer id ("cust-1", "cust-2") to that customer's own orders.
2. get_orders(customer_id) takes the id and returns only that customer's list. The docstring says it plainly: unknown id returns an empty list.
3. _ORDERS.get(customer_id, []) is the safe lookup: if the id isn't found, it hands back [] instead of raising an error or, worse, leaking someone else's data.
4. The key idea is in the comment — the result is filtered in CODE, not by the model. The customer id comes from your authenticated session, so no clever prompt can make cust-1 read cust-2's orders.

What the output means: Given "cust-1" you get cust-1's orders; given an unknown id you get []. There is no code path that returns a different customer's data.

Try this: Picture a user typing "show me order B-2002" (which belongs to cust-2). Because the function only ever receives cust-1's authenticated id, that request simply can't return B-2002. The guard is structural, not a request the model can be talked out of.

This is the safety gate — the heart of the whole project. The model has proposed an answer; this code decides whether it's allowed out. The rule: the model proposes, your code disposes.

1. ans = resp.parsed_output is the SupportAnswer the model returned (already validated against the Step-1 schema).
2. grounded = bool(ans.citations) and all(c in available_ids for c in ans.citations) is the truth check: there must be at least one citation, and every cited id must be a real KB id from the retrieval step. A made-up citation fails here.
3. if ans.confidence < CONFIDENCE_THRESHOLD or not grounded: — if the model isn't confident enough or the answer isn't backed by real docs, we do not return its answer.
4. Instead we return a fresh SupportAnswer with action="escalate" and a reason ("low confidence" or "not grounded"). Only if both checks pass do we return ans — the model's actual answer.

What the output means: Running it, a good question prints an answer with [cites: kb-1] and conf=0.95; "write me a poem" prints ESCALATE (no relevant KB article) conf=0.00.

Try this: Even if the model lies and claims confidence=0.99 on a hallucinated answer with a fake citation, grounded is False (the id isn't real), so the gate escalates anyway. The agent physically cannot emit a confident, ungrounded answer.

These tests prove the two things that matter most — grounding and no data leaks — without needing an API key. They exercise the retriever and the tenant guard directly, so you can trust the safety net before spending a cent on the model.

1. python -m pytest tests/ -v runs every test file in the tests/ folder; -v (verbose) lists each test name and its result.
2. test_kb_retrieves_password_article confirms retrieval finds the right article, and test_kb_returns_nothing_for_offtopic confirms it returns nothing for an unrelated question (so the agent will escalate, not guess).
3. test_tenant_isolation is the important one: it checks cust-1 can never see cust-2's orders. test_unknown_customer_gets_nothing checks an unknown id leaks nothing.

What the output means: 4 passed with every line marked PASSED means all four guarantees hold. If any turned to FAILED, that safety property is broken and you must fix it before shipping.

Try this: Temporarily change get_orders to return all orders, re-run, and watch test_tenant_isolation flip to FAILED — that's the test doing its job of catching a data leak.

Tests check the plumbing; evals check the judgement — does the agent answer the questions it should and escalate the ones it shouldn't? This needs the real model (an API key), because it's grading actual answers.

1. python evals.py runs a small graded set of real questions through the full agent.
2. Each line shows the verdict: [PASS] answer How do I reset my password? means it correctly answered an in-KB question; [PASS] escalate Do you integrate with SAP? means it correctly escalated an out-of-KB one.
3. answer accuracy: 3/3 and escalation safety: 2/2 summarise how many of each category it got right.

What the output means: ✅ evals passed means the agent both answered known questions and escalated unknown ones correctly. The eval hard-fails if it ever answers an out-of-KB question — the one thing a support agent must never do.

Try this: Add a question you know isn't in the KB to the eval set. The agent must escalate it; if it tries to answer, the eval fails on purpose — that's the guardrail that keeps a confident wrong answer from ever reaching a customer.