Evaluation and Tracing with LangSmith
O4 taught you to trace and evaluate agents by hand. LangSmith is the productized version: a platform that captures every step of every run, turns them into datasets, runs evals against them, and lets you compare versions — closing the LLMOps loop with a real tool instead of print statements.
- an Anthropic API key (
ANTHROPIC_API_KEY) +pip install anthropic
Learning objectives
- Explain what LangSmith adds over hand-rolled tracing (O4) and evals (Ch 5).
- Instrument an app to send traces, framework or not.
- Turn real traces into datasets and run offline evaluations.
- Use LLM-as-judge and custom evaluators inside the platform.
- Compare prompt/model versions and wire an eval gate into CI (O3).
Why a platform beats print statements advanced
Your O4 span() helper works — until you have thousands of runs, want to search them, compare two prompt versions across a hundred cases, or let a non-engineer inspect a failure. Hand-rolled logging doesn't scale to those. A tracing/eval platform does: it stores runs durably, makes them searchable and visual, and connects tracing to evaluation so the two reinforce each other.
| Job (from O4/Ch 5) | By hand | With LangSmith |
|---|---|---|
| See one run's steps | Read log lines, reconstruct the tree | Visual trace tree, timed & costed per step |
| Search runs | grep, if you logged the right fields | Filter/search over all runs |
| Build a golden set | Copy inputs into a file manually | Promote real traces into a dataset in a click |
| Run evals | Write a harness (Ch 5) | Run evaluators over a dataset; store scores |
| Compare versions | Eyeball two outputs | Side-by-side diff across the whole dataset |
Lab I4.1 · Tracing an app advanced
The foundation is capturing runs. LangSmith auto-instruments LangChain/LangGraph, and offers a decorator/wrapper for arbitrary code — so even your raw-SDK apps (C2) get traced.
shellpip install langsmith
export LANGSMITH_API_KEY=... # plus LANGSMITH_TRACING=true
trace.pyfrom langsmith import traceable
from anthropic import Anthropic
client = Anthropic()
@traceable # captures inputs, outputs, timing, errors
def answer(question: str) -> str:
r = client.messages.create(
model="claude-opus-4-8", max_tokens=512,
messages=[{"role":"user","content": question}],
)
return next(b.text for b in r.content if b.type == "text")
answer("What is prompt caching?") # run appears in LangSmith, fully traced
This is the smallest possible traced app. "Tracing" means recording what your code did on each run — the inputs it got, the output it produced, how long it took, and any error — so you can inspect it later instead of guessing. Here a single decorator turns an ordinary function that calls the model into one whose every run is captured by LangSmith.
- The two
importlines bring intraceable(LangSmith's recorder) andAnthropic(the client that talks to the model).client = Anthropic()builds the client once so the function can reuse it. @traceableis a decorator — a line placed directly above a function that wraps it with extra behaviour. This one records a span (one timed step) every timeanswer()runs: what question came in, what text went out, how many milliseconds it took, and whether it crashed. You did not have to write any logging.- Inside,
client.messages.create(...)is the actual model call — it names the model, caps the reply length withmax_tokens, and passes the user'squestionas one message. - The
return next(b.text ... )line pulls the first text block out of the reply. (The model can return several blocks; this grabs the text one.) That returned string is what LangSmith records as this span's output. - The last line,
answer("What is prompt caching?"), simply calls the function once. Because it is decorated, that single call now shows up as a full trace in the LangSmith UI.
What the output means: Nothing dramatic prints locally — the payoff is remote: a new run appears in LangSmith showing the input question, the model's answer, the timing, and the token cost, all captured automatically.
Try this: Add a second @traceable function (say a fake retrieve() step) and call it from inside answer(). LangSmith will nest the two into a tree — parent answer with retrieve as a child — which is the whole trace-tree idea from O4, built for you.
@traceable nests — that's the whole trace treeDecorate your model call, your retriever, and each tool, and LangSmith assembles the nested tree from O4 automatically: which step ran, how long, what it cost, what errored. For a LangChain/LangGraph app you often get this with zero code changes — just the env vars. It's your hand-rolled span(), but it builds the tree for you and stores it durably.Lab I4.2 · From traces to datasets advanced
The platform's superpower is the loop between the two halves: real traces become your evaluation dataset. When a run is interesting — a great answer, or a production failure — you promote it into a dataset. Over time your golden set (Ch 5) is built from real usage, not guesses.
This diagram shows the flywheel at the heart of LangSmith: real runs feed a dataset, the dataset feeds evaluations, and the failures those evaluations find get promoted back into the dataset — so the system keeps getting harder to break.
- Read it left to right. The first green box,
prod traces, is the real runs your live app produced (captured by the tracing from Lab I4.1). - The arrow into
dataset(the "golden set") means you promote interesting runs — great answers, or bad ones — into a saved collection of test cases. This is your evaluation data, built from real usage instead of guesswork. - The next arrow into
evaluation("scored") means the evaluators from Lab I4.3 run over that dataset and produce scores. - The curved dashed arrow looping back from evaluation to the first box is the key part: when an eval surfaces a failure, you add that case to the dataset. The red caption spells it out — failures found → promoted to the dataset → can't regress again.
In short: It is a loop, not a line: trace → collect → evaluate → feed failures back. Every bug you find becomes a permanent test, so the same mistake can never sneak back in.
Lab I4.3 · Running evaluators expert
With a dataset, you run evaluators over it — the same eval types from Chapter 5, executed and scored by the platform.
Requires: pip install langsmith
evaluate.pyfrom langsmith import evaluate
def exact_match(run, example): # deterministic evaluator
return {"key":"correct",
"score": run.outputs["output"] == example.outputs["expected"]}
def grounded_judge(run, example): # LLM-as-judge evaluator (Ch 5)
verdict = judge_model_says_grounded(run.outputs, example.inputs)
return {"key":"grounded", "score": verdict}
evaluate(
my_app, # the function/chain under test
data="my-golden-set", # the dataset
evaluators=[exact_match, grounded_judge],
) # scores stored & visualized per example
Once you have a dataset (a saved list of example inputs plus their expected answers), an evaluator is a small function that scores how well your app did on each example. This block defines two evaluators and then runs them across the whole dataset with one call to evaluate().
exact_match(run, example)is a deterministic evaluator — no AI, just a fixed rule. Every evaluator receives therun(what your app actually produced) and theexample(the saved input + expected answer). It comparesrun.outputs["output"]toexample.outputs["expected"]and returns a score ofTrue/False.- The returned dict —
{"key": "correct", "score": ...}— is the required shape:keynames the metric so LangSmith can chart it, andscoreis the value. grounded_judge(run, example)is an LLM-as-judge evaluator: instead of an exact rule it asks another model whether the answer was grounded in the input. That is the Chapter-5 idea — use a model to grade subjective quality — plugged into the same score-dict interface.evaluate(my_app, data="my-golden-set", evaluators=[...])ties it together: runmy_appon every example in the dataset namedmy-golden-set, then apply both evaluators to each result. LangSmith stores and visualises the scores.
What the output means: For each example you get two scores — a true/false correct and a grounded verdict — aggregated in the LangSmith UI so you can see, at a glance, what fraction of your golden set the current version passes.
Try this: Add a third evaluator that checks a latency budget — return {"key":"fast","score": run.latency < 2.0} — and pass it in the evaluators list. Any Python rule you can write becomes a tracked metric.
| Evaluator type | What it checks | From… |
|---|---|---|
| Deterministic | Exact match, regex, JSON-schema validity, contains-X | Ch 5 deterministic evals |
| LLM-as-judge | Groundedness, helpfulness, tone, rubric adherence | Ch 5 LLM-judge |
| Custom | Any Python: latency budget, cost cap, tool-call correctness | Your logic |
| Human | Manual annotation queues for the subjective calls | Human review (O4) |
Comparing versions & the eval gate expert
The payoff of storing evals: compare two versions of your app across the whole dataset. Change a prompt or swap a model, re-run the eval, and see per-example whether you improved or regressed — the safe-rollout discipline from O3, with data instead of vibes.
This diagram shows how you decide, with data, whether a change is safe to ship. You run two versions of your app over the same dataset and let the scores — not a gut feeling — decide.
- On the left are two boxes,
version A(today's app) andversion B(your change — a new prompt or a swapped model). Both are run against the same golden set. - The arrow leads to
compare("per example"): LangSmith lines the two runs up example by example, not just as one average, so you can see exactly which cases got better and which got worse. - The final arrow reaches
gate— the go/no-go decision. The caption states the rule: ship B only if it doesn't regress the dataset. - Why per-example matters: a version can raise the average score while quietly breaking a few important cases. The side-by-side comparison catches that; a single average would hide it.
In short: This is the O3 "eval gate": compare A vs B on the same data, then let the score gate the deploy. In CI, a regression fails the build automatically — no prompt change ships on vibes.
evaluate() call can run in your pipeline: on every prompt/model change, run the golden-set eval and fail the build if scores drop below threshold. That's the O3 eval gate made concrete — no prompt change ships without clearing the dataset. LangSmith stores the history so you can see quality trend over releases (the O4 continuous-eval idea).Online monitoring expert
Beyond offline evals, LangSmith runs on live traffic too: dashboards over production traces (the four signals from O4 — quality via sampled judges, cost, latency, errors), and alerts. This is O4's monitoring pillar with a UI, closing the loop from "trace one run" to "watch the whole system."
Not the only tool expert
LangSmith is popular and tightly integrated with LangChain/LangGraph, but the capabilities — tracing, datasets, evaluators, version comparison, monitoring — are the transferable knowledge. Other observability/eval platforms and open-source options (including OpenTelemetry-based tracing) offer the same jobs. Pick by your stack, data-residency needs (self-host vs hosted), and budget, exactly as you chose stack layers in O2.
Common pitfalls expert
| Pitfall | Fix |
|---|---|
| Tracing with no data-handling plan | Redact PII/secrets; control access; check residency (O4, T1) |
| Building a golden set from imagination | Promote real traces (incl. failures) into the dataset |
| Trusting the LLM judge blindly | Judges are fallible (Ch 5); spot-check & calibrate |
| Evals only offline | Also monitor live traffic for drift (O4) |
| Prompt changes with no version comparison | Compare across the dataset before shipping (O3) |
| Only tracing the top-level call | Decorate steps/tools so you get the full tree |
Exercises expert
Exercise I4.1 — Trace an agent
Context: The payoff of a tracing platform is seeing exactly where an agent run failed and what it cost — the O4 exercise, now inside a real tool.
Your task: Instrument an agent from an earlier lesson with @traceable (or LangChain auto-tracing) on the model call and each tool, then run a query that triggers a tool error.
Requirements:
- Trace the model call and every tool as separate spans
- Run a query that deliberately makes a tool fail
- Confirm the trace tree pinpoints which span failed
- Confirm the trace shows the cost/latency of the run
💡 Hint: Auto-tracing wraps each step for you; the goal is to read the resulting tree and find the failing span, not to hand-log.
Exercise I4.2 — Build a dataset from failures
Context: The evaluation flywheel starts the moment you promote a real failure into a golden case — after which it can never regress silently.
Your task: Run your app on 10 inputs, find the 2–3 that produce bad answers, and promote them into a dataset as golden cases with the expected output.
Requirements:
- Exercise the app on ~10 real inputs
- Identify 2–3 genuinely bad outputs
- Add each as a dataset example with the correct expected output
- Note that these cases now guard against silent regression
💡 Hint: Bad outputs are the highest-value eval cases — curate the failures rather than inventing synthetic inputs.
Exercise I4.3 — Version comparison
Context: This is Chapter 5's manual A/B test, now scaled and stored: two prompt versions, scored across a dataset, with the winner kept for a reason.
Your task: Write two prompt versions for a task, run evaluate() for each over your dataset with a deterministic and a judge evaluator, compare per-example, and keep the winner.
Requirements:
- Two prompt versions, one dataset
- Score each with both a deterministic evaluator and an LLM-judge
- Compare per-example, not just on the aggregate
- Keep the winner and write down why
- Watch for a version that improves the mean but regresses specific cases
💡 Hint: The per-example diff is what catches a regression an average would hide — that's why you gate on the dataset, not a single example.
Show what to look for
Watch for a version that improves the average but regresses specific cases — the per-example diff catches what an aggregate score hides. That's exactly why you compare across the dataset, not just on one example, before gating a deploy.
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: Observability platforms like LangSmith start from one primitive: a trace that records what each run did. You'll model that primitive rather than read log lines.
Your task: Model one traced call offline as a decorator that captures inputs, output, latency, and any error — the job LangSmith's @traceable does.
Requirements:
- Implement it as a decorator using only the standard library
- Capture the function name, inputs, latency in ms, and error (None on success)
- Record the run even when the wrapped function raises, then re-raise
- Store captured runs somewhere inspectable and print one
💡 Hint: A try/finally around the call lets you time and record the run on both the success and the exception path; functools.wraps keeps the name.
Show solution
Tracing = recording what each run did. Model it in stdlib (the real thing is @traceable):
import time, functools
RUNS = []
def traced(fn):
@functools.wraps(fn)
def wrap(*a, **k):
t0 = time.perf_counter()
err = None
try:
out = fn(*a, **k)
return out
except Exception as e:
err = repr(e); raise
finally:
RUNS.append({"name": fn.__name__, "inputs": {"args": a},
"latency_ms": round((time.perf_counter()-t0)*1000, 2),
"error": err})
return wrap
@traced
def answer(q):
return f"echo: {q}"
answer("what is prompt caching?")
print(RUNS[-1]) # captured inputs, latency, error=NoneThat is exactly the job LangSmith's decorator does — you just don't hand-roll or read log lines.
Context: A real trace is not a flat log — it's a tree of timed spans (retrieve → generate → parse). That structure is what lets a platform beat grep.
Your task: Model a span tree: a parent run with nested, individually-timed child spans, and print the tree with indentation.
Requirements:
- A span has a name, a duration, and a list of child spans
- Support arbitrary nesting (a child can have its own children)
- Print the tree indented by depth, showing each span's time
- Build a small example (e.g. answer → retrieve, generate → format)
💡 Hint: A tiny recursive node class plus a recursive print indented by depth is all you need; the visual, timed tree is the whole payoff.
Show solution
Traces are step-trees, not flat logs. Runnable:
class Span:
def __init__(self, name):
self.name, self.children, self.ms = name, [], 0
def child(self, name, ms):
s = Span(name); s.ms = ms; self.children.append(s); return s
def show(span, depth=0):
print(" "*depth + f"- {span.name} ({span.ms} ms)")
for c in span.children:
show(c, depth+1)
root = Span("answer")
root.child("retrieve", 40)
gen = root.child("generate", 210)
gen.child("format", 5)
root.ms = 255
show(root)The visual, timed tree is what makes a platform beat grep: you see where the time and the failure actually went.
Context: LangSmith's flywheel turns real traffic into evaluation data: promote good runs into a golden dataset instead of inventing inputs by hand.
Your task: Given a list of captured runs, build a dataset of {input, reference_output} examples, dropping any run that errored or produced no output.
Requirements:
- Input is a list of run dicts carrying input, output, and error
- Keep only runs with no error and a non-empty output
- Emit each survivor as an
{input, reference}example - Report how many examples resulted (and that errored runs were dropped)
💡 Hint: A single list comprehension with the error/output filter is enough — the curation rule is the lesson, not the plumbing.
Show solution
A dataset is just curated examples pulled from real traffic. Runnable:
runs = [
{"input": "reset password?", "output": "Go to Settings > Security", "error": None},
{"input": "refund policy?", "output": "30 days", "error": None},
{"input": "crash case", "output": None, "error": "Timeout"},
]
def to_dataset(runs):
return [{"input": r["input"], "reference": r["output"]}
for r in runs if r["error"] is None and r["output"]]
ds = to_dataset(runs)
for ex in ds:
print(ex)
print(len(ds), "examples (errored run dropped)")Real traffic is the best source of eval cases — you promote the good ones instead of inventing inputs by hand.
Context: An evaluator scores an app's output against a dataset's reference. Real suites mix deterministic checks with an LLM-as-judge; both are just scoring functions.
Your task: Model two evaluators — exact-match and a keyword "LLM-judge" stand-in — and a harness that reports per-example and mean scores over a dataset.
Requirements:
- Exact-match returns 1.0/0.0 on a normalized string comparison
- The judge stand-in scores by whether the reference is contained in the output
- The harness runs a given evaluator over the dataset and returns per-example scores plus the mean
- Run both evaluators over the same dataset and show they can disagree
💡 Hint: Keep the evaluator as a parameter to the harness so swapping exact-match for the judge is a one-line change — exactly how LangSmith runs custom evaluators.
Show solution
Evaluation = score outputs against references. Model the harness offline:
dataset = [
{"input": "capital of France?", "reference": "Paris"},
{"input": "2+2?", "reference": "4"},
]
def my_app(q):
return {"capital of France?": "Paris", "2+2?": "four"}[q]
def exact_match(pred, ref):
return 1.0 if pred.strip().lower() == ref.strip().lower() else 0.0
def judge_contains(pred, ref): # stand-in for LLM-as-judge
return 1.0 if ref.lower() in pred.lower() else 0.0
def run_eval(dataset, evaluator):
scores = [evaluator(my_app(ex["input"]), ex["reference"]) for ex in dataset]
return scores, sum(scores)/len(scores)
s, mean = run_eval(dataset, exact_match)
print("exact-match:", s, "mean", mean) # [1.0, 0.0] 0.5
s, mean = run_eval(dataset, judge_contains)
print("judge:", s, "mean", mean)LangSmith runs your evaluators (exact, custom, or LLM-as-judge) over the dataset and stores the scores; the logic is what you're modeling here.
Context: The reason to store datasets and evaluators is version comparison: proving a prompt or model change actually helped, across the whole set, not on one lucky example.
Your task: Run two app versions over one dataset and report the per-example diff and the mean score delta.
Requirements:
- Score v1 and v2 with the same evaluator over the same dataset
- Show a per-example comparison flagging improvements and regressions
- Compute the mean delta between the two versions
- Make it obvious when the average improves but a specific case regressed
💡 Hint: Zip the two score lists with the dataset and label each row improved/regressed; the per-example view catches what the aggregate hides.
Show solution
Side-by-side scoring across the whole set beats eyeballing two outputs. Runnable:
dataset = [{"q": "a", "ref": "Paris"}, {"q": "b", "ref": "4"}, {"q": "c", "ref": "blue"}]
v1 = {"a": "Paris", "b": "four", "c": "blue"}
v2 = {"a": "Paris", "b": "4", "c": "blue"}
def score(app):
return [1.0 if app[ex["q"]].lower() == ex["ref"].lower() else 0.0
for ex in dataset]
s1, s2 = score(v1), score(v2)
for ex, a, b in zip(dataset, s1, s2):
flag = "" if a == b else (" <- improved" if b > a else " <- regressed")
print(f"{ex['q']}: v1={a} v2={b}{flag}")
print("mean delta:", round(sum(s2)/len(s2) - sum(s1)/len(s1), 3))This is how you prove a prompt/model change helped rather than hoping — the platform just does it over hundreds of cases.
Context: The LLMOps loop closes in CI: an eval gate blocks a deploy when quality drops. This is the O3 release-control pattern, wired to an exit code.
Your task: Model an eval gate that blocks the deploy if the mean score is below a threshold OR any critical example regressed versus the last release, returning a CI-style exit code.
Requirements:
- Fail if mean score < a threshold
- Fail if any named critical example scores worse than its previous score
- Return whether it passed plus the human-readable reasons
- Map the decision to an exit code (0 deploy / 1 block), as CI would
- Show both a passing run and a blocked run (e.g. a safety regression)
💡 Hint: Accumulate reasons into a list; empty reasons means pass. The critical-example check needs both the current and the previous per-example scores.
Show solution
The gate turns eval into a release control (O3). Runnable, returns a CI-style exit code:
def eval_gate(mean_score, per_example_now, per_example_prev,
threshold=0.9, critical=("safety-01",)):
reasons = []
if mean_score < threshold:
reasons.append(f"mean {mean_score:.2f} < {threshold}")
for k in critical:
if per_example_now.get(k, 0) < per_example_prev.get(k, 0):
reasons.append(f"critical '{k}' regressed")
passed = not reasons
return passed, reasons
now = {"safety-01": 1.0, "faq-02": 1.0}
prev = {"safety-01": 1.0, "faq-02": 1.0}
ok, why = eval_gate(0.95, now, prev)
print("deploy" if ok else "BLOCK", why) # deploy []
now2 = {"safety-01": 0.0, "faq-02": 1.0} # a safety regression
ok, why = eval_gate(0.95, now2, prev)
exit_code = 0 if ok else 1
print("deploy" if ok else "BLOCK", why, "exit", exit_code) # BLOCK ... exit 1Closing the LLMOps loop: traces become datasets, datasets feed evals, and the eval gate stops a regression before it ships. LangSmith hosts each step; the real client wraps your app with @traceable and its evaluate() API (needs the SDK).
✓ Checkpoint — you can move on when you can…
- Say what LangSmith adds over hand-rolled O4 tracing and Ch 5 evals.
- Instrument an app (framework or raw SDK) to capture traces.
- Promote real traces into a dataset and explain the flywheel.
- Run deterministic, judge, and custom evaluators over a dataset.
- Compare versions and wire an eval gate into CI (O3).
Knowledge check check yourself
What does LangSmith add over the hand-rolled span() tracing from O4, and why is tracing the foundation for everything else?
Show answer
Describe the trace→dataset→eval flywheel and why comparing versions per-example (not just on average) matters before gating a deploy.