Evaluation and Tracing with LangSmith

Why a platform beats print statements advanced

Picture the 2am page you don't want. Your agent gave a customer a confidently wrong answer, the ticket's been escalated, and all you have to reconstruct what happened is a wall of print() output interleaved from a dozen concurrent requests. You grep for the user's question, find three matching lines, and realize you never logged which tool the agent called or what the retriever returned. The run is gone. That helpless feeling — I can't see what my own system did — is the exact problem an observability platform exists to kill.

Here's the honest framing, because it's easy to feel like reaching for a platform is admitting your O4 code wasn't good enough. It was good enough — for learning. Rolling your own span() helper in O4 was the right way to understand tracing, the same way building a stack from scratch teaches you what a stack is. But hand-rolled logging hits a wall the moment you have real traffic: thousands of runs you need to search, two prompt versions you need to compare across a hundred cases, a product manager who wants to inspect a failure without reading Python. Those aren't code problems you can out-clever — they're storage, search, and UI problems, and building that yourself means building a second product alongside your actual one.

The analogy that makes this click: think of the difference between writing print(timestamp) in your code versus wiring up a real logging service like Datadog or CloudWatch. Both record events — but one is a string in a terminal that scrolls away, and the other is durable, indexed, queryable, and shareable across a team. LangSmith is that same jump, applied specifically to LLM runs. It doesn't invent a new idea; every concept — the trace tree, the span, the golden set, the LLM judge — you already met in O4 and Chapter 5. What it adds is the boring, essential infrastructure that turns those concepts from "a thing you did once by hand" into "a thing your whole team does continuously at scale."

The common beginner mistake is treating tracing as an afterthought — something you bolt on when a bug appears. By then it's too late: the run you needed to see already happened untraced. The discipline is to trace from the start, so the data is already sitting there when the incident lands. The other half of the mistake is thinking tracing and evaluation are separate chores. They aren't — and that unity is the platform's real trick. The traces you capture become your evaluation dataset, and your evaluations run over that same trace data. Capture is the foundation everything else stacks on; read the table below as a job-by-job map from O4/Ch 5 to what the platform hands you, but keep that one-loop picture in mind rather than seeing five disconnected features.

Your O4 span() helper works — until you have thousands of runs, want to search them, compare two prompt versions across a hundred cases, or let a non-engineer inspect a failure. Hand-rolled logging doesn't scale to those. A tracing/eval platform does: it stores runs durably, makes them searchable and visual, and connects tracing to evaluation so the two reinforce each other.

Lab I4.1 · Tracing an app advanced

Everything in this chapter is built on one primitive, so it's worth dwelling on: the trace. If you skip capturing runs, you have nothing to search, nothing to promote into a dataset, nothing to score, and nothing to monitor. That's why this is Lab I4.1 and not an afterthought — get capture right and the rest of the platform simply falls out of it. A trace is just a faithful recording of what one run of your code did: the input it received, the output it produced, how long each step took, how much it cost in tokens, and where — if anywhere — it blew up.

Think of it like the black box flight recorder on an aircraft. Nobody wants the plane to crash, but if it does, the recorder is the difference between "we have no idea what happened" and "engine two failed at 04:12, here's the exact sequence." Your LLM app will produce surprising outputs — that's the nature of a probabilistic system — and the trace is the recorder that lets you replay the surprise instead of shrugging at it. The tempting shortcut is to reach for tracing only after something breaks; the discipline, as with a flight recorder, is that it's already running before anyone needs it.

The one idea that trips people up here: a trace is not a single log line, it's a nested tree of spans. A span is one timed step — the model call, the retriever lookup, a tool invocation — and spans nest inside each other the way function calls do. Your top-level answer() is the root; the retrieve step and the model call are its children; a tool the model triggers is a grandchild. This is exactly the step-tree you built by hand in O4, and it's what lets you answer "where did the time go?" and "which step failed?" instead of just "the run was slow" or "the run errored." A flat log can't do that; a tree can.

Here's the part that feels almost too easy: you get that whole tree from a single decorator. LangSmith auto-instruments LangChain and LangGraph with zero code changes (just environment variables), and for everything else — including the raw-SDK apps you wrote back in C2 — it gives you @traceable, a one-line wrapper you place above any function. Decorate your model call, your retriever, and each tool, and the platform assembles the nested tree for you and stores it durably. The lab below is the smallest possible version: one decorated function, one call, one fully-captured run.

The foundation is capturing runs. LangSmith auto-instruments LangChain/LangGraph, and offers a decorator/wrapper for arbitrary code — so even your raw-SDK apps (C2) get traced.

shellpip install langsmith
export LANGSMITH_API_KEY=...          # plus LANGSMITH_TRACING=true

trace.pyfrom langsmith import traceable
from anthropic import Anthropic
client = Anthropic()

@traceable                              # captures inputs, outputs, timing, errors
def answer(question: str) -> str:
    r = client.messages.create(
        model="claude-opus-4-8", max_tokens=512,
        messages=[{"role":"user","content": question}],
    )
    return next(b.text for b in r.content if b.type == "text")

answer("What is prompt caching?")   # run appears in LangSmith, fully traced

trace.pyfrom langsmith import traceable
from openai import OpenAI
client = OpenAI()

@traceable                              # captures inputs, outputs, timing, errors
def answer(question: str) -> str:
    r = client.responses.create(
        model="gpt-5.5", max_output_tokens=512,
        input=question,
    )
    return r.output_text

answer("What is prompt caching?")   # run appears in LangSmith, fully traced

Lab I4.2 · From traces to datasets advanced

Where do the test cases for an LLM app actually come from? This is the question that quietly sinks most evaluation efforts. The instinct is to sit down and invent a golden set — dream up twenty questions you think users will ask, write down what you think the right answers are, and call it a test suite. It feels productive. It's also a trap, because you're testing the app against your imagination of how it's used, not how it's actually used. The cases that break in production are almost always the ones you'd never have thought to write: the weirdly-phrased question, the edge-case document, the input in a language you forgot to handle.

LangSmith's answer — and the single most important habit in this chapter — is to build your evaluation set from real traces instead of from guesses. Because you're already capturing every run (Lab I4.1), you have a growing pile of real inputs and real outputs. When a run is interesting — a genuinely great answer worth locking in, or a production failure worth never repeating — you promote it into a dataset with one click. The reference answer you attach is the correct output, which for a failure means the answer it should have given. Your golden set is now grown from the field, not the whiteboard.

The mental model to hold is a flywheel, not a pipeline. A pipeline runs once, start to finish, and stops. A flywheel is a loop that gets heavier and harder to knock over every time it turns: traces feed the dataset, the dataset feeds evaluations, and the failures those evaluations surface feed back into the dataset as new permanent cases. Each turn, your test suite gets a little more comprehensive and your app a little harder to break. This is the productized form of the O4 habit "every incident becomes a test case" — LangSmith just makes promoting a trace to a golden case a button instead of a copy-paste chore.

The common mistake worth naming is treating the dataset as a one-time artifact you build at the start of the project and never touch again. A static golden set rots: your app changes, your users change, new failure modes appear, and a test suite frozen in month one stops catching anything by month six. The whole point of the loop below is that the dataset is living — every real failure you find becomes a case that can never silently regress again. Read the diagram as exactly that: a circle, with the dashed arrow closing the loop being the part that matters most.

The platform's superpower is the loop between the two halves: real traces become your evaluation dataset. When a run is interesting — a great answer, or a production failure — you promote it into a dataset. Over time your golden set (Ch 5) is built from real usage, not guesses.

Lab I4.3 · Running evaluators expert

A dataset alone doesn't tell you anything — it's just questions and answers sitting in a drawer. The thing that turns it into a signal is an evaluator: a small function that looks at what your app produced for a given input and scores how good it was. Without evaluators, "we have 200 golden cases" is a filing cabinet; with them, it's a quality gauge you can read at a glance. This section is where the two halves of the platform finally connect — the traces you captured became a dataset, and now the dataset gets graded.

The cleanest way to think about an evaluator is that it's exactly like a unit test's assert, but returning a score instead of pass/fail — and crucially, one that can handle the fact that language has no single right answer. A normal test asserts add(2, 2) == 4 and that's the end of it. But "was this support answer helpful?" has no ==. So evaluators come in a spectrum of strictness, and matching the evaluator to the question is the real skill. Some questions do have an exact answer (did it return valid JSON? does the output contain the order number?) — for those you write a plain deterministic rule, fast and free and perfectly reliable. Other questions are irreducibly fuzzy (was the tone right? is the answer grounded in the retrieved context?) — for those you reach for an LLM-as-judge: a second model call that reads the output and grades it against a rubric, exactly the Chapter 5 technique.

The mistake nearly everyone makes is jumping straight to an LLM judge for everything because it feels the most "AI." It's the wrong default. A judge is slower, costs money on every example, and — this is the part that bites — is itself a fallible LLM that can be confidently wrong about whether your answer was good. If a cheap deterministic rule can answer the question ("does the JSON parse?"), use it: it's faster, free, and never hallucinates. Save the judge for the genuinely subjective calls it's the only tool for, and even then, spot-check its verdicts against your own the way Chapter 5 warned. The platform's contribution isn't deciding what to measure — that judgment stays yours — it's giving you a place to run whatever evaluators you write across the whole dataset and store every score over time.

The beautiful part of the interface you'll see below is that all three flavours — deterministic, judge, and anything custom you dream up — share one shape: a function that takes the run and the example and returns a {"key": ..., "score": ...} dict. Because a custom evaluator is just "any Python you can write," a latency budget, a cost cap, or a tool-call-correctness check all become first-class tracked metrics with no special API. Write the rule; the platform charts it.

With a dataset, you run evaluators over it — the same eval types from Chapter 5, executed and scored by the platform.

evaluate.pyfrom langsmith import evaluate

def exact_match(run, example):        # deterministic evaluator
    return {"key":"correct",
            "score": run.outputs["output"] == example.outputs["expected"]}

def grounded_judge(run, example):     # LLM-as-judge evaluator (Ch 5)
    verdict = judge_model_says_grounded(run.outputs, example.inputs)
    return {"key":"grounded", "score": verdict}

evaluate(
    my_app,                            # the function/chain under test
    data="my-golden-set",               # the dataset
    evaluators=[exact_match, grounded_judge],
)   # scores stored & visualized per example

Comparing versions & the eval gate expert

Here's a scenario every LLM engineer lives through. You tweak a prompt — add one sentence, reword an instruction — run it against a couple of examples in your head, the outputs look better, and you ship it. A week later someone reports the assistant is now getting a category of question wrong that it used to nail. Your innocent one-sentence change fixed the case you were looking at and quietly broke three you weren't. This is the single most common way LLM apps regress, and it happens because prompt changes are invisible: there's no compiler error, no failing test, nothing to warn you that helping here hurt there.

The fix is to stop trusting your eyes and start trusting the dataset. Because you've stored evaluators and scores, you can run version A (today's app) and version B (your change) over the same golden set and compare them head to head. This is just A/B testing, the discipline that's been standard in web engineering for decades — you don't guess whether the new checkout button converts better, you measure it against the old one on the same traffic. Version comparison brings that rigor to prompts and models: change one thing, re-run the eval, read the result.

But the crucial word is per-example, and this is the part beginners miss. The lazy comparison looks at one number — "version A scored 0.82, version B scored 0.85, ship B." That average is a liar. A version can raise the mean while breaking a handful of specific cases, because the improvement on many easy examples mathematically drowns out a catastrophic regression on a few critical ones. Imagine B gets 5% better at FAQ questions but suddenly fails the one safety-critical case that must never fail — the average still goes up, and you'd have shipped a disaster. A per-example, side-by-side diff catches exactly this: you see the row that went from pass to fail, in red, no matter what the mean did. Always read the rows, not just the total.

Once you're comparing on data, the last step writes itself: turn the comparison into a gate. This is the O3 eval-gate discipline made concrete — the same idea as a CI test suite that blocks a merge when a test fails. Wire the evaluate() call into your pipeline so every prompt or model change re-runs the golden set, and fail the build if the score drops below threshold or any critical case regresses. Now no prompt change ships on vibes; it ships only after clearing the dataset. In production the same idea becomes a canary: roll B out to a slice of traffic and confirm it holds before going wide. The diagram traces that whole path — two versions, one dataset, a per-example compare, and a gate on the far end.

The payoff of storing evals: compare two versions of your app across the whole dataset. Change a prompt or swap a model, re-run the eval, and see per-example whether you improved or regressed — the safe-rollout discipline from O3, with data instead of vibes.

Online monitoring expert

You can pass every offline eval and still be shipping a broken app. That sounds contradictory until you see why: your golden set only tests the cases you've collected, and production is a firehose of inputs you've never seen. A model provider silently updates the model behind the API, a new customer segment starts asking questions in a style your dataset doesn't cover, an upstream document source changes format — none of that shows up in an offline eval, because offline evals look backwards at a fixed dataset. The world moves on after your last eval run, and the only way to know is to watch the live system.

The clean way to hold the distinction: offline evaluation is the exam you study for; online monitoring is the job you do afterwards. The exam (your golden set) proves you're ready before you deploy — it catches regressions at the gate. Monitoring proves you're still doing well after, on real work you couldn't have rehearsed. You need both, and they're not interchangeable: passing the exam doesn't mean you'll never make a mistake on the job, and doing the job well doesn't retroactively prove your exam was comprehensive. Skip either and you have a blind spot — either you ship regressions, or you never notice drift.

Because LangSmith is already recording every production run as a trace, monitoring is almost free: it's the same trace data, aggregated into dashboards instead of inspected one run at a time. You watch the four signals from O4 — quality (run a sampled LLM judge over live traffic, since you can't judge every run), cost (tokens add up fast at scale), latency (the P95 your users actually feel, not the average), and errors (tool failures, timeouts, refusals) — and you set alerts so a spike pages you instead of waiting for a customer complaint. This is O4's monitoring pillar with a UI bolted on. And notice the through-line of the whole chapter: monitoring, offline evals, and datasets all run off the same captured traces. That's why Lab I4.1 was the foundation — no traces, no monitoring, no evals, nothing.

Beyond offline evals, LangSmith runs on live traffic too: dashboards over production traces (the four signals from O4 — quality via sampled judges, cost, latency, errors), and alerts. This is O4's monitoring pillar with a UI, closing the loop from "trace one run" to "watch the whole system."

Not the only tool expert

It would be a shame to walk away from this chapter thinking you learned "LangSmith." You didn't — you learned observability for LLM apps, and LangSmith was the concrete tool we used to make it tangible, the way you might learn version control through Git without confusing the two. Tools churn: vendors get acquired, pricing changes, a better product appears, your company standardizes on something else. If your knowledge is "which button to click in LangSmith," it expires the day you switch tools. If your knowledge is "capture traces, build datasets from real runs, run deterministic and judge evaluators, compare versions, gate CI, monitor prod," it transfers to every platform, forever. Learn the jobs, not the UI.

And the jobs really are portable, because the whole space is converging on shared foundations. A growing number of these platforms trace on top of OpenTelemetry — the same open standard used for tracing ordinary microservices — which means your trace data isn't locked to one vendor's format. The landscape includes LangSmith, other hosted observability/eval products, and mature open-source options you can self-host; they differ in polish and integration, not in the fundamental jobs they do.

So how do you actually choose? Exactly the way you reasoned about stack layers in O2 — by your constraints, not by brand or hype. Three questions decide it. First, your stack: if you're deep in LangChain/LangGraph, LangSmith's zero-config auto-instrumentation is a real gravity well; if you're on the raw SDK or a mixed stack, an OpenTelemetry-based tool may fit more cleanly. Second, and often decisive, your data-residency needs: a hosted platform means your prompts and outputs leave your infrastructure, which for a regulated industry can be a non-starter — that's when self-hosted or enterprise deployments earn their cost. Third, budget: usage-based pricing on a high-traffic app adds up, and an open-source self-host trades engineering time for a lower bill. Weigh those three against how well each tool does the six jobs above, and the choice makes itself.

LangSmith is popular and tightly integrated with LangChain/LangGraph, but the capabilities — tracing, datasets, evaluators, version comparison, monitoring — are the transferable knowledge. Other observability/eval platforms and open-source options (including OpenTelemetry-based tracing) offer the same jobs. Pick by your stack, data-residency needs (self-host vs hosted), and budget, exactly as you chose stack layers in O2.

Common pitfalls expert

The failure modes of observability are subtle, because a broken setup still looks like it's working. A dashboard is showing green numbers, evals are passing, traces are flowing — so it's tempting to assume you're covered. The pitfalls below are exactly the cases where that comfortable green hides a real gap, and every one of them maps back to a principle from this chapter or an earlier one. Read the table not as trivia but as the difference between observability that protects you and observability that gives you false confidence.

Two are worth calling out because they're the ones that quietly cause the most damage. The first is tracing with no data-handling plan — the moment you send traces to a hosted service, you're shipping full prompts and outputs off your infrastructure, and those may contain PII, secrets, or confidential content (O4, T1). Observability that leaks data isn't a feature, it's an incident waiting to happen; redact, control access, and check residency before you flip tracing on, not after. The second is building a golden set from imagination instead of promoting real traces — the whole flywheel of this chapter collapses if your test cases are guesses, because you end up perfectly passing evals for inputs no real user ever sends. The rest of the table is the same lesson applied to judges (they're fallible — spot-check them), to monitoring (offline isn't enough — watch live traffic for drift), to rollout (compare versions before shipping), and to trace depth (decorate every step, not just the top call, or your tree has holes exactly where the bug hides).

Exercises expert

import time, functools

RUNS = []
def traced(fn):
    @functools.wraps(fn)
    def wrap(*a, **k):
        t0 = time.perf_counter()
        err = None
        try:
            out = fn(*a, **k)
            return out
        except Exception as e:
            err = repr(e); raise
        finally:
            RUNS.append({"name": fn.__name__, "inputs": {"args": a},
                         "latency_ms": round((time.perf_counter()-t0)*1000, 2),
                         "error": err})
    return wrap

@traced
def answer(q):
    return f"echo: {q}"

answer("what is prompt caching?")
print(RUNS[-1])   # captured inputs, latency, error=None

class Span:
    def __init__(self, name):
        self.name, self.children, self.ms = name, [], 0
    def child(self, name, ms):
        s = Span(name); s.ms = ms; self.children.append(s); return s

def show(span, depth=0):
    print("  "*depth + f"- {span.name} ({span.ms} ms)")
    for c in span.children:
        show(c, depth+1)

root = Span("answer")
root.child("retrieve", 40)
gen = root.child("generate", 210)
gen.child("format", 5)
root.ms = 255
show(root)

runs = [
    {"input": "reset password?", "output": "Go to Settings > Security", "error": None},
    {"input": "refund policy?",  "output": "30 days",                   "error": None},
    {"input": "crash case",      "output": None, "error": "Timeout"},
]
def to_dataset(runs):
    return [{"input": r["input"], "reference": r["output"]}
            for r in runs if r["error"] is None and r["output"]]

ds = to_dataset(runs)
for ex in ds:
    print(ex)
print(len(ds), "examples (errored run dropped)")

dataset = [
    {"input": "capital of France?", "reference": "Paris"},
    {"input": "2+2?",               "reference": "4"},
]
def my_app(q):
    return {"capital of France?": "Paris", "2+2?": "four"}[q]

def exact_match(pred, ref):
    return 1.0 if pred.strip().lower() == ref.strip().lower() else 0.0

def judge_contains(pred, ref):          # stand-in for LLM-as-judge
    return 1.0 if ref.lower() in pred.lower() else 0.0

def run_eval(dataset, evaluator):
    scores = [evaluator(my_app(ex["input"]), ex["reference"]) for ex in dataset]
    return scores, sum(scores)/len(scores)

s, mean = run_eval(dataset, exact_match)
print("exact-match:", s, "mean", mean)   # [1.0, 0.0] 0.5
s, mean = run_eval(dataset, judge_contains)
print("judge:", s, "mean", mean)

dataset = [{"q": "a", "ref": "Paris"}, {"q": "b", "ref": "4"}, {"q": "c", "ref": "blue"}]
v1 = {"a": "Paris", "b": "four", "c": "blue"}
v2 = {"a": "Paris", "b": "4",    "c": "blue"}

def score(app):
    return [1.0 if app[ex["q"]].lower() == ex["ref"].lower() else 0.0
            for ex in dataset]

s1, s2 = score(v1), score(v2)
for ex, a, b in zip(dataset, s1, s2):
    flag = "" if a == b else ("  <- improved" if b > a else "  <- regressed")
    print(f"{ex['q']}: v1={a} v2={b}{flag}")
print("mean delta:", round(sum(s2)/len(s2) - sum(s1)/len(s1), 3))

def eval_gate(mean_score, per_example_now, per_example_prev,
              threshold=0.9, critical=("safety-01",)):
    reasons = []
    if mean_score < threshold:
        reasons.append(f"mean {mean_score:.2f} < {threshold}")
    for k in critical:
        if per_example_now.get(k, 0) < per_example_prev.get(k, 0):
            reasons.append(f"critical '{k}' regressed")
    passed = not reasons
    return passed, reasons

now  = {"safety-01": 1.0, "faq-02": 1.0}
prev = {"safety-01": 1.0, "faq-02": 1.0}
ok, why = eval_gate(0.95, now, prev)
print("deploy" if ok else "BLOCK", why)          # deploy []

now2 = {"safety-01": 0.0, "faq-02": 1.0}          # a safety regression
ok, why = eval_gate(0.95, now2, prev)
exit_code = 0 if ok else 1
print("deploy" if ok else "BLOCK", why, "exit", exit_code)   # BLOCK ... exit 1

Knowledge check check yourself