Multi-Agent Workflow with Observability

The brief advanced

Picture the pager going off at 2am. A customer says the agent handed them a confidently wrong answer, and it cost $4 in tokens to produce. You open your logs and find a wall of print statements from five different agents interleaved by timestamp — none of them telling you which agent decided what, or where the run went off the rails. You spend an hour reconstructing a story the system should have handed you in ten seconds. Every engineer who has shipped a multi-agent system has lived some version of this, and it is exactly the pain this brief exists to remove.

Here's the analogy that makes the problem click. A single LLM call is like a receipt — one prompt in, one answer out, easy to read at a glance. A multi-agent workflow is more like a court transcript: many actors, nested exchanges, decisions that only make sense in the order they happened. You can't debug a transcript from a pile of shuffled sticky notes, which is what raw logs are. So the brief on this page is not "add more logging" — it's "make the tree of decisions visible," so a bad run can be expanded like a stack trace until you land on the one step that broke.

Why frame it as a brief at all, rather than jumping to code? Because observability is a design constraint, not a feature you sprinkle on at the end. The most common mistake teams make is shipping the workflow first and promising to "add tracing later" — and later never comes, because retrofitting instrumentation across five agents is far more work than baking it in from the first line. Read the problem statement below as the contract you're signing: every run, fully explainable after the fact, without re-running it.

"Our multi-agent system works — until it doesn't, and we can't tell why." A run produces a bad answer, or costs 10× what it should, or hangs. With five agents and dozens of LLM/tool calls per run, logs alone won't save you. Observability — structured traces of every step, cost/latency metrics, and automated quality scoring — turns an opaque black box into a system you can debug, optimize, and trust.

1 · Discovery — what can't you see today? advanced

Before you build a single instrument, you have to name what's dark. The instinct is to jump straight to "install a tracing library" — but that's answering a question you haven't asked yet. Discovery is the deliberate act of listing the questions your system cannot currently answer, because those questions are what your observability layer must be able to answer. Build tooling before you know the questions and you get the classic anti-pattern: a dashboard full of pretty charts that never once helped you fix an incident.

Think of it like a doctor ordering tests. A good doctor doesn't run every scan money can buy — they start from the symptoms and order the specific test that will change the diagnosis. The table below is that triage: each row is a symptom you've felt ("the answer was wrong — but which agent caused it?"), matched to the exact signal that would resolve it. The star ratings aren't decoration; they're a priority ranking, telling you which blind spots to close first when you can't do everything at once.

The mistake to avoid here is confusing data with answers. Teams often already emit oceans of logs and still can't say which step failed — because logs are a flat stream, and the questions in this table are all about structure and attribution (which agent, which run, how much did this step cost). Discovery reframes observability from "collect more" to "collect what makes these specific questions answerable in seconds."

2 · Architecture advanced

The failure mode this architecture prevents is a specific, expensive one: a team builds the workflow, it works in the demo, they ship it, and only then discover there's no way to see inside it. Now every fix means adding print statements, redeploying, waiting for the bug to recur, and reading the logs — a debugging loop measured in days. The whole shape of the diagram below exists to make that loop impossible, by treating "being able to see the run" as part of the build, not a follow-up ticket.

The key architectural idea is separation of concerns, and a good analogy is a flight recorder. The plane flies itself — the black box doesn't steer, it only records. Notice in the diagram that the workflow (top row) never has to know it's being watched; each step simply emits a span, and a completely separate observability layer collects, aggregates, and scores those emissions. That decoupling is why you can change the workflow without touching your monitoring, and change your monitoring without risking the workflow. The two never entangle.

Why does this belong at the architecture stage rather than being a coding detail? Because it dictates a boundary: the workflow's job is to produce signals cheaply and reliably; the observability layer's job is to consume them. Get that boundary wrong — say, by having agents write directly to your dashboard database — and a slow dashboard can now stall the actual agent. Read the diagram with that split in mind: solid arrows are work flowing forward, dashed arrows are signals dropping down into a system that is deliberately kept off the critical path.

The workflow (a LangGraph multi-agent graph) runs as normal — but every step emits a trace span (inputs, outputs, tokens, latency, errors) to an observability layer (LangSmith or an OpenTelemetry-based stack). That layer computes metrics, runs online evals on live outputs, and feeds dashboards + alerts. Instrumentation is a first-class part of the design, not an afterthought.

3 · Risk & operability model advanced

The scariest failures in a traced system aren't the loud ones — they're the silent ones. A crash pages you; a stack trace tells you where. But a multi-agent workflow that quietly gets 5% worse after a prompt change, or slowly doubles its token spend as one agent starts retrying, doesn't crash — it just erodes, and you find out from an angry customer or a shocking invoice weeks later. This section is about naming those quiet risks and pairing each with a control that makes it loud.

Notice the difference from a normal software risk table: here observability itself is both the safety net and a new risk. Tracing every step means you're now capturing whatever flowed through those steps — including customer PII — and storing it somewhere queryable. So the same layer that saves you from undebuggable failures can, if you're careless, become a compliance breach and a fresh attack surface. That's why redaction and access controls sit right in the risk table, not as an afterthought. The tool that watches your system also has to be watched.

• Silent quality drift is the headline risk because it's invisible without an online score — a model or prompt change can pass every offline test and still degrade real traffic.
• Cost is a safety property here, not just a finance one: an agent stuck in a retry loop can burn a month's budget in an afternoon, so unmonitored cost is an outage waiting to happen.
• The observability layer has a cost of its own — capturing and storing every span at high volume isn't free, which is why sampling (keep all errors, sample successes) is a first-class control rather than a nice-to-have.

4 · Instrumentation, in practice advanced

If you've ever dreaded adding logging, this section is the good news. The mental picture most people have of instrumentation is grim: threading a logger through every function, remembering to log inputs and outputs, wrapping each call in try/except so failures still get recorded, and inevitably forgetting one of them. Done by hand, that boilerplate is so tedious that it's the first thing dropped under deadline pressure — which is exactly why so many systems ship blind.

The move that fixes this is the decorator. Think of it like a stamp on an envelope: you don't rewrite the letter to add postage, you press one mark on top and the postal system knows what to do. Adding @traceable above a function is that stamp — the function's own code doesn't change at all, but every call now silently records a span with its inputs, outputs, tokens, and timing. The instrumentation lives around the logic, not inside it, so you can add and remove it without ever risking the function's behavior.

The subtler point — and the reason nested spans matter — is that decorators compose into a tree automatically. When a traced agent calls other traced functions, their spans nest underneath it, mirroring the actual call structure. You get the court transcript for free, in the right hierarchy, without writing any "child of" bookkeeping. The one thing to internalize before the labs: the code below is only the shape a hosted tool gives you: the rest of this page rebuilds that same capturing in ~30 lines of plain Python, so the decorator stops being magic and becomes something you understand line by line.

Requires: pip install langsmith

observability (shape — LangSmith tracing)# Tracing is often just configuration + decorators
from langsmith import traceable

@traceable(run_type="chain")
def research_agent(query):
    hits = retrieve(query)      # nested spans captured automatically
    return llm_answer(query, hits)

# online eval: score every Nth live run
@traceable
def score_run(inputs, outputs):
    return {"faithfulness": judge(outputs, inputs),
            "cost_usd": outputs.usage.cost}

5 · Observability surface advanced

"Observability" is one word, but it's not one thing — and treating it as one is where teams go wrong. Someone buys a tracing tool, sees traces flowing, and declares the system observable — then is blindsided when quality drifts, because traces alone never told them whether the answers were good. The three pillars are distinct jobs, and skipping any one leaves a specific hole.

The classic framing is traces, metrics, and evals, and an analogy keeps them straight. Traces are the security-camera footage: the full, replayable record of one particular run, for when you need to see exactly what happened. Metrics are the dashboard gauges: numbers aggregated across many runs — average latency, total cost, error rate — for spotting that something is off before you go to the footage. And evals are the quality inspector: they don't ask "did it run?" but "was the output actually good?", which no amount of timing or token-counting can answer.

The table below lays out that surface component by component, and the notes carry the real engineering judgment — which pieces are the load-bearing "control plane" (🟢), which must be sampled to stay affordable (🟠), and which are non-negotiable when customer data is involved (🔴). Read it as a menu with a required course: you can phase in dashboards and online evals, but tracing is the foundation everything else is computed from, and redaction is mandatory the moment real user data touches a span.

6 · Evaluation expert

Here's the twist that catches people: the thing you're evaluating in this project isn't the agents — it's the observability layer itself. It's tempting to grade the workflow's answers, but that's Project 13's job. What decides whether this build succeeded is a different question: when a run breaks, does the trace actually lead you to the broken step, or do you still end up guessing? A tracing system that captures everything but never shortens a debugging session has failed, no matter how full its dashboards look.

That's why the payoff metric in the table is MTTR — mean time to repair, or here, time-to-debug. It's the honest scoreboard. An analogy: buying a smoke detector isn't the goal; the goal is getting out of the house faster when there's a fire. Trace completeness, per-agent cost attribution, and alert precision are all worth measuring, but they're inputs to the one number that proves the layer earns its keep — how fast a trace turns a vague complaint into a named root cause.

The other distinction this section forces is offline versus online evaluation, and conflating them is a common trap. Offline evals run a fixed, curated test set — they gate deploys by checking known cases before ship. Online evals score sampled live traffic — they watch for the drift, novel inputs, and edge cases a frozen test set can never contain. You need both: offline is the exam you pass to graduate, online is the ongoing health check that catches you getting sick afterward. A system with only offline evals looks green right up until reality serves it something the test set never imagined.

7 · Phased rollout expert

The temptation is to build the whole observability platform before you turn anything on. Resist it. Teams that try to ship tracing, per-agent metrics, dashboards, alerts, online evals, and PII redaction all at once usually ship none of it — the scope collapses under its own weight, and meanwhile production stays blind. The phased ladder below exists because each phase delivers real value on its own, and each earns the right to build the next.

The ordering isn't arbitrary — it's sequenced by payoff-per-effort. Phase 1 (just trace everything, in dev) is cheap and instantly transformative: your debugging goes from "guess and redeploy" to "expand the tree." Only once you can see runs do metrics and dashboards (Phase 2) become meaningful, because a dashboard is just traces aggregated. And online evals (Phase 3) come last because they're the most expensive and depend on everything before them — you can't score live traffic you aren't yet capturing. Notice too that redaction lands in Phase 2, right as data starts being stored at scale: safety arrives exactly when the risk does.

Read the red rung as the non-negotiable floor. The common, costly mistake is running a multi-agent workflow in production with no tracing at all — or worse, tracing that quietly logs customer PII unredacted into a queryable store. The phases are a menu you can pace to your team's capacity, but that last rung is not optional: it's the line that separates "operable system" from "incident waiting to happen."

Skills & course map expert

How to use this page expert

Steps in order. terminal = run it; file = create it with the exact contents shown.

Step 1 · Folder + venv + install expert

terminalmkdir -p observability/tests
cd observability
python3 -m venv .venv
source .venv/bin/activate     # Windows: .venv\Scripts\Activate.ps1
pip install pytest
pip freeze > requirements.txt

(.venv) ... Successfully installed pytest-8.3.4

Step 2 · The tracer expert

Create trace.py. A @traced decorator records a span for every wrapped call — critically, in a finally block so failures are captured too. A redact helper strips PII before storage.

trace.py"""A minimal tracer: one span per wrapped call, captured even on error."""
import re, time, functools

RUN = {"spans": []}       # collected spans for the current run
PII = re.compile(r"\b(\d{3}-\d{2}-\d{4}|\d{16})\b")


def reset():
    RUN["spans"] = []


def redact(value) -> str:
    return PII.sub("[REDACTED]", str(value))


def traced(name: str, tokens: int = 0):
    """Wrap a function so each call appends a span with I/O, ms, error."""
    def deco(fn):
        @functools.wraps(fn)
        def wrap(*a, **k):
            span = {"name": name, "input": redact(a),
                    "error": None, "tokens": tokens}
            t0 = time.perf_counter()
            try:
                out = fn(*a, **k)
                span["output"] = redact(out)
                return out
            except Exception as e:
                span["error"] = str(e)
                raise
            finally:
                span["ms"] = (time.perf_counter() - t0) * 1000
                RUN["spans"].append(span)   # recorded even on error
        return wrap
    return deco

Step 3 · Metrics rollup expert

Create metrics.py. It summarises a run's spans into total time, error count, and a cost estimate.

metrics.py"""Roll a run's spans up into the metrics an operator watches."""

PRICE_PER_TOKEN = 0.00001


def rollup(run: dict) -> dict:
    spans = run["spans"]
    return {
        "steps": len(spans),
        "total_ms": round(sum(s["ms"] for s in spans), 2),
        "errors": sum(1 for s in spans if s["error"]),
        "cost_usd": round(sum(s["tokens"] for s in spans) * PRICE_PER_TOKEN, 4),
    }


def failing_step(run: dict):
    """Return the name of the first span that errored, or None."""
    for s in run["spans"]:
        if s["error"]:
            return s["name"]
    return None

Step 4 · Instrument a workflow & run it expert

Create workflow.py — a small multi-step "crew" with each step traced, including one step that fails, so you can see the trace pinpoint it.

workflow.py"""A traced multi-step workflow — no model needed to see observability."""
from trace import traced, RUN, reset
from metrics import rollup, failing_step


@traced("planner", tokens=300)
def planner(q): return ["sub1", "sub2"]


@traced("researcher", tokens=1200)
def researcher(subq): return f"finding for {subq}"


@traced("writer", tokens=500)
def writer(findings): return "REPORT: " + "; ".join(findings)


def run(question):
    reset()
    subs = planner(question)
    findings = [researcher(s) for s in subs]
    report = writer(findings)
    return report


if __name__ == "__main__":
    print(run("why did churn rise?"))
    print("--- run tree ---")
    for s in RUN["spans"]:
        print(f"  {s['name']:12} {s['ms']:6.2f}ms  tok={s['tokens']}  "
              f"err={s['error']}")
    print("--- metrics ---")
    print(rollup(RUN))

terminalpython workflow.py

REPORT: finding for sub1; finding for sub2
--- run tree ---
  planner        0.01ms  tok=300  err=None
  researcher     0.00ms  tok=1200  err=None
  researcher     0.00ms  tok=1200  err=None
  writer         0.01ms  tok=500  err=None
--- metrics ---
{'steps': 4, 'total_ms': 0.02, 'errors': 0, 'cost_usd': 0.0322}

Step 5 · Tests (no key) expert

tests/test_trace.py"""Offline tracer + metrics tests — no key."""
import pytest
from trace import traced, RUN, reset, redact
from metrics import rollup, failing_step


def setup_function():
    reset()


def test_span_records_io_and_ms():
    @traced("step")
    def f(x): return x + 1
    f(1)
    span = RUN["spans"][0]
    assert span["name"] == "step" and "ms" in span


def test_error_span_is_captured():
    @traced("boom")
    def f(): raise ValueError("nope")
    with pytest.raises(ValueError):
        f()
    assert RUN["spans"][0]["error"] == "nope"   # captured despite raising


def test_rollup_counts_and_costs():
    @traced("a", tokens=100)
    def a(): return 1
    a(); a()
    m = rollup(RUN)
    assert m["steps"] == 2 and m["errors"] == 0
    assert m["cost_usd"] > 0


def test_pii_redacted_before_store():
    @traced("pii")
    def f(x): return x
    f("my ssn 123-45-6789")
    assert "[REDACTED]" in RUN["spans"][0]["input"]


def test_failing_step_is_findable():
    @traced("ok")
    def ok(): return 1
    @traced("bad")
    def bad(): raise RuntimeError("x")
    ok()
    with pytest.raises(RuntimeError):
        bad()
    assert failing_step(RUN) == "bad"


def test_redact_leaves_clean_text():
    assert redact("hello world") == "hello world"

terminalpython -m pytest tests/ -v

tests/test_trace.py::test_span_records_io_and_ms PASSED
tests/test_trace.py::test_error_span_is_captured PASSED
tests/test_trace.py::test_rollup_counts_and_costs PASSED
tests/test_trace.py::test_pii_redacted_before_store PASSED
tests/test_trace.py::test_failing_step_is_findable PASSED
tests/test_trace.py::test_redact_leaves_clean_text PASSED

6 passed in 0.04s

Step 6 · Go live with LangSmith (optional) expert

Your @traced has the same shape as LangSmith's @traceable. To get hosted dashboards, install and swap the decorator — no other changes:

terminalpip install langsmith
export LANGSMITH_API_KEY="ls-your-key-here"

workflow.py (change the import + decorator)from langsmith import traceable

@traceable(run_type="chain")       # same shape — dashboards for free
def researcher(subq): return f"finding for {subq}"

Troubleshooting — every error you might hit expert

RUN = {"spans": []}

def reset():
    RUN["spans"] = []

def record(name, ms, tokens=0, error=None):
    RUN["spans"].append(
        {"name": name, "ms": ms, "tokens": tokens, "error": error})

reset()
record("planner", 1.2, tokens=300)
record("writer", 0.4, tokens=500)
print(len(RUN["spans"]))                 # 2
print([s["name"] for s in RUN["spans"]])  # ['planner', 'writer']

import time, functools
RUN = {"spans": []}
def reset(): RUN["spans"] = []

def traced(name, tokens=0):
    def deco(fn):
        @functools.wraps(fn)
        def wrap(*a, **k):
            span = {"name": name, "tokens": tokens, "error": None}
            t0 = time.perf_counter()
            try:
                return fn(*a, **k)
            except Exception as e:
                span["error"] = str(e); raise
            finally:
                span["ms"] = round((time.perf_counter() - t0) * 1000, 3)
                RUN["spans"].append(span)
        return wrap
    return deco

@traced("boom", tokens=10)
def boom(): raise ValueError("nope")

reset()
try: boom()
except ValueError: pass
print(RUN["spans"][0]["error"], "ms" in RUN["spans"][0])  # nope True

import time, functools
RUN = {"spans": []}
STACK = []
def reset(): RUN["spans"].clear(); STACK.clear()

def traced(name):
    def deco(fn):
        @functools.wraps(fn)
        def wrap(*a, **k):
            sid = len(RUN["spans"])
            span = {"id": sid, "name": name,
                    "parent": STACK[-1] if STACK else None, "error": None}
            RUN["spans"].append(span); STACK.append(sid)
            t0 = time.perf_counter()
            try: return fn(*a, **k)
            except Exception as e: span["error"] = str(e); raise
            finally:
                span["ms"] = round((time.perf_counter()-t0)*1000, 2)
                STACK.pop()
        return wrap
    return deco

@traced("summarize")
def summarize(): return "s"
@traced("writer")
def writer(): return summarize() + "!"

reset(); writer()
def depth(s):
    d = 0; p = s["parent"]
    while p is not None: d += 1; p = RUN["spans"][p]["parent"]
    return d
for s in RUN["spans"]:
    print("  " * depth(s) + s["name"])   # writer / (indented) summarize

import re, math
PII = re.compile(r"\b(\d{3}-\d{2}-\d{4}|\d{16})\b")

def redact(v): return PII.sub("[REDACTED]", str(v))

def p95(values):
    if not values: return 0.0
    xs = sorted(values)
    idx = math.ceil(0.95 * len(xs)) - 1        # nearest-rank, clamp to last
    return xs[max(0, min(idx, len(xs) - 1))]

def rollup(spans):
    ms = [s["ms"] for s in spans]
    return {"steps": len(spans),
            "total_ms": round(sum(ms), 2),
            "p95_ms": round(p95(ms), 2),
            "errors": sum(1 for s in spans if s["error"])}

print(redact("ssn 123-45-6789 card 1234567812345678"))
spans = [{"ms": m, "error": None} for m in [5,6,7,8,9,10,100]]
print(rollup(spans))   # p95 tracks the tail (100), mean would hide it

PRICE = 0.00001                     # $/token
def should_keep(run_id, sample_n, had_error):
    if had_error: return True                       # never drop failures
    return (hash(str(run_id)) % sample_n) == 0      # deterministic 1-in-N

def budget_alarm(spans, ceiling_usd):
    cost = sum(s.get("tokens", 0) for s in spans) * PRICE
    over = cost > ceiling_usd
    return {"cost_usd": round(cost, 4), "over_budget": over,
            "alarm": "PAGE" if over else None}

kept = [rid for rid in range(10) if should_keep(rid, 5, had_error=False)]
print("kept run ids:", kept)                 # ~1-in-5
print(should_keep("r7", 5, had_error=True)) # True -- error always kept
print(budget_alarm([{"tokens": 900000}], ceiling_usd=5.0))  # over_budget True

from collections import deque

class BurnRateAlerter:
    def __init__(self, slo=0.99, short=5, long=60, mult=14.4):
        self.budget = 1 - slo          # allowed error fraction
        self.short = deque(maxlen=short)
        self.long = deque(maxlen=long)
        self.mult = mult

    def observe(self, errored: bool):
        e = 1 if errored else 0
        self.short.append(e); self.long.append(e)

    def _burn(self, window):
        if not window: return 0.0
        rate = sum(window) / len(window)
        return rate / self.budget if self.budget else 0.0

    def page(self):
        # multi-window: both must exceed the multiplier -> real, fast burn
        return self._burn(self.short) >= self.mult and \
               self._burn(self.long) >= self.mult

a = BurnRateAlerter(short=5, long=10)
for _ in range(10): a.observe(errored=True)    # sustained outage
print(a.page())                                 # True -- both windows burning
b = BurnRateAlerter(short=5, long=10)
for i in range(10): b.observe(errored=(i == 0)) # one blip
print(b.page())                                 # False -- doesn't flap

observability/
├─ .venv/
├─ requirements.txt
├─ trace.py            (@traced decorator + redact)
├─ metrics.py          (rollup + failing_step)
├─ workflow.py         (traced multi-step workflow)
└─ tests/
   └─ test_trace.py    (6 offline tests)

Knowledge check check yourself