Reliable, observable agents

1 · The reliability problem

Picture the demo that goes perfectly. You type a question, your agent calls a tool, thinks, calls another, and hands back a crisp answer. The room nods. Two weeks later the same agent is running unattended against real traffic, and one night it gets a request it can't quite satisfy — so it tries the same tool again, and again, and again, 400 times before dawn, burning through your entire month's token budget on a single stuck run. Nothing crashed. No exception was raised. It just quietly did the wrong thing, expensively, while everyone slept. That gap — between "it worked in the demo" and "it's safe to leave running" — is what this whole chapter closes.

Here's the mental shift that makes the rest click. A plain prompt is like asking one question and getting one answer: there's exactly one thing that can go wrong — the answer is bad. An agent is a loop, and a loop opens a whole new category of failure that has nothing to do with any single answer being wrong. Think of the difference between handing someone a math problem versus handing them your credit card and saying "buy what we need for dinner." The first can only be wrong once; the second can loop the grocery store forever, buy the wrong thing, spend more than you meant, or wander into the electronics aisle. The danger isn't in any one decision — it's in the repetition, the autonomy, and the time.

The common mistake is to treat reliability as a model problem — "we'll fix it by using a smarter model." It isn't. A smarter model drifts less often, but "less often" times "runs unattended thousands of times a day" is still a real number of 3 a.m. incidents. Reliability is not a property you can prompt or fine-tune into the model; it's a property of the harness you build around it — the caps, gates, logs, and checks that make misbehavior bounded, visible, and cheap instead of unbounded, silent, and expensive. Now the precise version: an agent is a loop where a non-deterministic model decides the next action over and over, so it inherits a whole family of failure modes that only appear over time. It can loop forever, ping-pong between two tools, blow a token budget, call a destructive tool, or drift off the original task without ever crashing. None of these show up in a quick demo — they show up at 3 a.m. when nobody is watching.

So “reliable” for an agent is not “the model is smart.” It is a set of properties you build around the model:

2 · Budgets & loop caps — bound every run

Every taxi has a meter, and the meter is a promise to you, the passenger. It doesn't make the driver take a better route — but it guarantees that whatever happens, the ride ends, and you find out what it cost. An agent loop with no caps is a taxi with no meter and no destination: it will happily keep driving as long as the model keeps saying "one more step," and the only thing that stops it is your credit card maxing out. The single most important reliability control isn't clever — it's just fitting a meter that the ride cannot run past.

Why is this the first control, before guardrails or tracing? Because it's the one that fails catastrophically rather than merely badly. A missing guardrail lets a bad thing through occasionally; a missing cap lets a single run consume unbounded money and time. And agents loop for perfectly ordinary reasons — a tool that always returns an error, a goal the model genuinely can't satisfy, two tools it keeps bouncing between because each looks slightly better than the other. There's no bug to fix; the loop is behaving exactly as written. The cap is what turns "behaving as written, forever" into "behaving as written, then stopping."

The rookie mistake is to bound only one axis — usually step count — and assume that's enough. It isn't, because the axes measure different kinds of runaway. A run can hit ten cheap steps or three ruinously expensive ones; it can make few model calls but hammer a slow tool for minutes. So you cap several dimensions at once: max steps (how many loop iterations), max tokens (your actual cost budget, since tokens are what you pay for), max tool calls (how many times it reaches into the outside world), and in production a wall-clock timeout (real seconds elapsed, the one that protects a user waiting on the other end). Whichever cap trips first wins. And here's the detail that separates a professional harness from a hack: when a cap trips, the loop must stop cleanly and say which cap and how far it got — a caught, reported, structured limit — never a silent hang or a raw crash. A stop you can read is a stop you can alert on; a hang is a page at 3 a.m.

Here is an agent loop wrapped in hard caps. The fake model would answer on its fourth turn; we run it three ways — once where it finishes normally, once with a tight token budget that trips first, and once with a tight step cap. Each returns a clean result saying exactly why it stopped. We simulate token counts with fixed values so there is no wall-clock and the output is deterministic:

budget.py# An agent loop with HARD CAPS. The 'model' is a deterministic fake so this
# runs offline; a real LLM drops into the same slot. We simulate tokens and
# tool calls with fixed counts (no wall-clock, no randomness).

def fake_model(step):
    # Turns 0..2 call a tool; turn 3 would answer -- but caps may stop us first.
    if step < 3:
        return {'type': 'tool_call', 'name': 'search', 'sim_tokens': 120}
    return {'type': 'final', 'text': 'done: found the answer', 'sim_tokens': 90}

def run_agent(question, max_steps=8, max_tokens=1000, max_tool_calls=5):
    used_tokens = 0
    tool_calls = 0
    for step in range(max_steps + 1):
        if step >= max_steps:
            return {'stop': 'STEP_CAP', 'step': step, 'tokens': used_tokens, 'tool_calls': tool_calls}
        decision = fake_model(step)
        used_tokens += decision['sim_tokens']
        if used_tokens > max_tokens:
            return {'stop': 'TOKEN_BUDGET', 'step': step, 'tokens': used_tokens, 'tool_calls': tool_calls}
        if decision['type'] == 'final':
            return {'stop': 'FINAL_ANSWER', 'step': step, 'tokens': used_tokens,
                    'tool_calls': tool_calls, 'text': decision['text']}
        tool_calls += 1
        if tool_calls > max_tool_calls:
            return {'stop': 'TOOL_CALL_CAP', 'step': step, 'tokens': used_tokens, 'tool_calls': tool_calls}
    return {'stop': 'STEP_CAP', 'step': max_steps, 'tokens': used_tokens, 'tool_calls': tool_calls}

# 1. Normal run: reaches a final answer within every budget.
print('normal :', run_agent('find the answer'))
# 2. A tight token budget trips first -- the loop stops cleanly and says why.
print('budget :', run_agent('find the answer', max_tokens=200))
# 3. A tight step cap trips before the model would ever answer.
print('stepcap:', run_agent('find the answer', max_steps=2, max_tokens=10000))

normal : {'stop': 'FINAL_ANSWER', 'step': 3, 'tokens': 450, 'tool_calls': 3, 'text': 'done: found the answer'}
budget : {'stop': 'TOKEN_BUDGET', 'step': 1, 'tokens': 240, 'tool_calls': 1}
stepcap: {'stop': 'STEP_CAP', 'step': 2, 'tokens': 240, 'tool_calls': 2}

The three runs are the whole point: the same agent stops for three different, named reasons, and every stop carries how far it got (step, tokens, tool calls). A monitored production agent emits exactly this — you alert on the ratio of FINAL_ANSWER to cap-hit exits, because a rising cap-hit rate means something is confusing your agent.

3 · Guardrails — gate input, tools, and output

Think about how a nightclub actually keeps trouble out. There's a bouncer at the door who turns away people who shouldn't come in. There's security inside who stops a fight before it starts. And there's a coat-check on the way out who makes sure nobody leaves with something that isn't theirs. Three checkpoints, three different jobs — because trouble can arrive at the entrance, erupt on the floor, or walk out the exit. One bouncer at the door can't catch all three. Guardrails are exactly this: checkpoints placed where harm can actually happen, not one big gate you hope covers everything.

Loop caps from §2 solve a different problem — they stop a run from running away. Guardrails stop a run from doing the wrong thing while it runs, and the two are complementary: a capped agent still needs guarding, because a run that's perfectly within budget can still be one that leaks a password or wipes a database in three tidy steps. So we place gates at the three boundaries where badness enters or leaves the system. The input boundary catches bad requests before they cost you a model call. The tool boundary catches dangerous actions in the gap between the model proposing a call and the code executing it — the last moment before something irreversible happens. The output boundary catches bad content on its way to the user, after the model has spoken but before anyone reads it.

Now the mistake that sinks most first attempts, and it's worth saying loudly: a guardrail is not a sentence in your prompt. "I told the system prompt never to reveal secrets" feels like a control, but it's a request made to a probabilistic system — and probabilistic systems can be talked out of things, jailbroken, or simply have an off day. A real guardrail is deterministic code sitting outside the model, in the one place the model cannot argue with: a plain if statement that refuses delete_database no matter how politely or cleverly the request was phrased. The prompt lowers how often the model tries something bad; the guardrail guarantees the attempt fails. You need both, but only one of them is a guarantee — and reliability is built on guarantees, not on hopes. They sit at three boundaries, because harm can enter or leave at any of them:

Here is a guardrail layer enforcing all three. The input gate refuses a request containing a banned term; the tool gate intercepts a dangerous tool the model proposed; the output gate redacts a secret that slipped into the answer. All three are plain rules here — a real system often adds a model-based classifier for fuzzier judgments, but the placement is identical:

guardrails.py# A guardrail layer with THREE gates around the agent:
#   input  -> block a disallowed request before the model ever runs,
#   tool   -> block a dangerous tool call the model proposes,
#   output -> validate/filter the final answer before it reaches the user.
# All checks are plain rules; a real system may add a model-based classifier.

class Blocked(Exception):
    pass

BANNED_INPUT = ('password', 'ssn', 'wire money')
DANGEROUS_TOOLS = {'delete_database', 'transfer_funds', 'run_shell'}

def input_guard(request):
    low = request.lower()
    for term in BANNED_INPUT:
        if term in low:
            raise Blocked(f'input refused: contains disallowed term {term!r}')
    return request

def tool_guard(name, args):
    if name in DANGEROUS_TOOLS:
        raise Blocked(f'tool refused: {name!r} is dangerous (needs human approval)')
    return name, args

def output_guard(answer):
    # Redact anything that looks like a secret before it leaves the system.
    import re
    redacted = re.sub(r'\bsk-[A-Za-z0-9]+', '[REDACTED_KEY]', answer)
    if len(redacted) > 200:
        redacted = redacted[:200] + ' ...'
    return redacted

def handle(request, proposed_tool, answer):
    try:
        input_guard(request)
        tool_guard(*proposed_tool)
    except Blocked as e:
        return {'status': 'blocked', 'reason': str(e)}
    return {'status': 'ok', 'answer': output_guard(answer)}

# 1. Clean request, safe tool, secret leaked in the answer -> runs, output redacted.
print(handle('what is our refund policy?', ('search_docs', {'q': 'refund'}),
             'Our policy is 30 days. (debug key sk-abc123XYZ)'))
# 2. Disallowed input -> blocked before the model runs at all.
print(handle('email me the admin password', ('search_docs', {}), 'irrelevant'))
# 3. Model proposes a dangerous tool -> tool guard intercepts it.
print(handle('clean up old records', ('delete_database', {'name': 'prod'}), 'irrelevant'))

{'status': 'ok', 'answer': 'Our policy is 30 days. (debug key [REDACTED_KEY])'}
{'status': 'blocked', 'reason': "input refused: contains disallowed term 'password'"}
{'status': 'blocked', 'reason': "tool refused: 'delete_database' is dangerous (needs human approval)"}

Three requests, three outcomes: one runs but has its output sanitized, one is refused at the input boundary before wasting a model call, one is refused at the tool boundary before touching the database. The guardrails are ordinary code — deterministic, testable, cheap — sitting outside the probabilistic model. That is the design principle: put the safety where you can guarantee it.

4 · Determinism & reproducibility — the honest version

You've spent your whole programming life with a comforting guarantee: run the same function on the same input, get the same output, every single time. That's what makes a bug reproducible, a test trustworthy, a system debuggable. Now here's the news that quietly unsettles most engineers the first time they hit it: an LLM breaks that guarantee. Feed it the identical prompt twice and you can get two different answers — not because you made a mistake, but because that's genuinely how the thing behaves. If your reliability plan secretly assumed "I'll just re-run it and reproduce the problem," that plan has a hole in it, and this section is about how to build around the hole instead of pretending it isn't there.

The instinct is to reach for the knob labelled temperature=0 and assume it makes the model deterministic. It doesn't — and understanding why stops you from trusting a false floor. Setting temperature to zero tells the model to always pick its single most likely next token, which removes the deliberate randomness. But variance still leaks in from underneath: floating-point arithmetic isn't perfectly associative, so the same numbers added in a slightly different order on a different GPU (or in a different batch alongside other users' requests) can round differently, and one flipped token early can cascade into a different sentence. On top of that, providers silently update models behind a friendly alias like "latest." So temperature=0 gives you less wandering, not no wandering — a lower floor, not a locked door.

The mistake, then, is to keep chasing perfect reproducibility — tuning seeds, pinning flags, hoping the next setting finally makes it bit-for-bit stable. You'll lose that fight, and worse, you'll build your reliability story on a foundation that was never solid. The honest reframe is this: the goal is not "eliminate variance" but "minimize the variance you can, and log enough that you can explain and replay any run you couldn't." Think of it like a black-box flight recorder. You can't stop turbulence, and you can't guarantee two flights are identical — but if every run records its exact inputs, each decision the model made, every tool call and result, and the final output, then when something goes wrong you can reconstruct precisely what happened and see where it diverged. Reliability rests on the trace, not on determinism — which is exactly why the tracing in §5 is the backbone of this whole chapter. The table below is the honest scorecard of what each lever buys you, and where each one stops.

5 · Observability — the trace is the unit of debugging

Imagine a detective handed only a dead body and asked to solve the murder — no timeline, no witnesses, no CCTV, just the final result. That's what debugging an agent from its answer alone feels like. The agent said "the population of Paris is 40 million," which is wrong, and… now what? The wrong answer is the corpse. It tells you a crime occurred but nothing about where, when, or how the run went off the rails. Was the geocoding tool given the wrong city? Did a lookup return stale data? Did the model ignore a correct result and hallucinate anyway? You genuinely cannot tell, because the failure lives somewhere in the middle of the run and the answer only shows you the end.

The fix is to record the timeline. The unit of agent debugging isn't the answer — it's the trace: a structured, ordered record of every step the agent took, each thought, each tool call with the exact arguments it used, each result that came back, plus the tokens and latency that step cost. A trace turns the black box into a glass box. With one in hand you can read the run top to bottom like a transcript and watch the agent "think out loud" after the fact, so the broken step stops hiding — if it called the same tool five times, or answered before a lookup returned, the trace shows it at a glance. This is the same shift good engineers already know from ordinary systems: you don't debug a web service by staring at the 500 error, you read the logs. The trace is the agent's log, and it's the difference between "the run failed, no idea why" and "step 3 called geocode with an empty city."

The subtle mistake here is treating the trace as a printout instead of as data. It's tempting to just print() a nice human-readable line at each step and call it observability — and for a single run on your laptop, that's fine. But in production you have thousands of runs, and the questions you need to answer are aggregate ones: "show me every run that touched charge_card," "what's the 95th-percentile tool-call count," "which runs failed the goal check." You can't grep your way to those answers reliably. So you record each step as a structured object — a typed, attributed span you can query, aggregate, and feed to the evaluator in §6 — not a formatted string. Notice the beautiful economy this creates: the very same per-step record is what the budget caps in §2 count against, what you read to debug, and what the trajectory evaluator scores. One data structure, three jobs. Build it once, well, and the rest of reliability engineering has something solid to stand on.

Here is a tiny tracer. A real agent calls tracer.record() at each step of its loop; we replay a fixed, deterministic trajectory (a two-tool lookup) so the numbers are stable. It prints a readable trace and a run summary — steps, tool calls, and simulated token/latency totals:

tracer.py# A tiny TRACER: record every step of a run (thought, tool, args, result,
# simulated tokens + latency), then print a readable trace and a summary.
# Numbers are simulated fixed values -- no wall-clock, no randomness.

class Tracer:
    def __init__(self):
        self.spans = []
    def record(self, kind, name, detail, sim_tokens, sim_ms):
        self.spans.append({'kind': kind, 'name': name, 'detail': detail,
                           'tokens': sim_tokens, 'ms': sim_ms})
    def summary(self):
        steps = len(self.spans)
        tool_calls = sum(1 for s in self.spans if s['kind'] == 'tool')
        tokens = sum(s['tokens'] for s in self.spans)
        ms = sum(s['ms'] for s in self.spans)
        return {'steps': steps, 'tool_calls': tool_calls, 'sim_tokens': tokens, 'sim_ms': ms}
    def show(self):
        for i, s in enumerate(self.spans):
            print(f"  [{i}] {s['kind']:7} {s['name']:14} {s['detail']:32} "
                  f"tok={s['tokens']:>3} ms={s['ms']:>3}")

# Simulate one agent run over the tracer. A real agent calls tracer.record()
# at each step of its loop; here we replay a fixed, deterministic trajectory.
def run_traced():
    t = Tracer()
    t.record('thought', 'plan',   'need city then its population',   40, 20)
    t.record('tool',    'geocode', "args={'city':'Paris'}",          25, 60)
    t.record('thought', 'reflect', 'have coords, now population',     35, 20)
    t.record('tool',    'lookup',  "args={'id':'FR-75'}",            25, 55)
    t.record('answer',  'final',   'Paris population is 2,100,000',   50, 25)
    return t

tracer = run_traced()
print('TRACE:')
tracer.show()
print('SUMMARY:', tracer.summary())

TRACE:
  [0] thought plan           need city then its population    tok= 40 ms= 20
  [1] tool    geocode        args={'city':'Paris'}            tok= 25 ms= 60
  [2] thought reflect        have coords, now population      tok= 35 ms= 20
  [3] tool    lookup         args={'id':'FR-75'}              tok= 25 ms= 55
  [4] answer  final          Paris population is 2,100,000    tok= 50 ms= 25
SUMMARY: {'steps': 5, 'tool_calls': 2, 'sim_tokens': 175, 'sim_ms': 180}

Read the trace top to bottom and you can see the reasoning: plan, look up coordinates, reflect, look up population, answer. If the agent had called geocode five times, or answered before the lookup returned, the trace would show it immediately. The summary is what you'd chart per run — steps, tool calls, and cost — and it's the same data the budget caps in §2 enforce and the evaluator in §6 scores. One record, three uses.

6 · Evaluating agent trajectories

Two students hand in the same correct final answer on a math exam. One showed clean, logical working; the other scribbled six wrong approaches, crossed them out, and stumbled onto the right number by luck. Do they deserve the same grade? Any teacher would say no — because the process tells you which student will get it right again next time and which one got lucky once. Grading an agent has exactly this problem. If you only check the final answer, the lucky guesser and the sound reasoner look identical, and you'll happily ship the one that's going to fail the moment the question changes slightly.

This is why evaluating an agent is genuinely harder than evaluating a chatbot, and it's a jump people underestimate. A one-shot chatbot has one thing to grade: does its answer match the reference? Easy. But an agent produces a path — a sequence of tool calls, results, and reasoning steps — and two paths can land on the same answer while being wildly different in quality. One run cleanly geocodes a city then looks up its population; another thrashes through three guesses, ignores a tool result, blows most of its budget, and arrives at a plausible-looking number by accident. Same last line, completely different reliability. So the object you evaluate is the whole trajectory, not just the final sentence.

The trap is vagueness — "was it a good run?" is a feeling, not a test, and feelings don't run in CI. The move that makes trajectory evaluation practical is to decompose that fuzzy judgment into concrete, checkable questions that a plain function can answer over a recorded trace: Did it use the right tools (not some tool that couldn't possibly answer the question)? In a sane order (a lookup after the geocode that feeds it, not before)? Did it actually reach the goal (a real success marker, not just an answer)? And did it stay within budget (steps and tool calls under their caps)? Each of these is deterministic — same trace in, same verdict out — which means you can run them over a whole suite of recorded runs and get a regression test for behavior, not just for answers. Change a prompt or a tool, re-run the suite, watch the pass rate: that's how you stop a "harmless" tweak from silently teaching your agent to thrash. The evaluator below scores one good trajectory and one bad one so you can see both verdicts side by side.

Here is a trajectory evaluator. It takes a recorded trace (the same shape the tracer in §5 produces) and scores it on three deterministic checks — expected tool set, goal reached, within budget — returning a pass/fail with a reason for each. We score a good trajectory and a bad one so you can see both:

trajectory.py# Evaluate a whole TRAJECTORY, not just the final answer. We score three
# things: did it use the expected tools, did it reach the goal, did it stay
# in budget? Each check is a deterministic rule over a recorded trace.

def evaluate(trace, expected_tools, goal_marker, max_steps, max_tool_calls):
    used_tools = [s['tool'] for s in trace if s['kind'] == 'tool']
    tool_calls = len(used_tools)
    steps = len(trace)
    reached = any(goal_marker in s.get('text', '') for s in trace if s['kind'] == 'answer')

    reasons = []
    # 1. right tool set (order-independent: did it use exactly what was expected?)
    tools_ok = set(used_tools) == set(expected_tools)
    reasons.append(('expected_tools', tools_ok,
                    f'used {used_tools}, expected {sorted(expected_tools)}'))
    # 2. reached the goal
    reasons.append(('goal_reached', reached, f'goal_marker={goal_marker!r}'))
    # 3. within budget
    budget_ok = steps <= max_steps and tool_calls <= max_tool_calls
    reasons.append(('within_budget', budget_ok,
                    f'{steps} steps / {tool_calls} tool calls '
                    f'(caps {max_steps}/{max_tool_calls})'))

    passed = all(ok for _, ok, _ in reasons)
    return passed, reasons

# A GOOD trajectory: uses both expected tools, reaches the goal, in budget.
good = [
    {'kind': 'tool',   'tool': 'geocode'},
    {'kind': 'tool',   'tool': 'lookup'},
    {'kind': 'answer', 'text': 'Paris population is 2,100,000 [done]'},
]
# A BAD trajectory: wrong tool, never reached the goal, over the tool cap.
bad = [
    {'kind': 'tool',   'tool': 'geocode'},
    {'kind': 'tool',   'tool': 'guess'},
    {'kind': 'tool',   'tool': 'guess'},
    {'kind': 'answer', 'text': 'I think it is somewhere around a lot'},
]

for label, tr in [('GOOD', good), ('BAD', bad)]:
    passed, reasons = evaluate(tr, expected_tools={'geocode', 'lookup'},
                               goal_marker='[done]', max_steps=5, max_tool_calls=2)
    print(f'{label}: {"PASS" if passed else "FAIL"}')
    for name, ok, detail in reasons:
        print(f'   {"ok " if ok else "X  "} {name:15} {detail}')

GOOD: PASS
   ok  expected_tools  used ['geocode', 'lookup'], expected ['geocode', 'lookup']
   ok  goal_reached    goal_marker='[done]'
   ok  within_budget   3 steps / 2 tool calls (caps 5/2)
BAD: FAIL
   X   expected_tools  used ['geocode', 'guess', 'guess'], expected ['geocode', 'lookup']
   X   goal_reached    goal_marker='[done]'
   X   within_budget   4 steps / 3 tool calls (caps 5/2)

The bad run fails on all three axes — wrong tools, goal never reached, over the tool budget — and the reasons tell you which so you can fix the right thing. Run this over a suite of recorded trajectories and you have an automated agent-eval harness: a regression test for behavior, not just for answers. When you change a prompt or a tool, you re-run the suite and watch the pass rate.

7 · Failure taxonomy & mitigations

Every experienced pilot carries a checklist not because they're forgetful, but because "I'll remember the important stuff" is exactly how disasters happen. The checklist is the accumulated wisdom of everyone who crashed before — each item is a specific failure someone once didn't catch. Reliability engineering for agents works the same way: the difference between a beginner and a veteran isn't intelligence, it's that the veteran has a named list of the ways an agent goes wrong and a rehearsed response for each one. A failure you can name is a failure you can anticipate, detect, and mitigate; a failure you've never thought about is the one that pages you at 3 a.m.

That's what this section is — the field guide, the pre-flight checklist for agents. Every row below is something you will eventually see in a system that runs long enough, not a hypothetical. And here's the payoff that ties the whole chapter together: notice that the mitigation column doesn't introduce anything new. It's the toolkit you already built — the loop caps from §2, the guardrails from §3, the trace from §5, the trajectory checks from §6, plus the tool discipline from the neighbouring chapters. The point of a taxonomy isn't to memorize failures for their own sake; it's to see that each failure mode maps to a control you now know how to build, so that "reliability" stops being a vague worry and becomes a concrete list of guards you can check off one at a time.