Robust tools & structured I/O

1 · Tool design — the schema is the prompt

Start with the failure you'll actually hit. You give your agent a tool called handler described as "processes requests," and in testing it works — because you know what it's for and you steer it. Then a real user asks something slightly off-script, the model looks at that vague name, shrugs, and either calls the tool with garbage arguments or ignores it entirely and hallucinates an answer instead. Nothing crashed. The tool was just unusable, and you won't find out until a support ticket lands. That whole class of bug is designed away before you write a line of tool logic — at the schema.

Here's the mental shift that fixes it: a tool schema is not an API doc, it's a job posting the model reads to decide whether to apply. Imagine hiring a contractor by handing them a one-line ad. "Looks stuff up" tells them nothing — they'll knock on the wrong door. "Return the shipping status of a single order given its id; use this when a customer asks where their package is" tells them exactly when to step forward and exactly what they need in hand. The model reads your tool the same way a candidate reads that ad: the name, the description, and every parameter are the only information it has about when this tool is the right one and what a correct call looks like. It never sees your source code.

The common mistake is writing the description for yourself — terse, jargon-heavy, assuming context the model doesn't have — because you already know what the tool does. But the model has no memory of your architecture, no sense of your naming conventions, and no way to ask a clarifying question mid-call. Every ambiguity you leave becomes a coin-flip: which of my five vaguely-named tools did the user mean, is q a search string or an order id, is this the tool for one order or all orders? Precision in the schema is not politeness — it is the mechanism by which the model routes correctly at all. Now the precise version of the idea:

Here is the single most important idea in this chapter: the tool's name and description are not documentation for you — they are the prompt the model reads to decide whether, when, and how to call it. A vague schema makes the model guess; a precise one makes the right call obvious. Good tool design is prompt engineering aimed at the model's tool-selection step.

To make “what the model sees” concrete, here we render the same tool two ways and print the text the model actually consumes. The bad version is untyped and unexplained; the good one is unambiguous:

schema.py# The tool SCHEMA is the prompt the model reads to decide when/how to call it.
# A vague name and a fuzzy description make the model guess. Here we render
# the same tool two ways and show what the model effectively "sees".

bad = {
    'name': 'do_lookup',
    'description': 'looks stuff up',
    'parameters': {'q': 'string', 'opt': 'string'},   # untyped, unexplained
}

good = {
    'name': 'get_order_status',
    'description': ('Return the current status of ONE order by its id. '
                    'Use when the user asks where an order is or if it shipped. '
                    "Example: get_order_status(order_id='A-1001')."),
    'parameters': {
        'order_id': {'type': 'string', 'required': True,
                     'description': "Order id like 'A-1001'."},
    },
}

def render(schema):
    lines = [f"{schema['name']}: {schema['description']}"]
    for pname, pinfo in schema['parameters'].items():
        if isinstance(pinfo, dict):
            req = 'required' if pinfo.get('required') else 'optional'
            lines.append(f"  - {pname} ({pinfo['type']}, {req}): {pinfo['description']}")
        else:
            lines.append(f'  - {pname} ({pinfo}, ?): ?')
    return '\n'.join(lines)

print('WHAT THE MODEL SEES (bad):')
print(render(bad))
print('\nWHAT THE MODEL SEES (good):')
print(render(good))

WHAT THE MODEL SEES (bad):
do_lookup: looks stuff up
  - q (string, ?): ?
  - opt (string, ?): ?

WHAT THE MODEL SEES (good):
get_order_status: Return the current status of ONE order by its id. Use when the user asks where an order is or if it shipped. Example: get_order_status(order_id='A-1001').
  - order_id (string, required): Order id like 'A-1001'.

2 · Input validation — never trust the model's args

Picture the moment right before your tool runs. The model has decided to call set_temperature, and it hands you {'celsius': '250'}. Two things are already wrong: 250 is a string, not a number, and 250°C would cook whatever you're heating. If your tool just does float(args['celsius']) and pushes the value to a real device, you have handed a probabilistic text generator direct control of the physical world with zero checks. This is not a hypothetical — models emit stringified numbers, skip required fields, invent enum values, and occasionally conjure whole arguments that were never in your schema. The question is never if the args will be wrong, only when.

The reframe that makes this click: treat the model exactly like an anonymous user submitting a web form. You would never take a raw HTTP request body and run it straight against your database — you validate types, check ranges, reject nonsense, and only then act. The tool boundary is that same boundary. It doesn't matter that "your own" model produced the args; from the tool's point of view they are untrusted input arriving from outside, and the tool is the last line of defense before something real happens. A friendly, capable assistant is still an untrusted source of arguments.

Here's the part people get subtly wrong even after they accept "validate the args": they validate by crashing. A bad argument raises an exception, the exception escapes, and the whole agent loop dies — so a recoverable mistake (the model guessed days=99) becomes an outage. The skill is to fail into a value, not out of the program: coerce what is safe to coerce (the string '3' clearly means the int 3), reject what isn't with a message specific enough to act on, and hand that rejection back as data. Do it in that order — coerce, then check ranges and choices, then fill defaults — so the model gets a clean, precise "here's what was wrong" it can correct on the next turn, instead of silence or a stack trace.

The model chooses your tool's arguments, and it can get them wrong: a number as a string, an out-of-range value, a missing field, or an argument it simply hallucinated. Your tool boundary is an untrusted input boundary exactly like an HTTP request handler. Validate and coerce every argument before you execute; reject clearly so the error can flow back as a tool result the model can fix.

Real code uses pydantic or a JSON-schema validator here. To keep this runnable offline we use a small stdlib schema-dict validator that checks types, coerces a stringified int, applies range/choice rules, and fills defaults:

validate.py# A validate-and-coerce wrapper: NEVER trust the model's tool args.
# Real code uses pydantic / JSON-schema; this is a stdlib stand-in that
# checks types, ranges, and required fields, coercing where it safely can.

SCHEMA = {
    'city':  {'type': str, 'required': True},
    'days':  {'type': int, 'required': True, 'min': 1, 'max': 14},
    'units': {'type': str, 'required': False, 'default': 'celsius',
              'choices': ['celsius', 'fahrenheit']},
}

class ToolArgError(Exception):
    pass

def validate(args, schema):
    clean = {}
    for name, rule in schema.items():
        if name not in args:
            if rule.get('required'):
                raise ToolArgError(f'missing required arg: {name!r}')
            clean[name] = rule.get('default')
            continue
        val = args[name]
        want = rule['type']
        # coerce a stringified number the model may have emitted ('3' -> 3)
        if want is int and isinstance(val, str):
            try:
                val = int(val)
            except ValueError:
                raise ToolArgError(f'{name!r} must be an int, got {val!r}')
        if not isinstance(val, want):
            raise ToolArgError(f'{name!r} must be {want.__name__}, got {type(val).__name__}')
        if 'min' in rule and val < rule['min']:
            raise ToolArgError(f'{name!r}={val} below min {rule["min"]}')
        if 'max' in rule and val > rule['max']:
            raise ToolArgError(f'{name!r}={val} above max {rule["max"]}')
        if 'choices' in rule and val not in rule['choices']:
            raise ToolArgError(f'{name!r}={val!r} not in {rule["choices"]}')
        clean[name] = val
    return clean

# The model emitted these tool args. Note: days is the STRING '3' (it happens),
# and there is no 'units' key at all.
model_args = {'city': 'Berlin', 'days': '3'}
print('clean:', validate(model_args, SCHEMA))

# Now a hallucinated / out-of-range arg the model made up:
bad = {'city': 'Berlin', 'days': 99}
try:
    validate(bad, SCHEMA)
except ToolArgError as e:
    print('rejected:', e)

clean: {'city': 'Berlin', 'days': 3, 'units': 'celsius'}
rejected: 'days'=99 above max 14

The good args are cleaned: the string '3' becomes the int 3, and the missing units gets its default. The bad call is rejected with a message specific enough that the model — or a human — can correct it. Coerce what is safe, reject what is not, and always default the optional.

3 · Structured outputs — reliable JSON out of the model

Here's the tax you pay for treating a model's output as text. You ask for JSON so you can store it in a database or pass it to the next tool, and 95% of the time you get clean JSON. But the model is a helpful machine, and helpfulness is the enemy here: it prepends "Sure! Here's the analysis:", wraps the object in a ```json code fence, adds a trailing "Let me know if you'd like more detail!", or renames your score key to confidence because that felt more natural. Your json.loads() throws, or worse, it succeeds and the shape is subtly wrong. At scale that 5% is thousands of broken records, and the failures are non-deterministic — the same prompt fails on Tuesday and works on Wednesday.

The useful analogy is a form-fill vs. an essay. When you ask a person to "describe the customer's sentiment," you get an essay — prose, framing, caveats, personality. When you hand them a form with labelled boxes — sentiment: ___, score (0–1): ___, topics: ___ — you get back exactly the fields you need and nothing else. Structured output is you handing the model the form instead of the essay prompt. The mistake beginners make is asking for the essay ("please return JSON") and then being surprised it reads like an essay. The model does what its training rewards: it communicates. You have to actively constrain it back into the boxes.

There are two levers, and the order matters. The primary lever is provider-side: modern APIs offer a structured-output or JSON mode that constrains the model's generation to your schema at the token level, so a code fence or stray prose literally cannot be produced. Reach for that first — it kills most failures at the source. The secondary lever, for the rare remainder and for providers without the feature, is a defensive loop on your side: parse the text, validate it against the expected shape, and if it fails, re-ask the model with the specific error appended so it can self-correct. Belt and suspenders. Never assume the JSON is well-formed just because you asked nicely; validate the shape (keys and types), not just that it parsed, because "valid JSON with the wrong keys" is the failure that slips past a naive check.

Often you need the model to return machine-readable JSON, not prose — to route it, store it, or feed it to another tool. Left to freeform text, a model will wrap JSON in a code fence, add “Sure! Here is…”, or drift from your keys. Two real levers fix this: (1) provider structured-output / JSON mode that constrains generation to a schema, and (2) a parse → validate → repair/re-ask loop on your side. Use both; the loop below is the safety net.

Here we simulate two model replies (fixed strings). The first is a typical first-try failure — valid intent wrapped in a code fence, so json.loads chokes. On failure we would re-ask with the error appended; the second reply is clean and passes:

structured.py# Getting reliable JSON out of a model. Real APIs offer schema-constrained /
# 'structured output' modes; here we SIMULATE the model's reply as a fixed
# string, then parse + validate it and RE-ASK once if it doesn't conform.
import json

EXPECTED = {'sentiment': str, 'score': float, 'topics': list}

def validate_json(text, expected):
    """Parse text as JSON and check it matches the expected shape.
    Returns (ok, value_or_error)."""
    try:
        obj = json.loads(text)
    except json.JSONDecodeError as e:
        return False, f'not valid JSON: {e.msg}'
    if not isinstance(obj, dict):
        return False, 'top level is not an object'
    for key, typ in expected.items():
        if key not in obj:
            return False, f'missing key: {key!r}'
        if not isinstance(obj[key], typ):
            return False, f'{key!r} should be {typ.__name__}'
    return True, obj

# Two SIMULATED model replies. The first is what a model often does on the
# first try: wraps JSON in prose / a code fence. The second is clean (the
# reply after we re-ask with a stricter instruction).
reply_1 = 'Sure! Here is the analysis:\n```json\n{"sentiment": "negative"}\n```'
reply_2 = '{"sentiment": "negative", "score": 0.82, "topics": ["latency", "billing"]}'

def ask_model(attempt):
    return reply_1 if attempt == 0 else reply_2

result = None
for attempt in range(2):
    ok, val = validate_json(ask_model(attempt), EXPECTED)
    print(f'attempt {attempt}: ok={ok} -> {val}')
    if ok:
        result = val
        break
    # repair step: on failure we re-ask the model with the error appended
    # so it can correct itself. Here attempt 1 returns clean JSON.

print('final:', result)

attempt 0: ok=False -> not valid JSON: Expecting value
attempt 1: ok=True -> {'sentiment': 'negative', 'score': 0.82, 'topics': ['latency', 'billing']}
final: {'sentiment': 'negative', 'score': 0.82, 'topics': ['latency', 'billing']}

4 · Parallel vs dependent tool calls

Two agents, same six tool calls, wildly different experiences. The first runs every call one after another, waiting for each to finish before starting the next — the user stares at a spinner for four seconds. The second notices that four of those calls don't need each other at all and fires them together, finishing in one second. Same work, same result, one feels broken and one feels instant. Latency in an agent is rarely the model thinking; it's tool calls waiting in a needless line. But go the other way — parallelize a call that secretly depends on another — and you get a race: the second tool runs before the data it needs exists, and you ship a confidently wrong answer.

The intuition is a kitchen on a busy night. You can boil the pasta and warm the sauce at the same time because neither needs the other — those are independent. But you cannot plate the dish before the pasta is cooked; plating depends on boiling being done. A good cook does everything independent in parallel and sequences only what genuinely must wait. Your agent's tool calls split the same way: "get the weather" and "get the user profile" are the pasta and the sauce (fire together), while "summarize this user's orders" is the plating — it needs the user's id from an earlier call before it can start.

The trap is deciding parallelism by vibes instead of by data flow. The honest test is one question per call: does this call need the output of another call? If yes, it's dependent and must be sequenced after it. If no, it's independent and can join a parallel group. When you're genuinely unsure whether B needs A's result, sequence them — a wrong-but-fast answer is worse than a right-but-slightly-slower one. The clean way to compute this without hand-waving is to model each call as a node listing its dependencies and peel off "everything whose dependencies are already satisfied" layer by layer. That's a topological sort — the exact algorithm a build system uses to decide what to compile in parallel — and as a bonus it detects the pathological case where two calls depend on each other and can therefore never run.

A model may request several tool calls at once. Some are independent — “get the weather” and “get the user profile” have nothing to do with each other and can run together. Others are dependent — “summarize the user's recent orders” needs the user's id first. Running dependents out of order gives you a race or a wrong answer; running independents one-at-a-time is just slow.

You don't need real async to reason about this. Model each call as a node with a list of dependencies and compute parallel groups: every call in a group has all its dependencies satisfied by an earlier group. (This is a topological sort — the same idea a build system uses.) We simulate execution and print the groups:

deps.py# A tiny dependency resolver. Some tool calls are INDEPENDENT (can run
# 'together'); some DEPEND on another's output and must be sequenced.
# No real async here — we simulate execution and show the parallel GROUPS.

# Each tool call: an id, the tool, and the ids it depends on.
calls = [
    {'id': 'A', 'tool': 'get_user',    'deps': []},
    {'id': 'B', 'tool': 'get_weather', 'deps': []},          # independent of A
    {'id': 'C', 'tool': 'get_orders',  'deps': ['A']},       # needs the user
    {'id': 'D', 'tool': 'summarize',   'deps': ['B', 'C']},  # needs both
]

def plan_groups(calls):
    """Return a list of GROUPS; every call in a group can run in parallel
    because all its deps were satisfied by an earlier group (Kahn-style)."""
    by_id = {c['id']: c for c in calls}
    done, groups = set(), []
    remaining = list(by_id)
    while remaining:
        ready = [cid for cid in remaining
                 if all(d in done for d in by_id[cid]['deps'])]
        if not ready:
            raise ValueError('cycle or missing dependency')
        ready.sort()                 # deterministic order within a group
        groups.append(ready)
        done.update(ready)
        remaining = [c for c in remaining if c not in done]
    return groups

groups = plan_groups(calls)
for i, g in enumerate(groups, 1):
    kind = 'parallel' if len(g) > 1 else 'single'
    print(f'group {i} ({kind}): {", ".join(g)}')

group 1 (parallel): A, B
group 2 (single): C
group 3 (single): D

Group 1 runs A and B together (both have no deps). C waits for A; D waits for both B and C. In a real system you'd launch each group with a thread pool or async gather and only advance when it finishes. The resolver also catches cycles — if two calls depended on each other, ready would be empty and we raise instead of looping forever.

5 · Tool result handling

The bug that teaches this section is a double charge. An agent calls charge_card, the network hiccups, the call times out, so the agent retries — and now the customer has been billed twice. Or a different flavor: a tool dumps a 50 KB database row back into the conversation, the useful field is buried in noise, the context window fills up, and the model's answer quietly degrades. Or the quietest failure of all: a tool hits an error, returns an empty string, and the model — seeing nothing wrong — confidently tells the user their order shipped. Everything you carefully validated on the way in can still fall apart on the way out.

The mental model is that a tool result is a message you're writing to the model, and the model reads it as literally true. It has no way to know that the empty string meant "the API was down" rather than "there are no results," or that the giant blob has one relevant line in it. Whatever you hand back is the model's reality for the next step. So the job of result handling is to make that message honest, compact, and safe to re-send: say what actually happened, say it in as few tokens as carry the meaning, and never let the same message trigger the same side effect twice.

Three habits cover most of it. Truncate or summarize before returning — the model needs the signal, not the 50 KB; a short synopsis ("312 rows, first: …") is often more useful than a raw slice that cuts off mid-token. Return errors as structured data, not exceptions — a tool that throws kills the loop and teaches the model nothing, whereas {'status': 'error', 'error': 'gateway timeout after 30s'} lets it reason about whether to retry, try another tool, or tell the user the truth. And make effectful tools idempotent: key each action by something stable (the order id), remember what you've already done, and serve a retry from that memory so "charge order-42" runs its effect exactly once no matter how many times the flaky loop asks. The unifying trick you'll see in the lab: give every outcome — success, truncation, error — the same small result shape, so the model always knows how to read what comes back.

What comes back from a tool matters as much as what goes in. Three habits keep the loop healthy: truncate or summarize large results before feeding them back (a 50 KB blob wastes context and buries the signal), return errors as structured data the model can reason about (not a raw stack trace or a silent empty string), and make retried tools idempotent so a re-run doesn't act twice.

results.py# Handling tool RESULTS before they go back to the model:
#  (1) truncate/summarize large output,
#  (2) return errors as STRUCTURED results the model can reason about,
#  (3) idempotency so a retried tool doesn't act twice.

MAX_CHARS = 80

def wrap_result(tool, ok, data):
    """Normalize every tool result into the same small dict the model reads."""
    if not ok:
        return {'tool': tool, 'status': 'error', 'error': data}
    text = str(data)
    if len(text) > MAX_CHARS:
        text = text[:MAX_CHARS] + f'... [truncated, {len(str(data))} chars total]'
    return {'tool': tool, 'status': 'ok', 'result': text}

# A large successful result gets truncated before it wastes the context budget:
big = 'customer record ' * 200
print(wrap_result('db_query', True, big))

# A failure becomes a structured error the model can act on (retry / apologize):
print(wrap_result('charge_card', False, 'gateway timeout after 30s'))

# Idempotency: a retried tool with the same key must NOT run its effect twice.
_seen = {}
def run_once(key, effect):
    if key in _seen:
        return {'status': 'cached', 'result': _seen[key]}
    _seen[key] = effect()
    return {'status': 'ran', 'result': _seen[key]}

charge = lambda: 'charged $50'
print(run_once('order-42', charge))   # first call runs the effect
print(run_once('order-42', charge))   # retry is served from cache, no double charge

{'tool': 'db_query', 'status': 'ok', 'result': 'customer record customer record customer record customer record customer record ... [truncated, 3200 chars total]'}
{'tool': 'charge_card', 'status': 'error', 'error': 'gateway timeout after 30s'}
{'status': 'ran', 'result': 'charged $50'}
{'status': 'cached', 'result': 'charged $50'}

Notice the shape: every result — success, truncated, or error — comes back in the same small dict, so the model always knows how to read it. The idempotency key means the second run_once returns the cached result with status: 'cached' and never charges twice, which is exactly what you want when an agent retries a flaky call.

6 · Sandboxing — safe tool execution

Everything so far has been about the model getting things wrong by accident. This section is about what happens when it does — and about the day someone gets it wrong on purpose. A tool is the exact point where a stream of tokens becomes an action in the real world: a file deleted, a card charged, a production database touched. If the only thing standing between a hallucinated rm -rf / and your disk is the hope that the model behaves, you don't have a safety story — you have a bet. And prompt injection makes it worse: a hostile string buried in a document the agent reads can try to steer it into calling your most dangerous tool. Betting on good behavior loses eventually.

The principle that saves you is one you already trust everywhere else: you don't give the new intern the master key to the building. You give them a badge that opens the two rooms they need, and nothing else. If they make a mistake — or someone tricks them — the blast radius is those two rooms, not the whole company. Safety lives in what the key can open, not in trusting the person to only open the right doors. Put the safety in the tool layer, where it's enforced mechanically, rather than in the prompt, where it's merely requested.

The mistake that quietly sinks agents is default-allow: build a tool that can do anything, then try to talk the model out of misusing it. That inverts the burden — every dangerous path is open unless you remembered to close it. Flip it to default-deny and grant narrowly, and the four controls below fall out naturally. Least privilege: each tool holds only the credentials for its one job (a read tool physically cannot write). Allow-list: constrain what it may touch — paths, hosts, accounts — so delete_file can only reach /tmp/. Dry-run / confirm: destructive actions are simulated until something explicitly proves intent, so a delete has to be both confirmed and taken out of dry-run to actually happen. Timeouts: a wall-clock cap so one hung call can't freeze the agent. Together they mean a model that goes off the rails hits a wall, not your infrastructure. Four tool-level controls carry most of the weight (deeper human-in-the-loop and org security come later, in 4.4 and 4.5):

Here is a declarative policy gate that enforces the first three (timeouts are a wrapper concern we note rather than simulate, since the labs are deterministic). Every tool call passes through guard(); an unknown tool, a path outside its allow-list, or a destructive action left in dry-run is refused:

sandbox.py# Tool-execution safety. A tool can delete files, spend money, or hit prod.
# Give each tool LEAST PRIVILEGE, gate destructive actions behind a confirm /
# dry-run, and allow-list what it may touch. (Timeouts noted in prose.)

# Declarative policy: what each tool is allowed to do.
POLICY = {
    'read_file':   {'destructive': False, 'paths': ['/data/']},
    'delete_file': {'destructive': True,  'paths': ['/tmp/']},
    'send_email':  {'destructive': True,  'paths': []},
}

class Denied(Exception):
    pass

def guard(tool, target=None, *, confirm=False, dry_run=True):
    rule = POLICY.get(tool)
    if rule is None:
        raise Denied(f'tool {tool!r} is not on the allow-list')
    # path allow-list: the target must sit under an allowed prefix
    if rule['paths'] and target is not None:
        if not any(target.startswith(p) for p in rule['paths']):
            raise Denied(f"{tool!r} may not touch {target!r} (allowed: {rule['paths']})")
    # destructive actions must be confirmed AND not left in dry-run
    if rule['destructive'] and not (confirm and not dry_run):
        return {'tool': tool, 'target': target, 'action': 'SKIPPED (needs confirm, not dry-run)'}
    return {'tool': tool, 'target': target, 'action': 'EXECUTED'}

# 1. safe read under an allowed path:
print(guard('read_file', '/data/report.csv'))
# 2. destructive delete but still a dry-run -> not executed:
print(guard('delete_file', '/tmp/scratch.log', confirm=True, dry_run=True))
# 3. same delete, confirmed and dry-run off -> executes:
print(guard('delete_file', '/tmp/scratch.log', confirm=True, dry_run=False))
# 4. path outside the allow-list -> denied:
try:
    guard('delete_file', '/etc/passwd', confirm=True, dry_run=False)
except Denied as e:
    print('DENIED:', e)
# 5. a tool nobody granted -> denied:
try:
    guard('run_shell', 'rm -rf /')
except Denied as e:
    print('DENIED:', e)

{'tool': 'read_file', 'target': '/data/report.csv', 'action': 'EXECUTED'}
{'tool': 'delete_file', 'target': '/tmp/scratch.log', 'action': 'SKIPPED (needs confirm, not dry-run)'}
{'tool': 'delete_file', 'target': '/tmp/scratch.log', 'action': 'EXECUTED'}
DENIED: 'delete_file' may not touch '/etc/passwd' (allowed: ['/tmp/'])
DENIED: tool 'run_shell' is not on the allow-list