Production agent systems

1 · A representative production architecture

A deployed agent is never just “the loop.” The loop from 4.1–4.4 sits at the centre, but it is wrapped in layers that make it safe to point at real users and real systems. The shape below is representative — a synthesis of common public patterns, not any single company's design — and you adapt it to your risk and scale.

The horizontal path is the request lifecycle; three cross-cutting concerns wrap the whole thing:

2 · Orchestration — one loop or many agents?

As scope grows, the tempting move is to split the work across several specialised agents coordinated by a supervisor. Sometimes that is right; often a single well-scoped loop with good tools (4.3) and routing (4.2) is simpler, cheaper, and easier to debug. The decision is about independence and context, not about how impressive it looks.

3 · Human-in-the-loop — approval gates for risky actions

Some actions are too costly to be wrong: spending money, deleting data, emailing a customer, changing a production config. For these, the agent should not act autonomously — it should pause and hand off to a person who approves, edits, or rejects. This is human-in-the-loop (HITL). It is different from the tool-level dry-run/confirm of 4.3: that stops a hallucinated call at the tool boundary; HITL puts a human decision in the workflow before a genuinely-intended risky action runs.

The gate is small but load-bearing. A policy classifies each proposed action as low-risk (auto-run) or high-risk (require approval). A high-risk action pauses the run and waits for a decision; it proceeds only on an explicit approve, and refuses on reject or on an unknown decision (default-deny). Here it is, fully deterministic — the “human” decision is a fixed input, not a prompt or a clock:

hitl_gate.py# HITL: high-risk actions PAUSE and require explicit human approval.
# Deterministic: the 'decision' is a fixed input, not a real prompt or clock.
# Real systems persist the pause and resume when the approval arrives (see s5).

HIGH_RISK = {'issue_refund', 'delete_account', 'send_customer_email'}

def needs_approval(action):
    return action in HIGH_RISK

def gate(action, args, decision):
    """Return the outcome of a proposed action under the HITL policy.
    decision is the human's choice ('approve' / 'reject') for high-risk actions;
    low-risk actions ignore it and auto-run. Default-deny on anything else."""
    if not needs_approval(action):
        return {'action': action, 'status': 'auto-ran', 'args': args}
    if decision == 'approve':
        return {'action': action, 'status': 'executed (approved)', 'args': args}
    # reject, missing, or anything unexpected -> refuse. Never execute by default.
    return {'action': action, 'status': 'refused', 'args': args, 'reason': decision or 'no approval'}

# A low-risk action runs without a human:
print(gate('read_order', {'id': 'A-1001'}, decision=None))
# A high-risk action WITH approval executes:
print(gate('issue_refund', {'order': 'A-1001', 'amount': 50}, decision='approve'))
# The same high-risk action WITHOUT approval is refused:
print(gate('issue_refund', {'order': 'A-1001', 'amount': 50}, decision='reject'))
# A high-risk action with NO decision at all is refused (default-deny):
print(gate('send_customer_email', {'to': 'x@example.com'}, decision=None))

{'action': 'read_order', 'status': 'auto-ran', 'args': {'id': 'A-1001'}}
{'action': 'issue_refund', 'status': 'executed (approved)', 'args': {'order': 'A-1001', 'amount': 50}}
{'action': 'issue_refund', 'status': 'refused', 'args': {'order': 'A-1001', 'amount': 50}, 'reason': 'reject'}
{'action': 'send_customer_email', 'status': 'refused', 'args': {'to': 'x@example.com'}, 'reason': 'no approval'}

The refund only fires with decision == 'approve'; reject and the no-decision case both refuse. In production the pause is asynchronous: the run stops, an approval task lands in a queue or a chat message, and the agent resumes when the human responds (which is why state must persist — section 5). If no one responds in time, the request should escalate (to a fallback approver) or expire — never silently execute.

4 · State, resumability & idempotency

An autonomous loop that can pause for a human (section 3) or crash mid-run must be able to stop and pick up exactly where it left off. That means the conversation and progress live in durable storage, not just in memory: the message history, which step it reached, any pending approval, and the idempotency keys of effects already applied. Rich long-term agent memory (what to remember across sessions, summarisation, retrieval of past runs) is its own subject — the agent-memory track — so here we focus only on the production concern: persistence and resumability without repeating side effects.

The lab persists a run to a dict (stand-in for a datastore), interrupts it after a paused approval, then resumes from the saved state. The idempotency ledger means the refund that already ran is not run again on resume:

resume.py# A run can PAUSE (for approval) or crash; we persist enough to RESUME without
# repeating side effects. STORE stands in for a durable datastore (db/kv).
# Deterministic: no clock, no randomness -- the same run always resumes the same.

STORE = {}          # run_id -> saved state
APPLIED = set()     # idempotency keys of effects already applied

def save(run_id, state):
    STORE[run_id] = dict(state)          # copy: the datastore owns its own snapshot

def load(run_id):
    return dict(STORE[run_id])           # resume reads the last saved snapshot

def apply_effect(key, effect):
    """Run an effect at most once. The key makes a retry/resume a no-op."""
    if key in APPLIED:
        return 'skipped (already applied)'
    APPLIED.add(key)
    return effect()

# ---- first pass: run reaches a high-risk step and PAUSES for approval ----
state = {'run_id': 'r-7', 'step': 0, 'history': ['user: refund my order'], 'pending': None}
state['step'] = 1
state['pending'] = {'action': 'issue_refund', 'key': 'refund:r-7', 'amount': 50}
save('r-7', state)
print('paused at step', state['step'], '-> awaiting approval for', state['pending']['action'])

# ---- ... process restarts / human approves later; RESUME from the store ----
resumed = load('r-7')
print('resumed at step', resumed['step'], 'pending:', resumed['pending']['action'])
pend = resumed['pending']
print('apply once :', apply_effect(pend['key'], lambda: f"refunded ${pend['amount']}"))
resumed['step'] = 2
resumed['pending'] = None
save('r-7', resumed)

# ---- a DUPLICATE resume (double-click, retry, redelivery) must NOT re-refund ----
print('apply again:', apply_effect('refund:r-7', lambda: 'refunded $50'))

paused at step 1 -> awaiting approval for issue_refund
resumed at step 1 pending: issue_refund
apply once : refunded $50
apply again: skipped (already applied)

The second apply_effect with the same key returns “skipped” — the refund happened exactly once even though resume ran the step again. That is the whole discipline: save enough to resume, key every side effect, and make a replay a no-op. Message queues redeliver, humans double-click, and processes restart mid-run; idempotency is what keeps those from turning into duplicate charges.

5 · Security — the threats unique to agents

Agents introduce failure modes ordinary software doesn't have, because an untrusted string (a user message, a retrieved document, a tool's output) can influence what privileged actions the agent takes next. These are real, publicly documented risks — named generically below — and the mitigations build directly on the tool discipline from 4.3 and the guardrails from 4.4.

6 · Cost & latency at scale

A single agent request is cheap; a million a day is a budget line. The cost of one request is roughly tokens × price + tool costs, summed over every step in the loop — and multi-step agents run several model calls per request, so it adds up fast. Before optimising, measure: attach the token counts and tool costs to the trace (4.4) and total them. Here is a cost estimator over a simulated trace — every number is illustrative; plug in current pricing:

cost.py# Estimate per-request cost from a trace. ALL prices are ILLUSTRATIVE --
# plug in current per-token pricing and your real tool costs. Deterministic.

# Illustrative price sheet ($ per 1,000 tokens) and per-call tool costs ($).
PRICE = {
    'big':   {'in': 0.0030, 'out': 0.0150},   # a capable, pricier model
    'small': {'in': 0.0003, 'out': 0.0015},   # a cheap, fast model (~10x less)
}
TOOL_COST = {'web_search': 0.005, 'db_query': 0.0, 'send_email': 0.0}

# A simulated trace: each model step (which model, tokens) and each tool call.
trace = [
    {'kind': 'model', 'model': 'big',   'in_tok': 1200, 'out_tok': 300},
    {'kind': 'tool',  'tool': 'web_search'},
    {'kind': 'model', 'model': 'big',   'in_tok': 1800, 'out_tok': 250},
    {'kind': 'tool',  'tool': 'db_query'},
    {'kind': 'model', 'model': 'big',   'in_tok': 2000, 'out_tok': 400},
]

def cost_of(trace):
    total = 0.0
    for ev in trace:
        if ev['kind'] == 'model':
            pr = PRICE[ev['model']]
            total += ev['in_tok'] / 1000 * pr['in']
            total += ev['out_tok'] / 1000 * pr['out']
        else:
            total += TOOL_COST.get(ev['tool'], 0.0)
    return total

per_req = cost_of(trace)
print(f'per request (illustrative): ${per_req:.4f}')
print(f'at 1,000,000 req/day       : ${per_req * 1_000_000:,.0f}/day')

# Route the two 'easy' steps to the cheap model and re-price:
routed = [dict(ev) for ev in trace]
routed[0]['model'] = 'small'      # first triage step doesn't need the big model
routed[2]['model'] = 'small'      # mid step is a simple extraction
per_req_routed = cost_of(routed)
saved = (per_req - per_req_routed) / per_req * 100
print(f'with model routing         : ${per_req_routed:.4f} ({saved:.0f}% cheaper)')

per request (illustrative): $0.0343
at 1,000,000 req/day       : $34,250/day
with model routing         : $0.0187 (45% cheaper)

Routing the two easy steps to a model ~10× cheaper cut the illustrative per-request cost almost in half — with no change to the hard step that actually needs the capable model. The four levers, in rough order of leverage:

7 · Operating agents in production

Shipping the agent is day one; keeping it healthy is every day after. Agents drift in ways ordinary services don't — a model update, a changed prompt, a tool whose upstream shifted, or simply new kinds of user input can quietly degrade quality without throwing a single error. So you watch a specific set of agent metrics, alert on movement, and change the agent only behind a canary you can roll back.

These metrics come straight from the traces (4.4). Here we roll a batch of finished runs up into the numbers you'd put on a dashboard and alert on — deterministic, illustrative counts:

metrics.py# Roll finished runs up into the metrics you monitor and alert on.
# Deterministic, illustrative data -- in prod these come from your traces (4.4).

runs = [
    {'ok': True,  'steps': 3, 'hit_budget': False, 'tool_errors': 0},
    {'ok': True,  'steps': 5, 'hit_budget': False, 'tool_errors': 1},
    {'ok': False, 'steps': 8, 'hit_budget': True,  'tool_errors': 2},
    {'ok': True,  'steps': 4, 'hit_budget': False, 'tool_errors': 0},
    {'ok': False, 'steps': 6, 'hit_budget': False, 'tool_errors': 3},
]

def metrics(runs):
    n = len(runs)
    return {
        'success_rate':   sum(r['ok'] for r in runs) / n,
        'avg_steps':      sum(r['steps'] for r in runs) / n,
        'budget_hit_rate': sum(r['hit_budget'] for r in runs) / n,
        'tool_error_rate': sum(r['tool_errors'] for r in runs) / sum(r['steps'] for r in runs),
    }

m = metrics(runs)
print(f"success rate    : {m['success_rate']:.0%}")
print(f"avg steps/run   : {m['avg_steps']:.1f}")
print(f"budget-hit rate : {m['budget_hit_rate']:.0%}")
print(f"tool error rate : {m['tool_error_rate']:.0%}")

# A simple drift alert: fire if success drops below an agreed threshold.
THRESHOLD = 0.80
print('ALERT: success below threshold' if m['success_rate'] < THRESHOLD else 'ok: within threshold')

success rate    : 60%
avg steps/run   : 5.2
budget-hit rate : 20%
tool error rate : 23%
ALERT: success below threshold

Four numbers tell you most of what you need: success rate (is it working?), avg steps (is it wandering — a jump means the loop is thrashing), budget-hit rate (how often runs hit the 4.4 cap instead of finishing), and tool error rate (is an upstream tool failing?). A sustained move in any of them is drift — alert on it before users do.

When you change the prompt, model, or tools — the three things that most alter an agent's behaviour — treat it like any risky deploy:

The loop closes back on itself: run the change through the offline eval suite (4.4's trajectory + output evals) first; if it passes, canary it to a small slice of traffic; watch the same metrics against the baseline; roll back on drift or ramp up when it holds. And every real failure you catch in production becomes a new eval case — so the suite gets stronger and the same regression can't ship twice. That feedback loop, not any single model, is what makes a production agent reliable over time.