Novel agent architectures

1 · The ReAct ceiling

Here's a demo that's probably burned you. You wire up a tool-using agent, it nails the happy path, and you ship it feeling clever. Then a user gives it a task where the first move matters — the agent picks a database it shouldn't have, or an approach that can't reach the goal — and instead of noticing and turning around, it doubles down. It keeps calling tools, each observation slightly worse, until it either loops on the same three actions or confidently returns a wrong answer. Nothing crashed. The loop just had no way to say "that was a mistake, let me go back." That missing ability is the whole subject of this lesson.

Think of ReAct like driving with your eyes only on the next fifty feet of road. You see enough to make the immediate turn, and on a straight highway that's plenty. But there's no map on the dashboard, no rear-view mirror, and no reverse gear. Take one wrong exit and you can only keep driving forward, hoping the road loops back. Every "novel" architecture in this lesson is really just bolting one missing instrument onto that dashboard: search adds a reverse gear and a memory of exits you skipped; reflection adds a mirror to check the work you just did; a planner adds the full map; a memory system adds a trip log; debate adds a passenger who might spot what you missed.

Precisely: the ReAct loop is greedy (it takes the locally best-looking action at each step) and linear (it keeps one single trajectory of Thought → Action → Observation). Those two properties are exactly why it's cheap, fast, and easy to reason about — and also why it hits a ceiling. Greedy means it never weighs an action against the alternatives it didn't take; linear means there is only one path in memory, so there is nothing to return to. The common mistake engineers make is treating these limits as bugs to prompt away ("just tell it to double-check!") when they are structural: no prompt gives a linear loop a frontier of shelved branches, and no instruction to "reconsider" gives a greedy step the value estimates of the roads not taken. When you feel an agent thrashing, resist the urge to add another sentence to the system prompt — the fix is almost always a different shape of loop, and the table below is your map from symptom to shape.

The ReAct loop — Thought → Action → Observation, repeat — is the workhorse of tool-using agents, and for many tasks it is enough. But it is fundamentally greedy and linear: it commits to one line of reasoning, takes an action, and if that action was a mistake it has no principled way to undo it. Three failure modes recur. It cannot backtrack when a branch dead-ends. It has no global plan, so on long-horizon tasks it loses the thread. And it cannot check its own work — a confidently wrong step propagates. Every richer architecture below is a targeted fix for one of these.

2 · Tree & graph search agents

Watch a strong human solve a hard puzzle — a chess position, a Sudoku, a tricky bit of debugging — and you'll notice they don't march forward in a straight line. They try a move in their head, imagine where it leads, and if it looks bad they rewind and try a different one. They hold several half-explored ideas at once and keep spending attention on whichever looks most promising. That is exactly what a ReAct agent cannot do and what a search agent can. The unlock is simple to state and powerful in effect: don't commit — explore.

Here's the analogy that makes it click. Imagine the task as a branching maze, where every fork is a decision the agent could make. ReAct walks the maze with no string behind it: pick a passage, walk, pick again, and if you hit a wall you're stuck staring at it. A search agent unrolls Theseus's thread. It keeps a frontier — a to-do list of forks it has seen but not yet walked down — and at each step it doesn't just take the next passage, it (1) branches: asks "what are my candidate moves from here?", (2) scores: estimates how promising each resulting spot looks, (3) expands the single most promising spot, and (4) backtracks — walks back up the thread to a shelved fork — the moment the current passage dead-ends. Nothing here is new to computer science: it is textbook graph search (breadth-first, depth-first, or best-first), and the only twist is who does the branching and scoring. The LLM plays two roles it's good at: proposing plausible next moves (the branch generator) and judging how good a state looks (the evaluator).

This family is what the research literature calls Tree of Thoughts and Graph of Thoughts — the "thoughts" are the intermediate reasoning states, and the tree or graph is the space of ways they connect. Be honest about the trade, because this is where the common mistake lives: search is not free reasoning. Every branch you keep is more model calls, so a wide, deep search can cost ten or a hundred times a single ReAct pass. It earns that cost only when two things hold — the goal is checkable (you can tell a good state from a bad one, or you have nothing to score with) and mistakes are expensive to reverse (so the ability to back out beats the ability to run fast). On a plain fact lookup, search just burns tokens re-deriving something one call would have answered. The diagram below is the four-step cycle you'll implement in the next section.

Instead of committing to one reasoning path, a search agent treats the task as exploring a tree (or graph) of states: from a state it branches into candidate next actions, scores each resulting state with a heuristic (or a value from the model), expands the most promising, and backtracks when a branch dead-ends. This is Tree-of-Thoughts / graph-of-thoughts in spirit, and it is just classic search (BFS/DFS/best-first) with the LLM as the branch generator and the state evaluator.

This buys the one thing ReAct lacks: the ability to recover from a wrong turn by returning to a frontier and trying another branch. The cost is real — each branch is more model calls — so search pays off on tasks with a checkable goal and expensive-to-reverse mistakes (planning, puzzles, multi-step tool chains), not on simple lookups.

3 · Lab · a best-first search agent, offline

Search agents sound intimidating until you see the loop with the model taken out. The usual reason they feel like black magic is that the LLM's branching and scoring hide the actual control flow — so it's hard to tell where the "intelligence" ends and the plumbing begins. So we do the honest thing here: we stub out the model. The branch generator is a deterministic function, the scorer is a hand-written heuristic, and what's left is pure, inspectable search plumbing — the exact same loop a real Tree-of-Thoughts agent runs, minus the API calls and minus the nondeterminism. If you understand this 30-line program, you understand the mechanism; swapping the stubs for real LLM calls changes nothing about the shape.

The toy task is deliberately small so you can trace it by hand: build the string "HELLO" one character at a time. A "state" is a partial string; from any state you can branch by appending a character. That gives an enormous space (26 choices per position, so 26⁵ ≈ 11.9 million full strings), which is the point — blind brute force is hopeless, and only the heuristic makes it tractable. Two pieces do the real work, and both map straight onto the concepts from §2:

• The heuristic is the scorer — it estimates "how far is this state from the goal?" Here it counts characters that are already wrong plus characters still missing. Lower is better, so a state that matches the target prefix perfectly and just needs more length scores well. This stands in for the LLM's value estimate: a cheap guess at promise, not a guarantee.
• The frontier is a heapq (a min-heap / priority queue — see D4 in the DSA track) keyed by that heuristic. "Best-first" means one thing operationally: always pop the lowest-scoring (most promising) state next. The heap is what lets the agent hold many half-explored branches at once and instantly grab the best one — the shelved-branches memory that ReAct structurally lacks.

Keep one subtlety in view as you read, because it's the whole payoff and it's easy to miss: the distractor branches (appending "X" or "Z") don't vanish — they get pushed onto the frontier with a worse score and simply wait. If the promising branch had dead-ended, the loop would have popped one of those shelved states and carried on from there. That waiting-and-returning is backtracking, and it's the recovery mechanism the whole lesson is about, made concrete in a few lines you can run.

Here is the search loop as a mechanism, in pure Python. The 'LLM' is stubbed by a deterministic branch generator and a heuristic so it runs offline and you see the search behavior — the same control flow a real agent uses, minus the model calls. The task: reach a target string by appending characters, with a heuristic guiding the frontier.

search_agent.pyimport heapq

TARGET = "HELLO"
ALPHABET = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"

def heuristic(state):
    # chars still wrong vs the target prefix of the same length + chars remaining
    wrong = sum(1 for i, c in enumerate(state) if i < len(TARGET) and c != TARGET[i])
    return wrong + (len(TARGET) - len(state))

def successors(state):
    # branch: append the correct next char AND two distractors (LLM stand-in)
    if len(state) >= len(TARGET):
        return []
    good = TARGET[len(state)]
    return [state + c for c in (good, "X", "Z")]

def search():
    start = ""
    frontier = [(heuristic(start), start)]     # best-first: lowest heuristic first
    expansions = 0
    seen = set()
    while frontier:
        h, state = heapq.heappop(frontier)
        if state in seen:
            continue
        seen.add(state)
        if state == TARGET:
            print(f"  GOAL reached: '{state}' in {expansions} expansions (vs 26^5 blind)")
            return
        expansions += 1
        for nxt in successors(state):
            heapq.heappush(frontier, (heuristic(nxt), nxt))
        if len(state) < len(TARGET):
            print(f"  expanded {state!r:8} (h={h}, frontier={len(frontier)})")

print(f"target={TARGET!r}")
search()
print("best-first search reaches the goal by scoring the frontier, not brute force")

target='HELLO'
  expanded 'H'      (h=4, frontier=3)
  expanded 'HE'     (h=3, frontier=5)
  expanded 'HEL'    (h=2, frontier=7)
  expanded 'HELL'   (h=1, frontier=9)
  GOAL reached: 'HELLO' in 5 expansions (vs 26^5 blind)
best-first search reaches the goal by scoring the frontier, not brute force

The agent reaches the goal in a handful of expansions by always growing the most promising node — and crucially it could backtrack to a shelved frontier node if a branch dead-ended. Swap the stub branch/score functions for LLM calls and this is a Tree-of-Thoughts agent.

4 · Reflection & planner–executor

You've felt the difference between these two ideas even if you've never named them. Reflection is what you do when you finish an email, reread it, and catch the typo before sending — a quick pass over the thing you just made. Planning is what you do before a road trip when you sketch the route on a map — a structure laid down before you start moving. They fix opposite problems, and mixing them up is one of the most common design mistakes in agent work, so it's worth slowing down to separate them cleanly.

Reflection attacks the ReAct weakness "can't check its own work." The mechanism is a loop with two hats: the agent generates an answer, then a critic pass — usually the same model with a different prompt — evaluates it and hands back a revision, and you repeat until the critic is satisfied or a budget runs out. Now the crucial subtlety, and the reason we hammer it below: reflection is only as good as the critic's signal. If the critic can check against something real — a failing unit test, a JSON schema that won't validate, a tool that returns an error, a contradiction with a retrieved fact — then each revision is genuinely informed and the loop converges. But if the critic is just the model re-reading its own output with no external anchor, it critiques with the same blind spots that produced the mistake, and the loop can happily "improve" a wrong answer forever, or oscillate, or declare a confident error correct. The two safeguards are therefore non-negotiable: bound the loop (a max-attempts budget so a signal-less critic can't spin) and, wherever possible, ground the critic in an external check rather than the model's own opinion. Unbounded, self-graded reflection is the single most common way this pattern wastes money in production.

Planner–executor attacks a different weakness — "no long-horizon plan." Instead of one model deciding both what to do next and how to do it at every step (which is where ReAct loses the thread on multi-step tasks), you split the roles. A planner looks at the whole task once and decomposes it into an ordered list of sub-goals — the global thread that survives across steps. An executor then carries out each sub-goal, often with a plain little ReAct loop of its own, staying simple because it only has to worry about one step at a time. The elegance is in the feedback edge: when the executor reports that a step failed, control returns to the planner, which can re-plan around the obstacle rather than blindly retrying. Think of it as a project manager (planner) writing the task list and an engineer (executor) working the tickets — and the manager rescheduling when a ticket turns out to be blocked. The diagram traces exactly that flow, including the re-plan feedback loop.

Reflection adds a self-critique loop: the agent generates an answer, then a critic pass (often the same model with a different prompt) evaluates it against the goal and produces a revision. Repeat until the critic is satisfied or a budget runs out. Reflection shines when errors are detectable from the output (a failing test, a schema violation, a contradiction) — the critic has real signal. It loops uselessly when there's no external check, because the model critiques with the same blind spots that produced the error. Always bound the loop and, where you can, ground the critic in an external signal (tests, a validator, a tool result).

Planner–executor splits the agent in two: a planner decomposes the task into an ordered plan of sub-goals once, and an executor carries out each step (often a plain ReAct loop). This gives the long-horizon coherence ReAct lacks — the plan is the global thread — while keeping each step simple. The planner can re-plan when the executor reports a step failed.

reflection.pydef external_check(answer):
    # a real critic grounded in an external signal (here: format rules)
    if "kg" not in answer:
        return False, "missing units"
    if "work" not in answer and "*" not in answer:
        return False, "show work"
    return True, "ok"

def generate(attempt, critique):
    # stubbed model: each revision addresses the last critique
    drafts = ["answer is 42", "answer is 42 kg", "work: 6*7; answer is 42 kg"]
    return drafts[min(attempt, len(drafts) - 1)]

def reflect(max_attempts=5):
    critique = ""
    for attempt in range(max_attempts):
        answer = generate(attempt, critique)
        ok, critique = external_check(answer)
        verdict = "PASS" if ok else f"FAIL ({critique})"
        print(f"attempt {attempt+1}: {answer!r} -> critic: {verdict}")
        if ok:
            print(f"converged in {attempt+1} attempts (bounded at {max_attempts}) "
                  f"-- external check gave real signal")
            return
    print("budget exhausted without convergence")

reflect()

attempt 1: 'answer is 42' -> critic: FAIL (missing units)
attempt 2: 'answer is 42 kg' -> critic: FAIL (show work)
attempt 3: 'work: 6*7; answer is 42 kg' -> critic: PASS
converged in 3 attempts (bounded at 5) -- external check gave real signal

5 · Memory architectures

Try this and you'll hit the wall immediately: build a chatbot whose only memory is the conversation you paste back into each request. It works beautifully for a dozen turns. Then the conversation gets long, you cross the context limit, and to fit you start dropping the oldest turns — and now the agent forgets the user's name, re-asks a question it already answered, and contradicts a decision made twenty turns ago. Cramming everything into one ever-growing context isn't memory; it's a bucket with a hole in the bottom. Real agents need memory the way a person does: not one flat log, but different kinds of memory for different jobs.

The design that works — and it's borrowed straight from how cognitive scientists model human memory — is tiered. Picture your own mind for a second. Right now you're holding this sentence in working memory: tiny, fast, and gone in seconds. You can also recall specific things that happened — what you had for breakfast, a conversation last Tuesday — that's episodic memory, a log of events tagged by time and place. And you know durable facts — that Paris is in France, that you dislike coriander — without remembering when you learned them; that's semantic memory, distilled knowledge stripped of its original context. Agents get the same three tiers: working memory is the current context window (small, always present, volatile); episodic memory is a store of past turns and events, pulled back by recency and relevance; semantic memory is a compact store of distilled facts and preferences that persists across sessions.

The verbs that connect the tiers are where the real engineering lives, and naming them is what turns "memory" from a vague wish into a system you can build. The agent writes important events into episodic memory as they happen, periodically consolidates — compresses many episodic events into a few durable semantic facts (the way "user asked for tea three times" becomes the standing fact "user prefers tea"), and retrieves from both stores each turn to refill the small working-memory context. Here is the mistake beginners make, and it's the opposite of what you'd expect: they obsess over storing everything and never think about retrieval and forgetting. But storing is the easy part. The hard, quality-determining questions are what to pull back into a limited context (retrieve too much and you drown the signal and blow the token budget) and what to let go (forget the wrong thing and the agent repeats old mistakes; keep everything and the durable store bloats into noise). Consolidation is the compression valve that keeps semantic memory small and sharp. The table lays out the three tiers side by side; read it as three answers to one question — "how long should this survive, and how do I get it back?"

A ReAct agent's only memory is its context window — which overflows and forgets. Real agents use a tiered memory architecture, borrowed from cognitive models: working memory (the current context — small, fast, volatile), episodic memory (a log of past interactions/events, retrieved by recency and relevance), and semantic memory (distilled facts and preferences, the durable knowledge base). The agent writes important events to episodic memory, periodically consolidates them into semantic memory, and retrieves from both to refill working memory each turn.

6 · Multi-agent debate

You already trust this idea in every other part of life. A hard decision at work gets a room of people, not one person's gut. A patient with a scary diagnosis gets a second opinion. A tricky code change gets a reviewer. The intuition is that independent minds make different mistakes, so putting them together lets the errors cancel while the shared truth survives. Multi-agent debate is that intuition turned into an architecture: run several agents on the same question, let them see and critique each other's answers, and iterate until they converge or a judge picks a winner.

Mechanically it's an ensemble with a conversation bolted on. In the simplest form, each agent answers independently in round one; then every agent sees its peers' answers and gets a chance to revise; you repeat for a few rounds, watching whether the group settles on a single answer. Sometimes the agents are given different roles or personas ("optimist" vs "skeptic," "the coder" vs "the tester") to deliberately widen the range of approaches. The lab below models the bare skeleton: three agents, one of them wrong in round one, revising toward the majority once it sees the others — and you'll watch consensus form.

Now the honest part, because debate is the architecture most often deployed on faith and most often disappointing. It has two hard costs and one seductive failure mode. The costs: you pay roughly N× the tokens and latency for N agents (and more if you run several rounds), which adds up fast. The failure mode is the subtle one, and it's the exact inverse of why debate works: the cancellation only happens when the agents' errors are independent. But your agents are usually the same model with the same training data and therefore the same biases — so on a question they'd get wrong, they tend to get it wrong the same way, and debate doesn't cancel the error, it amplifies it into a confident consensus. Three copies of the same mind agreeing is not three opinions; it's one opinion said loudly. This is why debate helps on open-ended, multi-path tasks (where there are genuinely different valid approaches to disagree about) and quietly wastes money on single-answer fact lookups (where there's nothing to diversify and the shared bias just compounds). The rule to carry out of this section: reach for debate only where a single agent has measurably plateaued and the task can absorb the cost — never as a default sprinkled over everything.

A single agent has one viewpoint and one set of blind spots. Multi-agent debate runs several agents on the same question — sometimes with different roles or personas — lets them see each other's answers and critiques, and iterates until they converge or a judge decides. The wins are real on tasks with diverse valid approaches and where errors are idiosyncratic: independent agents cancel each other's mistakes, much like an ensemble. The costs are equally real — N× the tokens and latency, plus the risk of convergence on a confident wrong answer if the agents share the same bias. This lab models a simple debate-to-consensus.

debate.pyfrom collections import Counter

def agent_answer(agent_id, round_no, peer_answers):
    # stubbed agents: agent 2 is wrong in round 1, revises after seeing peers
    if round_no == 1:
        return {0: 4, 1: 4, 2: 6}[agent_id]
    # round 2: revise toward the majority of peers if you were an outlier
    majority = Counter(peer_answers).most_common(1)[0][0]
    mine = {0: 4, 1: 4, 2: 6}[agent_id]
    return majority if mine != majority else mine

def debate(n_agents=3, max_rounds=3):
    answers = [None] * n_agents
    for rnd in range(1, max_rounds + 1):
        peer = list(answers)
        answers = [agent_answer(i, rnd, peer) for i in range(n_agents)]
        counts = Counter(answers)
        top, agree = counts.most_common(1)[0]
        note = "no consensus" if agree < n_agents else "consensus"
        print(f"round {rnd} answers: {answers}        ({note})")
        if agree == n_agents:
            print(f"consensus: {top} after {rnd} rounds (agreement {agree}/{n_agents})")
            return
    print("no full consensus within budget")

debate()
print("debate cancels idiosyncratic errors -- at N x the token cost")

round 1 answers: [4, 4, 6]        (no consensus)
round 2 answers: [4, 4, 4]        (saw peers, one revised)
consensus: 4 after 2 rounds (agreement 3/3)
debate cancels idiosyncratic errors -- at N x the token cost

FIX = {
    "can't undo a bad step":   "tree/graph search (branch + backtrack)",
    "no self-correction":      "reflection (generate, critique, revise)",
    "no long-horizon plan":    "planner-executor (plan, then act)",
    "forgets across turns":    "memory architecture (working/episodic/semantic)",
    "single fallible view":    "multi-agent debate",
}
for failure, fix in FIX.items():
    print(f"{failure:24} -> {fix}")

def tool(query):                      # stubbed tool
    return {"weather": "sunny", "time": "noon"}.get(query, "unknown")

def react_step(state):
    goal = state["goal"]
    thought = f"I need {goal}"
    action = goal
    obs = tool(action)
    state = {**state, "known": {**state["known"], goal: obs}}
    return thought, action, obs, state

def run(goal, max_steps=5):
    state = {"goal": goal, "known": {}}
    for step in range(max_steps):
        t, a, o, state = react_step(state)
        print(f"step {step+1}: thought={t!r} action={a!r} obs={o!r}")
        if state["known"].get(goal) not in (None, "unknown"):
            print("goal satisfied"); return
    print("budget exhausted")

run("weather")

import heapq
TARGET = "HELLO"

def h(state):
    wrong = sum(1 for i, c in enumerate(state) if i < len(TARGET) and c != TARGET[i])
    return wrong + (len(TARGET) - len(state))

def succ(state):
    if len(state) >= len(TARGET): return []
    return [state + c for c in (TARGET[len(state)], "X", "Z")]

def search():
    frontier, seen, exp = [(h(""), "")], set(), 0
    while frontier:
        _, s = heapq.heappop(frontier)
        if s in seen: continue
        seen.add(s)
        if s == TARGET:
            print(f"goal in {exp} expansions (blind would be 26^5)"); return
        exp += 1
        for n in succ(s): heapq.heappush(frontier, (h(n), n))

search()

def check(ans):
    if "kg" not in ans:   return False, "missing units"
    if "*" not in ans:    return False, "show work"
    return True, "ok"

def generate(attempt):                 # each revision fixes the last critique
    return ["42", "42 kg", "6*7 = 42 kg"][min(attempt, 2)]

def reflect(max_attempts=5):
    for attempt in range(max_attempts):
        ans = generate(attempt)
        ok, why = check(ans)
        print(f"attempt {attempt+1}: {ans!r} -> {'PASS' if ok else 'FAIL: '+why}")
        if ok:
            print(f"converged in {attempt+1} (bounded at {max_attempts})"); return
    print("budget exhausted")

reflect()

from collections import Counter

class Memory:
    def __init__(self):
        self.episodic, self.semantic = [], {}
    def write(self, event):
        self.episodic.append(event)
    def retrieve(self, query, k=2):
        scored = sorted(self.episodic,
                        key=lambda e: -len(set(e.split()) & set(query.split())))
        return scored[:k]
    def consolidate(self, threshold=2):
        counts = Counter(self.episodic)
        for fact, n in counts.items():
            if n >= threshold:
                self.semantic[fact] = n
        self.episodic = [e for e in self.episodic if e not in self.semantic]

m = Memory()
for e in ["user likes tea", "user likes tea", "booked flight to NYC"]:
    m.write(e)
print("retrieve 'tea':", m.retrieve("likes tea"))
m.consolidate()
print("semantic:", m.semantic, "  episodic left:", m.episodic)

def route(has_checkable_goal, reversible_mistakes_costly, open_ended_multipath, tight_budget):
    if tight_budget and not reversible_mistakes_costly:
        return "plain ReAct", "budget is tight and mistakes are cheap to redo"
    if reversible_mistakes_costly and has_checkable_goal:
        return "tree/graph search", "costly-to-reverse mistakes + a checkable goal"
    if open_ended_multipath and not tight_budget:
        return "multi-agent debate", "diverse valid paths; budget to cancel errors"
    return "planner-executor", "long-horizon single-path task needs a global plan"

cases = [
    (True,  True,  False, False),   # search
    (True,  False, True,  False),   # debate
    (False, False, False, False),   # planner-executor
    (True,  False, False, True),    # ReAct
]
for c in cases:
    arch, why = route(*c)
    print(f"{arch:18} <- {why}")