Reasoning models & test-time compute

1 · What a reasoning model is essential

A normal model reads your prompt and starts writing the answer immediately — one fast pass. A reasoning model first produces a private chain of thinking tokens — it works the problem out step by step — and only then writes the visible answer. Those thinking tokens are extra compute spent before the answer appears. You usually don't see them, but you pay for them, and they wait for them: more thinking means a slower, pricier, but often more correct reply on hard problems.

The setting that turns this on and up is the thinking / effort control introduced in Ch 02 · Prompting. This lesson is about when to reach for it and how much to spend.

Compare that to the fast path: question → answer, no thinking budget, cheap and low-latency. For an easy question the fast path is the right tool; spending thinking tokens on "what's 2+2" just burns money and time for no gain.

2 · Extended & adaptive thinking essential

Extended thinking means you allow the model a larger budget of thinking tokens so it can reason further before answering. Adaptive thinking means the model spends as much as the problem needs up to that budget — an easy question finishes quickly; a hard one uses more of the allowance. You set the ceiling; the model decides how much of it to use.

The key mental model: thinking tokens are spent before the answer, and they are billed like output tokens. A 2,000-token answer with a 6,000-token thinking budget can cost like an 8,000-token response and take proportionally longer to arrive.

thinking_cost.pydef response_cost(thinking_tokens, answer_tokens, price_per_1k=0.015):
    """Thinking tokens are billed before the answer even starts."""
    total = thinking_tokens + answer_tokens
    return {
        "thinking_tokens": thinking_tokens,
        "answer_tokens": answer_tokens,
        "billed_tokens": total,
        "cost_usd": round(total / 1000 * price_per_1k, 5),
    }

fast = response_cost(thinking_tokens=0,    answer_tokens=2000)
deep = response_cost(thinking_tokens=6000, answer_tokens=2000)
print("fast:", fast)
print("deep:", deep)
print("deep costs", round(deep["cost_usd"] / fast["cost_usd"], 1), "x the fast reply")

fast: {'thinking_tokens': 0, 'answer_tokens': 2000, 'billed_tokens': 2000, 'cost_usd': 0.03}
deep: {'thinking_tokens': 6000, 'answer_tokens': 2000, 'billed_tokens': 8000, 'cost_usd': 0.12}
deep costs 4.0 x the fast reply

3 · Test-time compute — buying accuracy at inference intermediate

Test-time compute is the idea behind all of this: instead of only making the model bigger at training time, you spend more compute at inference (test) time — more thinking tokens, or several attempts you vote over. On hard reasoning tasks this reliably lifts accuracy: the model that "thinks longer" gets more of them right. On easy tasks it does almost nothing, because the fast answer was already correct.

The classic curve: accuracy rises with compute but with diminishing returns — each extra unit of thinking helps less than the last. So the goal is never "max everything"; it's "enough compute for this task's difficulty."

test_time_compute.pydef accuracy(base_acc, compute_units):
    """Each extra unit of compute closes half the remaining gap to 1.0.
    Deterministic: no randomness, so the curve is reproducible."""
    acc = base_acc
    for _ in range(compute_units):
        acc = acc + (1.0 - acc) * 0.5
    return round(acc, 4)

print("difficulty: HARD (base 0.40)")
for units in range(0, 6):
    print(f"  compute={units}: accuracy={accuracy(0.40, units)}")
print("gain 0->1 unit:", round(accuracy(0.40, 1) - accuracy(0.40, 0), 3))
print("gain 4->5 unit:", round(accuracy(0.40, 5) - accuracy(0.40, 4), 3))

difficulty: HARD (base 0.40)
  compute=0: accuracy=0.4
  compute=1: accuracy=0.7
  compute=2: accuracy=0.85
  compute=3: accuracy=0.925
  compute=4: accuracy=0.9625
  compute=5: accuracy=0.9812
gain 0->1 unit: 0.3
gain 4->5 unit: 0.019

The first unit of thinking bought +0.30 accuracy; the fifth bought +0.019. That shape is why a sensible policy caps effort rather than always maxing it.

4 · The effort dial: low → medium → high → max advanced

In practice you don't hand-set token counts — you turn an effort dial with a few named settings. Higher effort = a bigger thinking budget = more accuracy on hard tasks, but more cost and more latency. Think of it as one knob trading three things at once.

effort_dial.pyEFFORT = {
    # effort: (thinking_tokens, base cost multiplier, latency seconds)
    "low":    (0,     1.0, 0.8),
    "medium": (2000,  3.0, 2.5),
    "high":   (6000,  6.0, 6.0),
    "max":    (12000, 10.0, 12.0),
}

def tradeoff(hard_task_base_acc=0.40):
    rows = []
    for name, (think, cost_mult, latency) in EFFORT.items():
        # more thinking closes the accuracy gap, with diminishing returns
        acc = hard_task_base_acc
        for _ in range(think // 2000):
            acc = acc + (1.0 - acc) * 0.5
        rows.append((name, think, cost_mult, latency, round(acc, 3)))
    return rows

print(f"{'effort':7} {'think':>6} {'cost':>5} {'lat_s':>6} {'acc':>6}")
for name, think, cost, lat, acc in tradeoff():
    print(f"{name:7} {think:6d} {cost:5.1f} {lat:6.1f} {acc:6.3f}")

effort   think  cost  lat_s    acc
low          0   1.0    0.8  0.400
medium    2000   3.0    2.5  0.700
high      6000   6.0    6.0  0.925
max      12000  10.0   12.0  0.991

5 · Routing: fast model vs reasoning by difficulty advanced

The single biggest lever is routing: send easy work to the fast, cheap path and only the hard work to the reasoning path. Most real traffic is easy; if you route it all through a reasoning model you pay 5–10x for accuracy you didn't need. A cheap difficulty classifier in front of the models pays for itself immediately.

router.pydef estimate_difficulty(task):
    """Cheap heuristic difficulty score in [0,1] — deterministic, no model call."""
    hard_signals = ("prove", "derive", "multi-step", "optimize", "why", "plan", "debug")
    score = 0.1
    if task.get("needs_math"):        score += 0.4
    if task.get("multi_step"):        score += 0.3
    text = task.get("text", "").lower()
    score += 0.1 * sum(sig in text for sig in hard_signals)
    return min(score, 1.0)

def route(task, threshold=0.5):
    d = estimate_difficulty(task)
    path = "reasoning (high effort)" if d >= threshold else "fast (low effort)"
    return round(d, 2), path

tasks = [
    {"text": "What is the capital of France?"},
    {"text": "Summarize this email in one line."},
    {"text": "Derive the optimal batch size, show the multi-step math.", "needs_math": True, "multi_step": True},
    {"text": "Why does the checkout service crash under load? Debug it.", "multi_step": True},
]
for t in tasks:
    d, path = route(t)
    print(f"diff={d:<4} -> {path:24} | {t['text'][:38]}")

diff=0.1  -> fast (low effort)        | What is the capital of France?
diff=0.1  -> fast (low effort)        | Summarize this email in one line.
diff=1.0  -> reasoning (high effort)  | Derive the optimal batch size, show th
diff=0.6  -> reasoning (high effort)  | Why does the checkout service crash un

6 · Production cost control: budgets & self-consistency professional

At scale you can't let effort float freely — you give it a budget. A budget-aware selector picks the highest effort you can still afford for the remaining requests, degrading gracefully instead of blowing the daily spend by lunchtime.

budget_selector.pyCOST = {"low": 0.01, "medium": 0.03, "high": 0.06, "max": 0.10}  # $ / request

def pick_effort(remaining_budget, remaining_requests, desired):
    """Pick the desired effort if the budget can sustain it for all remaining
    requests, else step down until it fits. Deterministic."""
    order = ["max", "high", "medium", "low"]
    start = order.index(desired)
    for effort in order[start:]:
        if COST[effort] * remaining_requests <= remaining_budget:
            return effort
    return "low"  # cheapest fallback

print(pick_effort(remaining_budget=100.0, remaining_requests=500,  desired="high"))
print(pick_effort(remaining_budget=20.0,  remaining_requests=500,  desired="high"))
print(pick_effort(remaining_budget=3.0,   remaining_requests=500,  desired="high"))

high
medium
low

Another test-time-compute technique is self-consistency: instead of one long think, sample several independent answers at moderate effort and take a majority vote. On problems with a single right answer, voting over a handful of samples often beats one expensive attempt — trading parallel calls for a bigger single thinking budget.

self_consistency.pydef fake_sample(task_seed, i):
    """A DETERMINISTIC stand-in for an independent model sample.
    Varies by index (no random): most samples land on the correct answer,
    a minority drift to wrong ones — like real reasoning noise."""
    correct = "42"
    distractors = ["41", "43", "40"]
    # every 4th sample goes astray, cycling through distractors
    if i % 4 == 3:
        return distractors[(i // 4) % len(distractors)]
    return correct

def self_consistency(task_seed, n_samples=8):
    votes = {}
    for i in range(n_samples):
        ans = fake_sample(task_seed, i)
        votes[ans] = votes.get(ans, 0) + 1
    winner = max(votes, key=lambda k: votes[k])
    return winner, votes

winner, votes = self_consistency("hard-math-q", n_samples=8)
print("votes:", votes)
print("majority answer:", winner)

votes: {'42': 6, '41': 1, '43': 1}
majority answer: 42

7 · Turning it on in code professional

In real code you enable extended thinking on the request. This snippet is illustrative — it needs the SDK and a network call, so it does not run offline like the labs above.

thinking_sdk.py# needs the SDK: pip install anthropic; requires network + credentials.
# Illustrative only — the offline labs above model the same tradeoffs.
from anthropic import Anthropic

client = Anthropic()

resp = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=4000,
    thinking={"type": "enabled", "budget_tokens": 6000},  # the effort dial
    messages=[{"role": "user",
               "content": "Prove that the sum of the first n odd numbers is n^2."}],
)

# The response carries hidden `thinking` blocks (billed) plus the final answer.
for block in resp.content:
    print(block.type)

8 · A reasoning strategy for the system tech-lead

A lead doesn't set effort per prompt by feel — they design the whole reasoning tier: a difficulty router in front (fast path for the easy majority), default effort tuned per route, a spend budget with graceful degradation, and evals proving the extra compute actually buys accuracy on your tasks (not just benchmarks). Reasoning is a cost center you manage, not a switch you leave on.

The reasoning-tier playbook

1. Route by difficulty first — most traffic never needs the reasoning path (router.py).
2. Set a default effort per route, not a global one — easy routes stay low, hard routes go high.
3. Cap spend with a budget that degrades effort gracefully under pressure (budget_selector.py).
4. Prove the win on evals — measure accuracy vs cost at each effort on your own tasks; stop where returns flatten.
5. Consider self-consistency for single-answer tasks where parallel votes beat one long think.

This connects to two neighbours. When even high effort on a base model can't reliably hit a narrow behavior, that's the signal to consider the fine-tuning decision (XT4) — reasoning tunes compute at inference, fine-tuning tunes the model itself. And the effort/thinking setting you're dialing here is the one introduced in Ch 02 · Prompting.

def difficulty(prompt):
    hard_signals = ["prove", "step by step", "derive", "plan", "why"]
    score = sum(sig in prompt.lower() for sig in hard_signals)
    return "hard" if score or len(prompt) > 200 else "easy"

def route(prompt):
    return "reasoning-model" if difficulty(prompt) == "hard" else "fast-model"

print(route("What's the capital of France?"))          # fast-model
print(route("Derive the shortest path step by step"))  # reasoning-model

THINKING_BUDGET = {"low": 1000, "medium": 4000, "high": 16000, "max": 32000}
PRICE_PER_1K = 0.015   # illustrative output-token price in dollars

def est_cost(effort, answer_tokens=500):
    tokens = THINKING_BUDGET[effort] + answer_tokens
    return round(tokens / 1000 * PRICE_PER_1K, 4)

for e in THINKING_BUDGET:
    print(f"{e:6} -> ~{THINKING_BUDGET[e]} think tokens, ${est_cost(e)}")
# max costs ~20x low; use the lowest effort that clears your accuracy bar

from collections import Counter

def self_consistency(answers):
    counts = Counter(answers)
    best, n = counts.most_common(1)[0]
    return {"answer": best, "votes": n, "confidence": round(n / len(answers), 2)}

# imagine 5 independent reasoning samples for one arithmetic problem
samples = ["42", "42", "42", "41", "42"]
print(self_consistency(samples))
# {'answer': '42', 'votes': 4, 'confidence': 0.8}

class ReasoningBudget:
    def __init__(self, ceiling=50000):
        self.ceiling = ceiling; self.used = 0
    def effort_for(self, requested):
        order = ["low", "medium", "high", "max"]
        budget = {"low": 1000, "medium": 4000, "high": 16000, "max": 32000}
        # downgrade until it fits the remaining budget
        i = order.index(requested)
        while i > 0 and self.used + budget[order[i]] > self.ceiling:
            i -= 1
        chosen = order[i]; self.used += budget[chosen]
        return chosen

b = ReasoningBudget(ceiling=20000)
print(b.effort_for("max"))    # high  (32k would overflow -> downgraded)
print(b.effort_for("max"))    # low   (little budget left)
print("used:", b.used)

from anthropic import Anthropic

client = Anthropic()

resp = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=4000,
    thinking={"type": "enabled", "budget_tokens": 8000},
    messages=[{"role": "user",
               "content": "A train leaves at 3pm going 60mph... solve step by step."}],
)

for block in resp.content:
    if block.type == "thinking":
        print("[reasoning]", block.thinking[:80], "...")
    elif block.type == "text":
        print("[answer]", block.text)

def reasoning_policy(request):
    tier = request["tier"]           # 'free' | 'pro'
    kind = request["kind"]           # 'chat' | 'math' | 'code' | 'plan'
    latency_sla_ms = request["sla_ms"]

    if kind == "chat" or latency_sla_ms < 500:
        return {"model": "fast", "thinking": None}      # cheap, snappy
    if kind in {"math", "plan"}:
        effort = "high" if tier == "pro" else "medium"  # gate cost by tier
        return {"model": "reasoning", "thinking": effort}
    if kind == "code":
        return {"model": "reasoning", "thinking": "medium"}
    return {"model": "fast", "thinking": None}

print(reasoning_policy({"tier": "pro", "kind": "plan", "sla_ms": 5000}))
# {'model': 'reasoning', 'thinking': 'high'}
print(reasoning_policy({"tier": "free", "kind": "chat", "sla_ms": 300}))
# {'model': 'fast', 'thinking': None}

Knowledge check check yourself