Speculative & parallel decoding

The idea intermediate

Here's the frustration that makes this whole chapter worth it. A 70B model on a good GPU can crunch a whole prompt — thousands of tokens — in a single fast pass. Yet when it writes a reply, it slows to a crawl: one token, then a full forward pass, then the next token, then another full pass. The hardware is barely breaking a sweat — it's sitting idle waiting for the previous token before it's allowed to start the next. That's not a compute problem; it's a serialization problem, and speculative decoding is the trick that breaks it.

The analogy that makes it click: imagine a slow, careful editor who must approve every sentence of a book, one at a time, before the next can be written. Painful. Now put a fast junior writer next to them who drafts the next five sentences ahead on a hunch. The editor reads all five at once — reading five sentences takes them barely longer than reading one — and keeps every sentence that matches what they'd have written anyway, stopping at the first one they disagree with. When the junior guesses well, the editor approves five sentences in the time it used to take to approve one. The final book is word-for-word what the editor alone would have written — the junior only made it faster, never changed the content. That editor is your big model; the junior is the draft model.

Why reading five guesses costs almost the same as reading one is the load-bearing fact, and it's the same insight from IC3: decode is memory-bandwidth-bound, not compute-bound. Each forward pass spends most of its time hauling the model's weights from GPU memory; once those weights are loaded, checking one candidate token versus checking five is nearly free extra math. So the big model can verify a whole batch of k guessed tokens in one pass for roughly the price of the single token it was going to produce anyway. Guessing is cheap (small model); verifying is where the money is, and we get to verify in bulk.

The common misconception to kill right now: people hear "guessing" and assume the output must be an approximation, something you'd only turn on if you could tolerate slightly-wrong answers. Not so. The draft model never gets the final say — it only proposes. The big model verifies every token and accepts a guess only when it exactly matches the token it would have generated itself, keeping the correct prefix up to the first mismatch and then emitting its own token there. The result is mathematically identical to plain decoding, bit for bit. A bad draft model doesn't corrupt your output — it just wastes effort and slows you down. That's the safety guarantee that makes this a free lunch when it works.

Decode is serial: one token per expensive forward pass. Speculative decoding breaks the serialization: a small, fast draft model guesses the next few tokens, then the big model verifies them all in a single pass. Correct guesses are accepted for free; wrong ones are discarded. Same output distribution, fewer big-model passes — lower latency.

When it helps advanced

Speculative decoding is not a switch you flip and forget — it's a bet, and sometimes the bet loses money. Because you're paying for the draft model's guesses on top of the big model's verification, there's a real scenario where you spend extra and get nothing back: the draft keeps guessing wrong, almost nothing gets accepted, and you've just added the draft's cost to every step for a speedup of roughly zero — or worse, below 1×. So the honest question isn't "does it help?" but "does it help on my workload?" That answer hinges on one number.

That number is the acceptance rate: of the tokens the draft proposes, what fraction does the big model actually keep? Think of it as the junior writer's hit rate. If they nail four of every five guesses, the editor breezes through — huge speedup. If they're wrong four times out of five, the editor is constantly stopping to correct them, and you'd have been faster without the junior at all. Everything about "when it helps" reduces to: can a cheap model predict what the expensive model is about to say?

And that predictability is exactly what varies by workload. Generating code, JSON, or structured extraction at low temperature is highly predictable — after def reverse(lst): the next tokens are nearly forced, so a tiny draft model guesses them right almost every time and acceptance is high. Creative writing at high temperature is the opposite: the whole point is surprise and variety, the token distribution is deliberately flat, and even a good draft can't anticipate which of many plausible words the big model will roll. Low acceptance, little or no speedup. This is why the same feature is a clear win for a code-completion API and a waste for a story generator — same code, opposite verdict.

The mistake to avoid: judging the feature by end-to-end latency alone and concluding "it doesn't work." If a benchmark disappoints, the first thing to measure is the acceptance rate, because that tells you why — a low rate points at a mismatched draft model or too-high temperature, both fixable, whereas staring at latency tells you nothing about the cause. A second knob is k, how many tokens the draft guesses per round: guess too few and you leave speedup on the table; guess too many and every rejected tail token is wasted draft work. The table below sketches the three families of draft source you can choose from — a separate small model, extra prediction heads bolted onto the big model, or guessing straight from the prompt text itself.

Speedup depends on the acceptance rate — how often the draft guesses right. It helps most when text is predictable (code, structured output, low temperature) and a good small draft model exists. It helps less on highly creative, high-temperature generation where the draft rarely matches.

speculative.py# vLLM exposes speculative decoding as config — you enable, not implement it.
from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    speculative_model="meta-llama/Llama-3.2-1B-Instruct",  # small draft
    num_speculative_tokens=5,                               # guess 5 ahead
)
out = llm.generate(["Write a Python function to reverse a list."],
                   SamplingParams(max_tokens=128, temperature=0.0))
print(out[0].outputs[0].text)

def expected_accepted(k, p):
    # E[min(first miss position, k)] = sum_{i=1..k} p**i  (tokens accepted so far)
    return sum(p**i for i in range(1, k+1))

for p in (0.5, 0.8, 0.95):
    e = expected_accepted(4, p)
    print(f"p={p}: ~{e:.2f} tokens accepted per big-model pass")
# p=0.5: ~0.94   p=0.8: ~2.36   p=0.95: ~3.52

def speedup(k, p, draft_frac=0.15):
    accepted = sum(p**i for i in range(1, k+1))
    toks_per_round = accepted + 1          # +1: the verify step's own token
    # cost of a round = 1 big pass + k cheap draft passes (draft_frac each)
    cost = 1 + k * draft_frac
    return toks_per_round / cost

for p in (0.4, 0.7, 0.9):
    print(f"p={p}: ~{speedup(5, p):.2f}x")
# p=0.4: ~0.95x   p=0.7: ~1.68x   p=0.9: ~2.68x

def recommend(workload, temperature):
    predictable = workload in ("code", "structured", "extraction") and temperature <= 0.3
    if predictable:
        return "ENABLE — high acceptance likely; start k=5"
    if temperature >= 0.9:
        return "SKIP — creative/high-temp; draft rarely matches"
    return "MAYBE — benchmark; start k=3 and measure acceptance"

print(recommend("code", 0.1))          # ENABLE ... k=5
print(recommend("story", 1.0))         # SKIP
print(recommend("chat", 0.5))          # MAYBE ... k=3

def target_next(prefix):                 # the 'true' model: deterministic here
    table = {(): "the", ("the",): "cat", ("the","cat"): "sat",
             ("the","cat","sat"): "down"}
    return table.get(tuple(prefix), "END")

def speculative_round(prefix, draft_guess, k):
    # draft_guess: what the small model proposed (may be wrong)
    emitted = []
    for i in range(k):
        correct = target_next(prefix + emitted)
        if i < len(draft_guess) and draft_guess[i] == correct:
            emitted.append(correct)      # accepted for free
        else:
            emitted.append(correct)      # verify emits the correct token, then stop
            break
    return emitted

print(speculative_round([], ["the","cat","XXX"], k=3))  # ['the','cat','sat']
# 'the','cat' accepted from draft; 'XXX' rejected -> target's 'sat' emitted instead

# Draft-model speculative decoding
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --speculative-model meta-llama/Llama-3.1-8B-Instruct \  # small, fast draft
  --num-speculative-tokens 5                                # k

# Draft-free variant: n-gram / prompt-lookup (no second model to host)
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --speculative-model "[ngram]" \
  --num-speculative-tokens 5 \
  --ngram-prompt-lookup-max 4

def spec_accept(workload, temp):
    return ("code" in workload or "structured" in workload) and temp <= 0.3

workloads = [("code-completion", 0.1, "latency"),
             ("creative-writing", 1.0, "throughput")]
for name, temp, goal in workloads:
    if spec_accept(name, temp):
        print(f"{name}: SPECULATIVE — high acceptance cuts latency for a single stream")
    elif goal == "throughput":
        print(f"{name}: BIGGER BATCH — low acceptance kills spec; batch amortizes cost")
    else:
        print(f"{name}: benchmark both")

Knowledge check check yourself