Speculative & parallel decoding
Speculative decoding breaks decode's serialization: a cheap draft model guesses tokens the big model verifies in one pass. Same output, lower latency — when the draft guesses well.
- a GPU +
pip install vllm(serving engine)
Learning objectives
- Explain the core idea of speculative decoding: guess cheaply, verify in one pass.
- Reason about when a draft model helps and when it doesn't.
- Name related techniques (medusa, n-gram, prompt lookup).
- Decide whether speculative decoding fits your workload.
code/ic5-speculative/ in the course, with a README. Run the scripts or copy the configs directly.The idea intermediate
Decode is serial: one token per expensive forward pass. Speculative decoding breaks the serialization: a small, fast draft model guesses the next few tokens, then the big model verifies them all in a single pass. Correct guesses are accepted for free; wrong ones are discarded. Same output distribution, fewer big-model passes — lower latency.
This shows the trick behind speculative decoding. Normally the big model writes one token per slow step. Here a tiny, fast model guesses several tokens ahead, and the big model checks a whole batch of guesses in a single step — so several tokens can come out of one expensive pass.
- Draft: guess k (small model) — a small, cheap model quickly proposes the next
ktokens (say 5). It may be wrong, but it's fast. - Verify in 1 pass (big model) — the real, expensive model checks all
kguesses at once in one forward pass, instead of one token at a time. - Accept prefix (correct = free) — the big model keeps the guesses that match what it would have said anyway, up to the first mistake. Those accepted tokens cost almost nothing extra.
- Repeat (continue) — start over from where we got to and guess the next batch. If the draft guesses well, many tokens are produced per big-model pass.
- Read left to right as one round; the loop repeats until the answer is done.
In short: Speculative decoding = a cheap model guesses ahead, the big model verifies in one shot and keeps the correct guesses. The output is identical to normal decoding — only faster when guesses land often.
When it helps advanced
Speedup depends on the acceptance rate — how often the draft guesses right. It helps most when text is predictable (code, structured output, low temperature) and a good small draft model exists. It helps less on highly creative, high-temperature generation where the draft rarely matches.
| Technique | Draft source | Note |
|---|---|---|
| Draft-model spec. | a small model | classic; needs a matching small model |
| Medusa | extra heads on the model | no separate draft model |
| Prompt lookup / n-gram | the prompt itself | great for RAG/repetitive text |
speculative.py# vLLM exposes speculative decoding as config — you enable, not implement it.
from vllm import LLM, SamplingParams
llm = LLM(
model="meta-llama/Llama-3.1-70B-Instruct",
speculative_model="meta-llama/Llama-3.2-1B-Instruct", # small draft
num_speculative_tokens=5, # guess 5 ahead
)
out = llm.generate(["Write a Python function to reverse a list."],
SamplingParams(max_tokens=128, temperature=0.0))
print(out[0].outputs[0].text)
This shows that turning on speculative decoding is a configuration, not an algorithm you write. The vLLM serving engine does the guess-and-verify dance for you; you just name a small draft model and how many tokens it should guess ahead.
LLM(model="...Llama-3.1-70B-Instruct")is the big, accurate model that produces the final answer — slow on its own.speculative_model="...Llama-3.2-1B-Instruct"names the small, fast draft model that does the guessing. A 1B model guessing for a 70B model is a typical pairing.num_speculative_tokens=5tells the draft to guess 5 tokens ahead each round; the big model then verifies those 5 in one pass.SamplingParams(max_tokens=128, temperature=0.0)requests a deterministic answer (temperature=0.0). Predictable text is exactly where the draft guesses best, so speculative decoding helps most here.
What the output means: The reversed-list function, produced faster than the 70B model alone would manage — because many tokens were accepted from the cheap draft's guesses.
Try this: Raise temperature to 0.9 for creative output. The draft will guess wrong more often, fewer tokens get accepted, and the speedup shrinks — that's the acceptance rate at work.
Exercise IC5.1 — Does it help your workload?
Context: Whether speculative decoding helps is entirely a function of acceptance rate, so comparing a deterministic and a creative workload makes the dependence obvious.
Your task: Enable speculative decoding on a served model for two workloads — deterministic code generation and creative writing — compare TPOT and acceptance rate, and explain why one benefits far more than the other.
Requirements:
- Enable speculative decoding for both workloads
- Measure TPOT and acceptance rate for each
- Show the deterministic workload accepts far more
- Explain acceptance rate drives the speedup
- Conclude which workload benefits and why
💡 Hint: The low-temperature deterministic workload lets the draft land its guesses; the creative one doesn't, so its acceptance — and speedup — collapses.
🪜 Practice ladder beginner → industry
Six graded exercises, easy to real-world. Try each before opening its solution.
Context: In speculative decoding a cheap draft model proposes k tokens that the big model verifies in one pass, accepting the correct prefix; if each token is accepted independently with probability p, the expected accepted run is a short geometric series.
Your task: Compute the expected number of accepted tokens per verify pass for k=4, p=0.8.
Requirements:
- Expected accepted =
Σ p^ifor i=1..k (truncated geometric) - Evaluate at k=4 for a few acceptance rates
- Note the big model always emits one more token on verify
- Show higher acceptance means more tokens per expensive pass
💡 Hint: Sum p, p², …, p^k — that truncated series is the expected run before the first miss.
Show solution
The accepted-prefix length is a truncated geometric expectation:
def expected_accepted(k, p):
# E[min(first miss position, k)] = sum_{i=1..k} p**i (tokens accepted so far)
return sum(p**i for i in range(1, k+1))
for p in (0.5, 0.8, 0.95):
e = expected_accepted(4, p)
print(f"p={p}: ~{e:.2f} tokens accepted per big-model pass")
# p=0.5: ~0.94 p=0.8: ~2.36 p=0.95: ~3.52Plus the one token the big model always produces on verify, higher acceptance means more tokens emitted per expensive pass — the whole source of the speedup.
Context: Speculative decoding does one big pass per (accepted + 1) tokens but adds cheap draft cost, so at low acceptance the overhead nearly cancels the win — it only pays when the draft guesses right often.
Your task: Model the effective speedup for k=5 at several acceptance rates, including the draft's cost.
Requirements:
- Tokens per round =
accepted + 1 - Round cost =
1 + k × draft_frac(one big pass + k cheap drafts) - Compute speedup = tokens per round / round cost
- Evaluate across low, medium, and high acceptance
- Show low acceptance can make speedup ≤ 1×
💡 Hint: Divide the expected accepted-plus-one tokens by the cost of one big pass plus the k cheap draft passes — below some acceptance the ratio dips under 1.
Show solution
Turn acceptance into an effective tokens-per-pass ratio:
def speedup(k, p, draft_frac=0.15):
accepted = sum(p**i for i in range(1, k+1))
toks_per_round = accepted + 1 # +1: the verify step's own token
# cost of a round = 1 big pass + k cheap draft passes (draft_frac each)
cost = 1 + k * draft_frac
return toks_per_round / cost
for p in (0.4, 0.7, 0.9):
print(f"p={p}: ~{speedup(5, p):.2f}x")
# p=0.4: ~0.95x p=0.7: ~1.68x p=0.9: ~2.68xAt low acceptance the draft overhead nearly cancels the win; at high acceptance you approach a 2x+ speedup. Speculative decoding is only worth it when the draft guesses right often.
Context: Acceptance depends on predictability: low-temperature code/structured generation guesses well, high-temperature creative writing doesn't — so speculative decoding is worth enabling only in the predictable regime.
Your task: Write a helper that recommends whether to enable speculative decoding and a starting k, given workload type and temperature.
Requirements:
- Predictable workload (code/structured/extraction) + low temp → ENABLE, start k=5
- High temp (≥0.9) → SKIP
- Otherwise → MAYBE, benchmark, start k=3
- Reinforce that low-temp predictable generation is where a small draft lands
💡 Hint: Gate on both the workload type and the temperature — only the predictable-and-cool corner is a confident ENABLE.
Show solution
Encode the lesson's 'when it helps' guidance:
def recommend(workload, temperature):
predictable = workload in ("code", "structured", "extraction") and temperature <= 0.3
if predictable:
return "ENABLE — high acceptance likely; start k=5"
if temperature >= 0.9:
return "SKIP — creative/high-temp; draft rarely matches"
return "MAYBE — benchmark; start k=3 and measure acceptance"
print(recommend("code", 0.1)) # ENABLE ... k=5
print(recommend("story", 1.0)) # SKIP
print(recommend("chat", 0.5)) # MAYBE ... k=3Predictable, low-temperature generation (code, JSON, extraction) is where a small draft model lands its guesses; high-temperature creative text is where it wastes effort.
Context: Speculative decoding's key guarantee is that the output is identical to normal decoding: on each position it emits the accepted draft token or the target's own correct token, never a draft token the target wouldn't have produced.
Your task: Simulate one draft-and-verify round — draft proposes tokens, verify accepts the matching prefix and always emits one correct token — and show the emitted sequence equals what the target alone would produce.
Requirements:
- Model the target as a deterministic next-token lookup (the “true” model)
- Accept a draft token only when it matches the target's next token
- On a mismatch, emit the target's own token and stop the round
- Show a mixed example (some accepted, one rejected-and-replaced)
- Prove every emitted token is the target's own — output is exactly plain decoding
💡 Hint: Verification only accepts a draft token when it equals what the target would have emitted anyway, so the output can't diverge — speculation changes speed, never the tokens.
Show solution
A deterministic sim showing verify never changes the output:
def target_next(prefix): # the 'true' model: deterministic here
table = {(): "the", ("the",): "cat", ("the","cat"): "sat",
("the","cat","sat"): "down"}
return table.get(tuple(prefix), "END")
def speculative_round(prefix, draft_guess, k):
# draft_guess: what the small model proposed (may be wrong)
emitted = []
for i in range(k):
correct = target_next(prefix + emitted)
if i < len(draft_guess) and draft_guess[i] == correct:
emitted.append(correct) # accepted for free
else:
emitted.append(correct) # verify emits the correct token, then stop
break
return emitted
print(speculative_round([], ["the","cat","XXX"], k=3)) # ['the','cat','sat']
# 'the','cat' accepted from draft; 'XXX' rejected -> target's 'sat' emitted insteadWhatever the draft guesses, every emitted token is the target model's own token, so the final sequence is exactly what plain decoding would produce. Speculation changes speed, never the output.
Context: vLLM supports speculative decoding with either a draft model or a draft-free n-gram/prompt-lookup variant; the draft model needs a well-matched small model, n-gram guesses from the prompt itself.
Your task: Show the real vLLM flags for a draft model and mention the n-gram/prompt-lookup draft-free variant, using no invented flags and labelling it as needing a GPU.
Requirements:
- Draft-model launch:
--speculative-model+--num-speculative-tokens(k) - Draft-free launch:
--speculative-model "[ngram]"with a prompt-lookup window - Contrast: draft models need a matched small model; n-gram needs none
- Note n-gram/prompt-lookup only helps on repetitive text
- Label it as needing a GPU
💡 Hint: One flag names the draft model and one sets k; the n-gram variant swaps the draft model for a prompt-lookup window and costs no second model.
Show solution
Real vLLM config — needs a GPU + pip install vllm:
# Draft-model speculative decoding
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.1-70B-Instruct \
--speculative-model meta-llama/Llama-3.1-8B-Instruct \ # small, fast draft
--num-speculative-tokens 5 # k
# Draft-free variant: n-gram / prompt-lookup (no second model to host)
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.1-70B-Instruct \
--speculative-model "[ngram]" \
--num-speculative-tokens 5 \
--ngram-prompt-lookup-max 4A draft model needs a well-matched small model; n-gram/prompt-lookup skips that by guessing from the prompt itself, cheaper to run but only helps on repetitive text.
Context: Speculative decoding attacks single-stream latency while batching attacks throughput — different axes — so on a mixed fleet the lever is chosen per workload, not globally.
Your task: You serve a code-completion API (low temp, latency-critical) and a creative-writing API (high temp, throughput-focused). Decide per workload whether speculative decoding or a bigger batch is the better lever and justify.
Requirements:
- Flag high acceptance for code/structured at low temp
- Code-completion (low temp, latency goal) → SPECULATIVE
- Creative-writing (high temp, throughput goal) → BIGGER BATCH
- Explain low acceptance kills spec; batching amortizes cost
- Conclude the choice is per-workload, not global
💡 Hint: Match the lever to the goal: speculative cuts latency where acceptance is high, batching raises throughput where it isn't.
Show solution
Match the lever to each workload's shape:
def spec_accept(workload, temp):
return ("code" in workload or "structured" in workload) and temp <= 0.3
workloads = [("code-completion", 0.1, "latency"),
("creative-writing", 1.0, "throughput")]
for name, temp, goal in workloads:
if spec_accept(name, temp):
print(f"{name}: SPECULATIVE — high acceptance cuts latency for a single stream")
elif goal == "throughput":
print(f"{name}: BIGGER BATCH — low acceptance kills spec; batch amortizes cost")
else:
print(f"{name}: benchmark both")Speculative decoding attacks single-stream latency where guesses land (code), while batching attacks throughput where many requests overlap (creative). They optimize different axes, so the right answer is per-workload, not one global setting.
✓ Checkpoint — you can move on when you can…
- Explain speculative decoding and why it preserves the output.
- Explain how acceptance rate drives the speedup.
- Name medusa and prompt-lookup as alternatives.
- Decide whether it fits a given workload from its predictability.
Knowledge check check yourself
Explain speculative decoding, and why it produces output identical to normal decoding rather than an approximation.
Show answer
What determines the speedup from speculative decoding, and why does deterministic code generation benefit more than high-temperature creative writing?