Chapter 9 — Speculative Decoding

Ninth post of the chapter-by-chapter walkthrough of LLM Primer VI: Scaling AI Systems. The chapter that turns out the sequential bottleneck of autoregression has a mathematical loophole — and shows the arithmetic of when it pays off.

Why this chapter exists

Chapters 7 and 8 attacked concurrency, keeping the GPU busy across many sequences. Neither changes the fact that each sequence still receives one output token per iteration. That floor looks inescapable because the transformer's forward pass genuinely does depend on the previously chosen token. Speculative decoding is the observation that the dependence is on generation, not on verification: given a candidate for token t+1, the target model can run one forward pass that both asks "would I have chosen this?" and computes the distribution for t+2 assuming the candidate is correct. Chapter 9 walks the mechanism, the draft algorithms (EAGLE, Medusa, Lookahead, MTP, n-gram, suffix), and the arithmetic of when speculation is a win.

9.1 The verification insight preserves correctness exactly

A cheap draft mechanism proposes N candidate tokens. The target — the expensive model we want — runs one forward pass on the prefix concatenated with the N candidates. Because a transformer's forward pass is parallel across positions, the target produces its own next-token distribution at each of the N+1 positions in that single pass. The longest prefix on which the candidates agree with the target is accepted; the first disagreement is where the engine commits to the target's own choice and discards the rest. The acceptance rule — accept x drawn from draft q with probability min(1, p(x)/q(x)), otherwise sample from the normalized residual (p−q)₊ — makes the output distribution identical to plain target sampling. Speculation is correctness-preserving; the draft affects only speed. On modern GPUs the verification pass on N=4 costs roughly 1.2–1.5× a normal decode step, because the attention now spans N+1 query positions but remains well inside the memory-bound regime.

9.2 EAGLE ties the draft to the target's hidden state

Early implementations used a separate small model as the draft — Llama-7B drafting for Llama-70B — which works but costs a second HBM stream for the draft's weights and caps acceptance because the two models do not share representations. EAGLE, refined through EAGLE-2 and EAGLE-3 across 2024–2025, ties the draft to the target: a single transformer layer trained to predict the target's next-layer hidden state, projected through the target's own output embedding. There is no separate 7B set of weights to stream; the draft head is a few hundred megabytes. EAGLE-2 adds dynamic draft-tree expansion — a tree of candidates verified in one attention pass with a custom mask — so the target picks the best of several paths instead of the draft's single guess. EAGLE-3 adds multi-layer feature mixing, consuming intermediate target layers rather than just the penultimate one. Acceptance rates land at 75–85 percent on chat and code with N in 5–8, translating to end-to-end speedups of 3–4× over plain decoding. Medusa takes the alternate route — parallel draft heads that predict several future tokens in one pass rather than autoregressively — with a simpler training story and slightly lower acceptance. N-gram and suffix decoding are the free lunch for repetitive workloads (code editing, template output) where the draft is just a lookup into recent context.

9.3 The verification pass itself becomes the ceiling

The speedup formula is exact enough to write down: E[accepted_tokens] / (T_draft/T_decode + 1 + α·N), where α is the per-position verification overhead, typically 0.05–0.15. Two ceilings show. First, α·N grows with N, so increasing N indefinitely doesn't help; the peak is around N=6–8 at p=0.8, N=10–12 at p=0.9. Engines that hard-code N miss the optimum on workload variation. Second, and more subtly, the asymptotic speedup as p approaches 1 is 1/α — around 10× for α=0.10, 20× for α=0.05 — because the verification pass itself is the wall. Production engines have measured peak speedups in this range on highly repetitive workloads and have not exceeded them. Speculation also interacts with batching: at high batch sizes the target's forward pass is already close to compute-bound, and the extra verification tokens push it further, so speedup shrinks as batch grows. The sweet spot for speculation is low-to-medium batch size on latency-critical workloads — precisely the regime where the underused compute of Chapter 1 lives.

What Chapter 9 sets up

Chapters 5 through 9 have exhausted the model-and-runtime toolkit — quantization, pruning, distillation, batching, paged KV, speculative decoding. None of these ships as a library you can pip install and forget. Someone has to wire them together into a runtime that owns a model on a GPU, runs a continuous batch, and exposes an inference API. That runtime is the engine. Chapter 10 walks the five engines that dominate the 2026 stack — vLLM, TensorRT-LLM, SGLang, TGI, and Ollama — and the mechanical trade-offs that decide which is the right choice for a given deployment.

Next — Chapter 10: The LLM Engine Layer. vLLM, TensorRT-LLM, SGLang, TGI, Ollama — engines as the single-node runtime, chosen on mechanism rather than benchmark.