Speculative Decoding Cuts LLM Inference Latency by Up to 3x Without Accuracy Loss
Large language models generate text one token at a time, and every token requires a full forward pass through billions of parameters. This serial dependency is the core reason inference is expensive. Speculative decoding breaks that bottleneck — not by changing the model, but by changing the generation strategy. The technique can reduce wall-clock latency by 2x to 3x on tasks like code completion and chat, with no degradation in output quality.
The Core Mechanism
Speculative decoding uses two models: a small "draft" model and the large "target" model. The draft model generates several candidate tokens quickly. The target model then evaluates all those candidates in a single parallel forward pass — accepting tokens that it would have predicted itself and rejecting the rest. When a token is rejected, generation falls back to the target model's distribution for that position, and the process restarts.
Because the target model's forward pass over a batch of candidate tokens is only marginally more expensive than a single-token forward pass (thanks to GPU parallelism), the net result is more tokens generated per unit of compute. The math works when the draft model has reasonable agreement with the target — typically 70–85% token acceptance rates on factual and structured tasks like code generation.
Why Acceptance Rate Is Everything
The speedup from speculative decoding scales directly with the average number of draft tokens accepted before a rejection. On common coding benchmarks like HumanEval, acceptance rates with a well-matched draft model hover around 75–80%, yielding 2.5–3x latency reduction. On open-ended creative tasks, acceptance rates drop to 55–65%, and the speedup shrinks to 1.5–2x.
This means the choice of draft model matters enormously. Research from DeepMind in 2023 (the original speculative decoding paper by Leviathan et al.) showed that even a 3-order-of-magnitude size difference — a 7B draft vs. a 70B target — still achieves meaningful speedup because the smaller model's predictions are surprisingly aligned with the larger one on structured tasks.
Self-Speculative Decoding: No Draft Model Required
One practical barrier to speculative decoding in production is the overhead of running and maintaining a separate draft model. Self-speculative decoding, introduced in 2024 by researchers at CMU and Microsoft, eliminates this requirement. The approach uses early exit from intermediate layers of the target model itself as the draft mechanism. Specifically, it routes tokens through a subset of the model's layers to produce a fast draft, then validates with the full model.
The EAGLE-2 method (from researchers at Peking University, also 2024) takes a different approach: it trains a lightweight single-layer "draft head" that attaches to the target model and predicts future tokens based on internal hidden states. EAGLE-2 achieved acceptance rates above 80% on MT-Bench and outperformed prior speculative methods by 20–40% in throughput on A100 GPUs. The draft head adds less than 1% to the model's parameter count.
Production Deployments
Speculative decoding is no longer just a research curiosity. Google's production serving infrastructure for Gemini uses it. Anthropic has described using speculative approaches in Claude serving. The vLLM inference framework (the most widely used open-source LLM serving library, with over 30,000 GitHub stars) shipped speculative decoding support in version 0.3 in early 2024.
For organizations running their own inference stacks, the practical implications are direct: the same hardware serving a 70B model at 20 tokens/second can serve 50–60 tokens/second with properly tuned speculative decoding. That's a 2.5–3x reduction in cost-per-token without any model changes, quantization, or accuracy tradeoffs.
Limitations and When It Doesn't Help
Speculative decoding helps with latency — the time to generate a response — but it doesn't reduce total compute. In fact, it slightly increases total FLOPs because of rejected draft tokens. This means it doesn't lower energy costs per request; it lowers time-to-completion, which matters for user-facing latency but not for batch processing throughput.
It also performs worst on high-entropy tasks: creative writing, brainstorming, or any output where the model has high uncertainty at each step. In these cases, draft acceptance rates fall below 60% and the overhead of running the draft model starts to eat into gains.
Actionable Takeaways
- If you're running Llama 3.1 70B or similar models with vLLM: enable speculative decoding with a matching smaller model (e.g., Llama 3.2 3B as draft). Expect 2–2.5x latency improvement on chat/code tasks with minimal configuration.
- If you're building on hosted APIs: speculative decoding is likely already running in the backend. Focus your optimization effort on prompt structure and token efficiency instead.
- If latency is your bottleneck but not cost: speculative decoding is your best lever — it beats quantization for quality-sensitive tasks and doesn't require model retraining.
- If you're doing batch inference (summarization, classification at scale): speculative decoding won't help. Look at continuous batching and quantization instead.