Speculative Decoding: The Inference Trick That Makes LLMs 3x Faster Without Sacrificing Quality

Speculative decoding cuts LLM inference latency by 2–3x without changing model weights, output quality, or the underlying probability distribution. It does this by exploiting a structural asymmetry that most inference pipelines ignore: verifying tokens is far cheaper than generating them from scratch.

The technique was formalized in DeepMind's 2023 paper and has since been adopted in production systems at Google, Meta, and a growing number of inference optimization frameworks including vLLM and TensorRT-LLM. Understanding why it works — and where it breaks down — is essential for any team running LLMs at scale.

How Speculative Decoding Actually Works

Standard autoregressive generation is sequential by nature: the model produces one token at a time, each conditioned on all previous tokens. This means a 70B-parameter model running on a single A100 generates roughly 20–30 tokens per second, spending the vast majority of its compute on memory bandwidth rather than arithmetic. The GPU is mostly waiting for weights to load from HBM, not crunching numbers.

Speculative decoding introduces a second, much smaller "draft" model — typically 7B parameters or fewer — that proposes a sequence of candidate tokens (usually 4–8 at a time) ahead of the larger "target" model. The target model then evaluates all draft tokens in a single forward pass. Because attention over a short sequence is nearly as fast as a single-token pass on a large model, the target can verify or reject each draft token in parallel.

When the draft tokens match what the target model would have generated (i.e., fall within its acceptance probability), all of them get accepted and the process advances multiple positions at once. When a draft token is rejected, the pipeline falls back to sampling from the target distribution at that position and restarts drafting from there. The key insight, proven rigorously in the original paper, is that this rejection sampling scheme produces an output distribution identical to what you would get from the target model alone — so quality is mathematically preserved.

Acceptance Rates and Where Draft Quality Matters

The speedup multiplier is directly proportional to the average acceptance rate of draft tokens. If the draft model accepts 80% of proposed tokens and you speculate 4 ahead, you effectively process ~3.2 tokens per target forward pass instead of 1. Real-world acceptance rates vary significantly by task:

Repetitive or templated outputs (code boilerplate, structured data): 85–95% acceptance, 3–4x speedup common
General chat and instruction following: 60–80% acceptance, 1.5–2.5x speedup typical
Creative generation and reasoning chains: 40–65% acceptance, more modest gains

This means speculative decoding works best when the task has predictable structure — which describes a large share of production LLM workloads: code generation, document summarization with consistent formatting, and chatbot responses following prompt templates. It performs less well on open-ended creative tasks where the draft model systematically diverges from the target.

Self-Speculative Decoding and Medusa Heads

Running a separate draft model adds operational complexity: you need to colocate two models, manage memory for both, and keep them synchronized. Several approaches have emerged to address this.

Self-speculative decoding uses the target model itself as the draft generator, but with early-exit layers. Llama 3.1's 70B model, for instance, can be configured to exit at layer 32 (instead of the full 80 layers) for drafting, then re-enter the full stack for verification. This eliminates the need for a separate model but requires architecture support for layer-skipping.

Medusa, developed at Princeton and open-sourced in 2024, takes a different approach: it adds multiple additional prediction heads to the target model's final hidden state, each predicting tokens at positions +1, +2, +3, etc. These heads are trained cheaply via supervised fine-tuning on existing model outputs. Medusa-2, combined with tree-based verification, achieves 2.8x speedup on Vicuna-13B without a separate draft model and with minimal accuracy degradation.

SpecInfer from Carnegie Mellon extends the idea further by using a tree of draft sequences rather than a single sequence, verified in a single batched forward pass. This raises acceptance probability because the target model only needs to find one valid path in the tree.

Production Considerations

Deploying speculative decoding in production requires attention to several factors that benchmarks often understate:

Batch size interaction: Speculative decoding provides the largest gains at batch size 1 (interactive use cases). As batch size grows, the memory bandwidth bottleneck shrinks relative to arithmetic, and the overhead of running the draft model becomes less justified. At batch sizes above 32–64, standard generation often outperforms speculative decoding.
KV cache management: The draft model and target model both maintain KV caches. Memory allocation must account for both, complicating serving frameworks that optimize cache utilization (like vLLM's PagedAttention).
Draft model selection: The draft model should be from the same family as the target. Using Llama-3-8B to draft for Llama-3-70B yields much higher acceptance rates than using a model trained on a different dataset or with different tokenization.
Latency vs. throughput tradeoff: Speculative decoding optimizes for latency (time to complete a single request) more than throughput (tokens per second across all requests). Teams running high-QPS inference serving need to evaluate which metric matters more for their SLA.

Where the Technique Is Heading

Google's 2024 deployment of speculative decoding in Gemini's serving infrastructure reportedly cut inference costs by 40% for latency-sensitive workloads. NVIDIA integrated speculative decoding support into TensorRT-LLM 0.8 with multi-token prediction (MTP) heads, and the approach is now a first-class feature rather than an experimental plugin.

The next frontier is cascade speculative decoding: using three or more models in a hierarchy (e.g., 1B → 7B → 70B), where the smallest model drafts for the medium model, which drafts for the large model. Early results from Meta's research team suggest 4–5x latency reduction on structured tasks with this approach, though memory footprint and orchestration complexity increase accordingly.

There is also active work on speculative decoding for mixture-of-experts models, where the irregular compute patterns of sparse expert routing interact in non-obvious ways with draft verification. Mistral's Mixtral architecture has been specifically examined for this; early results show the acceptance rate dynamics differ meaningfully from dense models.

Actionable Takeaways

If your workload is latency-sensitive and batch size is low (1–8), speculative decoding should be your first optimization to evaluate — frameworks like vLLM support it with a single config flag.
Match your draft model to your target model's family and training data; mismatched drafts will give you poor acceptance rates and may actually slow things down.
Measure acceptance rates on your actual production prompts, not benchmarks. Task structure matters more than model size for predicting speedup.
For teams who cannot run a second model, evaluate Medusa heads as a middle ground — they add minimal memory overhead and can be fine-tuned in hours on a single GPU.
Do not apply speculative decoding to high-throughput batch inference unless you've verified the batch-size dynamics at your specific operating point.