Research map / Large language models / Reasoning and chain of thought
EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EAGLE, a speculative sampling framework that drafts autoregressively at the second-to-top-layer feature level and feeds in a one-step-advanced token sequence to resolve feature sampling uncertainty, giving lossless LLM speedups.
Problem
Autoregressive decoding of LLMs is slow; existing speculative sampling needs a good small draft model (none exists for LLaMA2-7B-sized models) or has low-accuracy drafts, and feature-level drafting is hampered by sampling uncertainty.
Approach
Autoregress over second-to-top-layer features (simpler than tokens) with a lightweight decoder-layer draft head, and feed in the token sequence advanced by one step so the draft sees the sampled outcome, removing feature ambiguity; tree attention and multi-round speculative sampling keep output distribution exact.
Key results
- On MT-bench at temperature 0, EAGLE is 1.7x-2.1x faster than Lookahead and 1.5x-1.6x faster than Medusa.
- For LLaMA2-Chat 70B, EAGLE achieves a 2.7x-3.5x latency speedup and about 2x throughput (1.99x at bs>1 in the batch table).
- Feature&shifted-token input raised the speedup from about 1.9x (feature-only) to 2.8x on Vicuna 7B, showing resolving sampling uncertainty matters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use feature-level autoregression instead of token-level autoregression (ablation) | speculative draft model · MT-bench, Vicuna 7B | speedup: 1.5x to 1.9x | Helped |
| add a token sequence advanced by one time step to the feature inputs (ablation) | EAGLE draft model · MT-bench, Vicuna 7B | speedup: 1.9x to 2.8x | Helped |
| use tree draft and verification instead of chain draft and verification (ablation) | EAGLE · MT-bench, T=0 | average acceptance length: 3.20 to 3.94 (+0.74); 3.00 to 3.62 (+0.62); 3.23 to 3.98 (+0.75); 3.18 to 3.90 (+0.68); 2.97 to 3.68 (+0.71); 3.12 to 3. | Helped |
| use fixed ShareGPT answers rather than answers generated by the target LLM for training (ablation) | EAGLE · MT-bench, LLaMA2-Chat 7B, T=0 | speedup: 2.88x to 2.78x | Hurt |
| use feature inputs rather than token-only inputs (ablation) | draft model · MT-bench, Vicuna 7B | draft performance | Helped |
| combine feature and token inputs rather than use feature-only inputs (ablation) | draft model · MT-bench, Vicuna 7B | acceptance rate with one imprecise feature | Helped |
| use feature-and-shifted-token inputs rather than feature-and-unshifted-token inputs (ablation) | EAGLE draft model · MT-bench, Vicuna 7B | draft performance | Helped |
Surprising findings
- Using a 7B model as a draft for 13B targets yields no speedup, and simple speculative sampling with a 7B draft gives only 1.12x for 33B and 1.88x for 70B.
- Distillation of the draft model gave only a slight speedup gain, because draft-model overhead rather than acceptance rate is the bottleneck.
Limitations
- Mixtral 8x7B speedup is modest (1.5x) because MoE verification reads more expert weights.
- Acceptance rate drops with feature errors; the paper relies on noise augmentation and tree drafting to mitigate error accumulation.
- Tree structure was chosen by intuition rather than rigorously optimized; speedup diminishes at larger batch sizes.
Experimental setup
Datasets: ShareGPT
Benchmarks: MT-bench HumanEval GSM8K Alpaca
Metrics: walltime speedup ratio average acceptance length acceptance rate n-alpha throughput
Baselines: Lookahead Medusa Speculative sampling (7B draft) DistillSpec vanilla autoregressive decoding gpt-fast
Builds on: speculative sampling Medusa SpecInfer DistillSpec
Closest papers by meaning
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (NeurIPS 2025 · similarity 0.92)
- Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (ICLR 2025 · similarity 0.90)
- ConFu: Contemplate the Future for Better Speculative Sampling (ICML 2026 · similarity 0.89)
- Falcon: Faster and Parallel Inference of Large Language Models Through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree (AAAI 2025 · similarity 0.89)
- A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.88)
- Nearest Neighbor Speculative Decoding for LLM Generation and Attribution (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.