Research map / Large language models / Reasoning and chain of thought
EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EAGLE-3, a speculative decoding draft model that drops feature prediction, fuses low/mid/high target-layer features, and uses training-time test so it keeps improving with more training data.
Problem
EAGLE's feature-level autoregression constrains the draft model, so scaling up its training data yields limited speedup gains for LLM inference acceleration.
Approach
Remove the feature prediction loss so the draft model predicts tokens directly, fuse low-, mid-, and high-level target features instead of top-layer features, and simulate multi-step drafting during training (training-time test) so the draft model adapts to its own predictions.
Key results
- EAGLE-3 reaches up to 6.5x speedup (5.51x mean on Vicuna 13B at temperature 0) on HumanEval, versus 4.22x for EAGLE-2 mean on that model.
- On SGLang with H100 and LLaMA-Instruct 3.1 8B, EAGLE-3 gives 1.38x throughput over no-speculation at batch size 64, while EAGLE drops below 1x beyond batch size 24.
- Ablation on MT-bench with LLaMA-Instruct 3.1 8B: EAGLE-2 3.16x; removing feature constraint 3.82x; adding fused features 4.40x.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace EAGLE-2 with EAGLE-3 | speculative decoding · MT-bench, HumanEval, GSM8K, Alpaca, CNN/DM mean; temperature=0 | speedup: 4.22x -> 5.51x; 3.23x -> 4.44x; 2.85x -> 4.12x; 3.26x -> 4.16x | Helped |
| replace EAGLE-2 with EAGLE-3 | speculative decoding · MT-bench, HumanEval, GSM8K, Alpaca, CNN/DM mean; temperature=1 | speedup: 3.76x -> 4.65x; 2.80x -> 3.45x; 2.65x -> 3.95x; 2.77x -> 3.52x | Helped |
| remove the feature prediction constraint (ablation) | EAGLE-3 draft model · MT-bench | speedup: 3.16x -> 3.82x | Helped |
| remove the feature prediction constraint (ablation) | EAGLE-3 draft model · GSM8K | speedup: 3.39x -> 3.77x | Helped |
| replace top-layer features with fused low-, middle-, and high-level features (ablation) | EAGLE-3 draft model · MT-bench | speedup: 3.82x -> 4.40x | Helped |
| replace top-layer features with fused low-, middle-, and high-level features (ablation) | EAGLE-3 draft model · GSM8K | speedup: 3.77x -> 4.48x | Helped |
| use EAGLE-3 rather than EAGLE | speculative decoding in SGLang · MT-bench, batch size 64 | throughput improvement: 0.99x -> 1.38x | Helped |
| use EAGLE-3 rather than EAGLE-2 | speculative decoding in SGLang · MT-bench, batch size 1 | throughput: 244.10 tokens/s -> 373.25 tokens/s | Helped |
Surprising findings
- Speculative sampling methods like EAGLE-3 still yield throughput gains at large batch sizes in SGLang, contrary to the common belief that speculation hurts throughput at high batch size.
- EAGLE shows throughput below baseline at batch sizes beyond 24 in SGLang, while EAGLE-3 retains gains.
Limitations
- Not evaluated on 405B or 671B models due to GPU constraints.
- Generation quality not evaluated; relies on lossless speculative sampling guarantee.
- Gains on DeepSeek-R1-Distill-LLaMA 8B GSM8K may be partly due to training on OpenThoughts-114k-math.
Experimental setup
Datasets: ShareGPT UltraChat-200K OpenThoughts-114k-math MT-bench HumanEval GSM8K Alpaca CNN/Daily Mail
Benchmarks: MT-bench HumanEval GSM8K Alpaca CNN/Daily Mail
Metrics: speedup ratio average acceptance length acceptance rate n-alpha throughput
Baselines: Speculative sampling (Vicuna-68M draft) PLD Medusa Lookahead Hydra HASS EAGLE EAGLE-2
Builds on: EAGLE EAGLE-2 speculative sampling HASS Medusa
Closest papers by meaning
- EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (ICML 2024 · similarity 0.92)
- EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism (ICML 2024 · similarity 0.89)
- PARD: Accelerating LLM Inference with Low‑Cost PARallel Draft Model Adaptation (ICLR 2026 · similarity 0.87)
- Nearest Neighbor Speculative Decoding for LLM Generation and Attribution (NeurIPS 2024 · similarity 0.87)
- Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs (ICLR 2026 · similarity 0.87)
- ATTS: Asynchronous Test-Time Scaling via Conformal Prediction (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.