Research map / Large language models / Reasoning and chain of thought
Flatter Tokens are More Valuable for Speculative Draft Model Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes flatness (cosine similarity of target distribution to uniform) as a token/sample importance score and SFDD, a one-pass offline filter that keeps the flattest 50% of ShareGPT samples to speed up EAGLE-2 draft model training with minimal loss in inference speedup.
Problem
Training draft models for speculative decoding requires large datasets, and not all training samples contribute equally to the acceptance rate; it is unclear which data matter most.
Approach
Tokens whose target-model distributions are flatter (closer to uniform, measured by cosine similarity to uniform) yield larger acceptance-related gains per training step; sample-level flatness averages token scores and retains high-flatness samples, computed offline from the target model alone.
Key results
- At 50% retain ratio, SFDD averages 2.41x speedup vs 2.49x for No Filter and 2.23x for Top-1 Probability (best baseline), with average acceptance length 2.56 vs 2.78.
- SFDD reduces training time from 58,227 s to 28,787 s (2.02x training speedup) at 50% retain on the stated setup.
- SFDD outperforms random filtering across retain ratios 5% to 70%, e.g., at 5% retain average speedup 1.82x vs 1.68x for random.
Surprising findings
- Filtering to 70% of the data slightly exceeds full-data speedup on Alpaca (2.77x vs 2.71x).
- Acceptance length does not translate proportionally to speedup; prompt length in prefilling affects speedup.
Limitations
- Theoretical analysis relies on Gaussian toy model and a single-step KD budget approximation.
- Experiments limited to one target model (LLaMA3-8B-Instruct) and the EAGLE-2 framework.
- Data-selection cost (2,242 s) is non-trivial and the main results use temperature 1.0 with a single dataset.
Experimental setup
Datasets: ShareGPT GSM8K Alpaca MT-Bench CNN/DM Natural Questions
Benchmarks: GSM8K Alpaca MT-Bench CNN/DM Natural Questions
Metrics: speedup average acceptance length training time
Baselines: Random filtering Entropy Top-1 Probability Margin Energy Score PPL No Filter
Builds on: EAGLE-2 speculative decoding knowledge distillation
Closest papers by meaning
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.91)
- Online Speculative Decoding (ICML 2024 · similarity 0.91)
- Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration (NeurIPS 2024 · similarity 0.91)
- A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.90)
- DistillSpec: Improving Speculative Decoding via Knowledge Distillation (ICLR 2024 · similarity 0.90)
- Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.