Research map / Architectures and efficiency / Attention and transformers
Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Gumiho, a hybrid speculative decoding draft head that uses a serial two-layer Transformer for early draft tokens and parallel MLP heads for later ones, plus full tree attention, and reports speedups over Eagle-2 on several LLMs.
Problem
Existing multi-head speculative decoding methods treat all draft positions as equally important and use a single serial or parallel paradigm, which limits mean accepted tokens and drafting speed.
Approach
Early draft tokens matter more for accepted length, so allocate a serial Transformer to the first two positions and lightweight parallel MLPs to the remaining five, and add full tree attention that lets shorter candidate paths borrow tokens from longer paths at matching positions without extra computation.
Key results
- At temperature 0 on Vicuna 7B, Gumiho reaches 2.97x mean speedup and tau 4.89 versus Eagle-2's 2.74x and 4.68.
- On LLaMA3 70B at temperature 0, Gumiho achieves 3.45x mean speedup vs Eagle-2's 2.98x, reported as a 15.8% gain.
- Full tree attention ablation on MT-Bench (Vicuna 7B) raises speedup from 3.10x to 3.15x and tau from 5.18 to 5.29.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Gumiho instead of Eagle-2 | speculative decoding · Mean across six datasets, temperature 0, Vicuna-7B | speedup: 2.74× -> 2.97× | Helped |
| use Gumiho instead of Eagle-2 | speculative decoding · Mean across six datasets, temperature 0, LLaMA2-70B | speedup: 2.47× -> 2.76× | Helped |
| use Gumiho instead of Eagle-2 | speculative decoding · Mean across six datasets, temperature 0, LLaMA3-70B | speedup: 2.98× -> 3.45× | Helped |
| use Gumiho instead of Eagle-2 | speculative decoding · Mean across six datasets, temperature 1, LLaMA3-70B | speedup: 2.91× -> 3.37× | Helped |
| add Full Tree Attention (ablation) | Gumiho · MT-Bench, temperature 0, Vicuna-7B | speedup: 3.10× -> 3.15× | Helped |
| add Full Tree Attention (ablation) | Gumiho · MT-Bench, temperature 0, Vicuna-7B | mean accepted tokens (τ): 5.18 -> 5.29 | Helped |
| reduce the Transformer serial-head depth below 2 layers (ablation) | Gumiho · MT-Bench, temperature 0, Vicuna-7B | mean accepted tokens (τ) | Hurt |
| increase the Transformer serial-head depth from 2 to 3 layers (ablation) | Gumiho · MT-Bench, temperature 0, Vicuna-7B | speedup | Hurt |
| increase the Transformer serial-head depth from 2 to 3 layers (ablation) | Gumiho · MT-Bench, temperature 0, Vicuna-7B | mean accepted tokens (τ) | Helped |
| increase the number of parallel MLP heads from the default width (ablation) | Gumiho · MT-Bench, temperature 0, Vicuna-7B | mean accepted tokens (τ) | Mixed |
Surprising findings
- Adding more MLP parallel heads initially helps but degrades performance beyond a point, attributed to the shared embedding being overloaded.
Limitations
- Draft model is more parameter-heavy than Eagle and Medusa, increasing training GPU memory.
- The theorem assumes acceptance probabilities are fixed/position-dependent and uses a simplified uniform example; the gains depend on the assumed improved distribution.
- Evaluated only on ShareGPT-trained heads and a fixed set of target LLMs and benchmarks.
Experimental setup
Datasets: ShareGPT
Benchmarks: MT-Bench HumanEval GSM8K Alpaca CNN/Daily Mail Natural Questions
Metrics: speedup ratio mean accepted tokens (tau) draft time
Baselines: Medusa Hydra Eagle Eagle-2
Builds on: Medusa Hydra Eagle Eagle-2
Closest papers by meaning
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.91)
- SpecBranch: Speculative Decoding via Hybrid Drafting and Rollback-Aware Branch Parallelism (ICLR 2026 · similarity 0.90)
- TokenSwift: Lossless Acceleration of Ultra Long Sequence Generation (ICML 2025 · similarity 0.90)
- GRIFFIN: Effective Token Alignment for Faster Speculative Decoding (NeurIPS 2025 · similarity 0.90)
- polybasic Speculative Decoding Through a Theoretical Perspective (ICML 2025 · similarity 0.90)
- Mixture of Attentions For Speculative Decoding (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.