Research map / Large language models / Reasoning and chain of thought
Speculative Decoding with Big Little Decoder
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add prediction alignment by fine-tuning the small model on large-model outputs | BiLD · IWSLT 2017 De-En, WMT 2014 De-En, XSUM, CNN/DailyMail | speedup: 1.43$\times$ -> 1.62$\times$; 1.34$\times$ -> 1.47$\times$; 1.48$\times$ -> 1.50$\times$; 1.71$\times$ -> 1.85$\times$ | Helped |
| Add prediction alignment by fine-tuning the small model on large-model outputs | BiLD · IWSLT 2017 De-En, WMT 2014 De-En, XSUM, CNN/DailyMail | BLEU / ROUGE-L: 40.33 -> 40.24; 31.28 -> 31.26; 35.12 -> 35.05; 41.44 -> 41.52 | Mixed |
| Remove the rollback policy (ablation) | Aligned BiLD · IWSLT 2017 De-En and XSUM | BLEU / ROUGE-L | Hurt |
| Add the rollback policy (ablation) | Aligned BiLD · IWSLT 2017 De-En and XSUM | BLEU / ROUGE-L | Helped |
| Remove confidence-triggered fallback and hand over periodically instead (ablation) | Aligned BiLD · IWSLT 2017 De-En and XSUM | BLEU / ROUGE-L | Hurt |
| Add confidence-triggered fallback (ablation) | Aligned BiLD · IWSLT 2017 De-En and XSUM | BLEU / ROUGE-L | Helped |
| Replace rejection sampling with BiLD's distance-based hard rollback and dynamic fallback | Speculative decoding · IWSLT | BLEU: 0.61 higher BLEU score | Helped |
| Replace rejection sampling with BiLD's distance-based hard rollback and dynamic fallback | Speculative decoding · XSUM | ROUGE-L: up to a 0.30 ROUGE-L score gain | Helped |
| Use BiLD rather than rejection sampling at matched BLEU/ROUGE-L | Speculative decoding · IWSLT and XSUM | speedup: 1.28$\times$ -> 1.49$\times$; 1.25$\times$ -> 1.50$\times$ | Helped |
| Use BiLD early exiting with rollback and actual key/value caches | CALM early exiting · IWSLT 2017 De-En and WMT 2014 De-En | BLEU: up to 2∼2.5 points better BLEU score | Helped |
Closest papers by meaning
- Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies (ICML 2025 · similarity 0.88)
- MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding (ICLR 2025 · similarity 0.88)
- Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment (ICLR 2025 · similarity 0.88)
- MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE (NeurIPS 2025 · similarity 0.88)
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.88)
- RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.