Research map / Large language models / Reasoning and chain of thought
Dynamic-Width Speculative Beam Decoding for LLM Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dynamic-Width Speculative Beam Decoding (DSBD), which combines speculative decoding with beam sampling using a draft forest, adaptive per-layer beam width, and forest-based parallel verification, reporting 1.5-1.9x speedup over beam sampling with lower energy.
Problem
LLM autoregressive inference is slow and energy-costly; speculative decoding preserves multinomial sampling distribution but yields single sequences of possibly suboptimal quality, while beam sampling gives better quality but is expensive in time, energy, and memory.
Approach
Verify draft beam trajectories from a small model layer by layer as a forest of trees so that the output follows the large model's beam sampling distribution, adapt the number of accepted beams per layer via an acceptance-count probability threshold, verify multiple trees in parallel with topology-aware masks, and keep only one KV cache by retaining the best beam for memory-constrained settings.
Key results
- Achieves 1.5-1.9x speed-up and 1.8-2.5x lower energy than beam sampling at fixed effectiveness on SQuAD and Spider.
- On MT-Bench with Llama-3-8B, DSBD scores 4.11 vs 4.10 for beam (W=5) at 17.8 vs 10.5 token/s.
- Under single-KV-cache memory constraint on SQuAD with Llama-2, DSBD reaches EM 86 vs 75 for speculative decoding and 74 for multinomial sampling at similar speed.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DSBD instead of beam sampling (W=5) | Llama-2-13B · MT-Bench | Token/s: 12.1 -> 16.5 | Helped |
| Use DSBD instead of beam sampling (W=5) | Llama-2-13B · MT-Bench | J/token: 26.3 -> 16.1 | Helped |
| Use DSBD instead of beam sampling (W=5) | Llama-3-8B · MT-Bench | Token/s: 10.5 -> 17.8 | Helped |
| Use DSBD instead of beam sampling (W=5) | Llama-3-8B · MT-Bench | J/token: 33.3 -> 22.9 | Helped |
| Use DSBD instead of beam sampling (W=5) | Llama-2-13B · MT-Bench | Score: 3.51 -> 3.52 | No clear effect |
| Use one input beam and one KV-cache sequence | DSBD · SQuAD | EM/EA: 74 -> 86 | Helped |
| Use one input beam and one KV-cache sequence | DSBD · Spider | EM/EA: 20 -> 31 | Helped |
| Use one input beam and one KV-cache sequence | DSBD · SQuAD | tokens/s: 27.11 -> 26.67 | No clear effect |
| Use one input beam and one KV-cache sequence | DSBD · SQuAD | J/token: 12.36 -> 8.75 | Helped |
| Use a higher dynamic-width threshold (t=0.9 rather than t=0.7) (ablation) | DSBD · Llama-2, Spider | speed | Helped |
| Use a higher dynamic-width threshold (t=0.9 rather than t=0.7) (ablation) | DSBD · Llama-2, Spider | EA score | Hurt |
| Increase the draft beam width W_S (ablation) | DSBD · Llama-2, Spider | EA score | Helped |
Surprising findings
- Under a single KV cache constraint, the method's exact match (86) far exceeds speculative decoding (75) despite similar throughput.
Limitations
- Gains depend on acceptance rate between small and large model; memory-constrained variant keeps only one KV cache.
- Evaluation limited to three tasks and to OPT, Llama-2, Llama-3 families on a single GPU; large-model evaluation with GPT-4 judge is limited to MT-Bench single table.
- Omits OPT results on Spider because execution accuracy is near zero.
Experimental setup
Datasets: SQuAD Spider MT-Bench
Benchmarks: SQuAD Spider MT-Bench
Metrics: exact match execution accuracy MT-Bench score tokens/s joules per token
Baselines: beam sampling speculative decoding SpecInfer multinomial sampling
Builds on: speculative decoding SpecInfer beam sampling
Closest papers by meaning
- Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios (AAAI 2026 · similarity 0.92)
- Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decoding (NeurIPS 2025 · similarity 0.91)
- A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.91)
- Break the Sequential Dependency of LLM Inference Using Lookahead Decoding (ICML 2024 · similarity 0.91)
- Optimized Multi-Token Joint Decoding With Auxiliary Model for LLM Inference (ICLR 2025 · similarity 0.91)
- AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelism (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.