Research map / Large language models / Reasoning and chain of thought
polybasic Speculative Decoding Through a Theoretical Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a polybasic speculative decoding framework with multiple draft models chained to one target model, backed by a theorem on optimal inference time and model-insertion efficiency, and reports 3-4.4x speedups on Vicuna, LLaMA2-Chat, LLaMA3 and Qwen2 with output distribution preserved.
Problem
Speculative decoding is dominated by dualistic draft-verify pipelines that lack theoretical grounding for how many models to chain and how acceptance lengths and costs interact, limiting LLM inference speed.
Approach
Generalizes draft-verify to a chain of n models where each intermediate model verifies the drafts of smaller models before the target, derives an optimal-time expression and a model-insertion criterion, and argues speculative sampling stabilizes acceptance length.
Key results
- Overall speedups on SpecBench of 3.48x (Vicuna-7B), 3.66x (LLaMA2-Chat-7B), 3.31x (LLaMA3-8B-Instruct) and 3.28x (Qwen2-7B-Instruct), versus EAGLE2 at 2.61x, 2.70x, 2.44x and 1.94x respectively.
- Vicuna-7B reaches up to 4.43x speedup on math tasks with average acceptance length around 10 tokens.
- Inserting a quantized Vicuna-7B (W4A16) between Vicuna-7B and EAGLE2 raises speedup from 2.61x to 3.48x, while inserting Vicuna-1B lowers it to 1.08x.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| insert a lightweight Vicuna-1B model between Vicuna-7B and EAGLE2 (non-compliant insertion) (ablation) | Vicuna-7B polybasic system · Vicuna-7B model insertion case study | speedup c: 2.61x -> 1.08x | Hurt |
| insert quantized Vicuna-7B (W4A16) between Vicuna-7B and EAGLE2 (compliant insertion) (ablation) | Vicuna-7B polybasic system · Vicuna-7B model insertion case study | speedup c: 2.61x -> 3.48x | Helped |
| insert FLAN-T5-base between FLAN-T5-XXL and FLAN-T5-small in CS Drafting (ablation) | CS Drafting · FLAN-T5 cascade | speedup c: 3.19x -> 3.88x | Helped |
| use speculative sampling instead of greedy verification in the three-model setup (ablation) | polybasic three-model system · 50 queries, acceptance-length variance | variance of acceptance length | Helped |
| polybasic multi-model system vs EAGLE2 dualistic drafting | Vicuna-7B, LLaMA2-Chat-7B, LLaMA3-8B, Qwen2-7B on SpecBench · SpecBench overall | speedup c: e.g., Qwen2-7B 1.94x -> 3.28x; Vicuna-7B 2.61x -> 3.48x | Helped |
| polybasic system vs EAGLE baseline on larger models | Vicuna-13B and LLaMA-2-chat-70B · speedup on larger models | speedup c: Vicuna-13B 2.30x -> 2.69x; LLaMA-70B 2.46x -> 2.92x | Helped |
| four-or-more-model systems | polybasic framework · off-the-shelf models | speedup c | No clear effect |
| summarization task with polybasic framework | polybasic system · CNN/Daily Mail summarization | speedup c: speedup 2.95x to 3.41x, lower than other tasks | Mixed |
Surprising findings
- Inserting an extra small Vicuna-1B draft model between target and EAGLE2 drops speedup from 2.61x to 1.08x, showing that naive added models can hurt.
Limitations
- Four-or-more-model systems were not realized because suitable off-the-shelf models satisfying the insertion condition were hard to find.
- Speedup is lower on summarization and RAG tasks with long contexts due to KV cache overhead across multiple models.
- The stability theorem's derivation is partly omitted ('after careful algebra') and the three-model proof has notational inconsistencies.
Experimental setup
Datasets: ShareGPT MT-bench WMT14 DE-EN CNN/Daily Mail Natural Questions GSM8K DPR
Benchmarks: SpecBench MT-bench WMT14 DE-EN CNN/Daily Mail Natural Questions GSM8K DPR
Metrics: walltime speedup ratio average acceptance length
Baselines: EAGLE2 Speculative Sampling Cascade Speculative Drafting greedy sampling
Builds on: speculative sampling EAGLE2 Cascade Speculative Drafting TRIFORCE SpecBench
Closest papers by meaning
- A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.93)
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.92)
- Online Speculative Decoding (ICML 2024 · similarity 0.91)
- SpecBranch: Speculative Decoding via Hybrid Drafting and Rollback-Aware Branch Parallelism (ICLR 2026 · similarity 0.91)
- MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE (NeurIPS 2025 · similarity 0.90)
- Fast Large Language Model Collaborative Decoding via Speculation (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.