Research map / Large language models / Reasoning and chain of thought
Fast Large Language Model Collaborative Decoding via Speculation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoS, which accelerates LLM collaborative decoding (weighted ensemble and contrastive decoding) by extending speculative decoding to verify against the combined distribution and alternating proposer/verifier roles, achieving 1.11x–2.23x speedups without quality loss.
Problem
Collaborative decoding that combines multiple LLMs at every step requires a forward pass per model per token, making it much slower than single-model decoding.
Approach
Generalizes speculative decoding so verification and resampling use the combined distribution of the proposal and target models (Naive-CoS), and alternates the proposer and verifier roles so the bonus token from the target is reused as a proposal, further reducing model invocations.
Key results
- CoS achieves 1.11x–2.23x speedup over standard collaborative decoding across weighted ensemble and contrastive decoding settings.
- In the weighted ensemble two-model setting CoS reaches 1.34x–1.85x speedup; Llama-3 contrastive decoding on HumanEval (T=0) reaches 2.23x.
- Vanilla SD can slow collaborative decoding (e.g., 0.94x for Llama-2 pair with T=1 on HumanEval), whereas CoS stays above 1.0x in all reported settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the alternate proposal framework to weighted-ensemble CoS (ablation) | WE-CoS · HumanEval, Llama-Vicuna | speedup: 1.46x -> 1.58x | Helped |
| Add the alternate proposal framework to weighted-ensemble CoS (ablation) | WE-CoS · GSM8K, Llama-Vicuna | speedup: 1.39x -> 1.52x | Helped |
| Add the alternate proposal framework to weighted-ensemble CoS (ablation) | WE-CoS · HumanEval, Qwen-3b | speedup: 1.40x -> 1.62x | Helped |
| Add the alternate proposal framework to weighted-ensemble CoS (ablation) | WE-CoS · GSM8K, Qwen-3b | speedup: 1.32x -> 1.52x | Helped |
| Add the alternate proposal framework to contrastive-decoding CoS (ablation) | CD-CoS · HumanEval, T=0, Llama-3 | speedup: 1.98x -> 2.23x | Helped |
| Add the alternate proposal framework to contrastive-decoding CoS (ablation) | CD-CoS · GSM8K, T=0, Llama-3 | speedup: 1.72x -> 2.00x | Helped |
| Add the alternate proposal framework to contrastive-decoding CoS (ablation) | CD-CoS · HumanEval, T=1, Llama-3 | speedup: 1.41x -> 1.65x | Helped |
| Add the alternate proposal framework to contrastive-decoding CoS (ablation) | CD-CoS · GSM8K, T=1, Llama-3 | speedup: 1.24x -> 1.44x | Helped |
| Add the alternate proposal framework to contrastive-decoding CoS (ablation) | CD-CoS · HumanEval, T=0, Llama-2 | speedup: 1.18x -> 1.26x | Helped |
| Add the alternate proposal framework to contrastive-decoding CoS (ablation) | CD-CoS · GSM8K, T=0, Llama-2 | speedup: 1.59x -> 1.65x | Helped |
| Add the alternate proposal framework to contrastive-decoding CoS (ablation) | CD-CoS · HumanEval, T=1, Llama-2 | speedup: 0.89x -> 1.15x | Helped |
| Add the alternate proposal framework to contrastive-decoding CoS (ablation) | CD-CoS · GSM8K, T=1, Llama-2 | speedup: 1.21x -> 1.20x | Hurt |
Surprising findings
- Vanilla speculative decoding can be slower than standard collaborative decoding (speedups as low as 0.92x–0.96x in some settings), while CoS does not slow down.
Limitations
- Speedup depends on acceptance rate, which varies with task determinism and model alignment; low-acceptance settings yield smaller gains.
- Speedups reported on consumer/research GPUs (RTX 3090, A6000) and an Ascend NPU; results at larger scale not reported.
- Proposal length gamma is a hyperparameter selected by reporting the optimal of gamma=5 and gamma=1.
Experimental setup
Datasets: HumanEval GSM8K MMLU CNNDM
Benchmarks: HumanEval GSM8K MMLU CNNDM
Metrics: speedup ratio tokens per second
Baselines: standard collaborative decoding (WE) CD parallel collaborative decoding (WE-P, CD-P) speculative decoding (WE-SD, CD-SD) SCD
Builds on: speculative decoding contrastive decoding speculative contrastive decoding
Closest papers by meaning
- Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding (ICML 2025 · similarity 0.91)
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.91)
- polybasic Speculative Decoding Through a Theoretical Perspective (ICML 2025 · similarity 0.90)
- A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.90)
- MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE (NeurIPS 2025 · similarity 0.90)
- Mixture of Attentions For Speculative Decoding (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.