Research map / Large language models / Reasoning and chain of thought
Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Collaborative Speculative Decoding (CoSD), which fuses two LLMs at test time by using speculative-decoding-style draft/assistant verification with a probability-based rule or a small decision tree to decide token replacement.
Problem
Fusing complementary knowledge of multiple LLMs at inference time efficiently, without retraining, when different models excel in different domains.
Approach
Uses speculative decoding's draft-and-verify loop but replaces the acceptance test with a decision that swaps a draft token for the assistant's token when the draft is low-confidence and the assistant's probability is much higher (rule), or via a decision tree trained on token-probability pairs from a small domain sample.
Key results
- On Pair 1 (complementary knowledge), CoSD-Rule achieves 52.41 average accuracy vs. 50.79 for the draft model and 49.62 for Average Decoding.
- On Pair 3 (catastrophic forgetting recovery), CoSD-Rule averages 52.16 vs. 48.29 for Co-LLM and 45.94 for Average Decoding.
- Token latency of CoSD-Rule (132.31 ms) is close to Speculative Decoding (131.22 ms), with acceptance rate 0.81 vs. 0.89.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CoSD-Rule instead of the strongest baseline | Pair 1 · Pair 1 AVG. | average score: 50.90 -> 52.41 | Helped |
| Use CoSD-Rule instead of the strongest baseline | Pair 2 · Pair 2 AVG. | average score: 55.77 -> 56.77 | Helped |
| Use CoSD-Rule instead of the strongest baseline | Pair 3 · Pair 3 AVG. | average score: 48.29 -> 52.16 | Helped |
| Use CoSD-Rule rather than the assistant model | Pair 4 · Pair 4 AVG. | average score: 37.10 -> 35.71 | Hurt |
| Use CoSD-Rule rather than Char-ED | Pair 5 · Pair 5 GSM8K | accuracy: 37.54 -> 44.88 | Helped |
| Use CoSD-Tree rather than Char-ED | Pair 5 · Pair 5 GSM8K | accuracy: 37.54 -> 37.01 | Hurt |
| Train the decision tree on GSM8K rather than MMLU (ablation) | Pair 3 CoSD-Tree · GSM8K | accuracy: 35.04 -> 37.17 | Helped |
| Train the decision tree on MMLU rather than GSM8K (ablation) | Pair 3 CoSD-Tree · MMLU | accuracy: 60.88 -> 63.94 | Helped |
| Train the decision tree on MMLU rather than AlpacaEval (ablation) | Pair 3 CoSD-Tree · HumanEval | accuracy: 23.17 -> 25.62 | Helped |
| Use CoSD-Rule rather than Speculative Decoding | Pair 3 · Pair 3 | token latency (ms): 131.22 -> 132.31 | Hurt |
Surprising findings
- Speculative Decoding is not better than CoSD in the disparate-capacity setting, despite being designed for that case.
- Assistant model's higher-confidence wrong answers can override a correct draft, showing confidence is not a reliable correctness signal.
Limitations
- No advantage when one model is significantly better than the other; user must know which model is better.
- Confidence-based replacement does not guarantee the assistant token is better; overconfident wrong assistant can override correct draft.
- Evaluation limited to small set of model pairs and benchmarks, with some results on fewer benchmarks.
Experimental setup
Datasets: AlpacaEval MMLU GSM8K HumanEval
Benchmarks: MMLU GSM8K HumanEval Hellaswag TruthfulQA
Metrics: accuracy average accuracy token latency acceptance rate
Baselines: Speculative Decoding Average Decoding Co-LLM CharED
Builds on: speculative decoding Co-LLM
Closest papers by meaning
- Fast Large Language Model Collaborative Decoding via Speculation (ICML 2025 · similarity 0.91)
- Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies (ICML 2025 · similarity 0.91)
- Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling (ICLR 2026 · similarity 0.90)
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.90)
- Reward-Guided Speculative Decoding for Efficient LLM Reasoning (ICML 2025 · similarity 0.90)
- Mixture of Attentions For Speculative Decoding (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.