Research map / Large language models / Reasoning and chain of thought

SpecEM: Training-Free LLM Ensembling via Iterative Drafting, Verification, and Online Feedback

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SpecEM, a training-free LLM ensemble that iteratively drafts segments from multiple LLMs, verifies them via in-line masked scoring, and reweights models online with multiplicative weight updates.

Problem

Existing LLM ensembles either incur first-token delay (generate-then-ensemble) or weak long-range cross-model collaboration (token-level ensemble-while-generation), and they assume equal model contributions regardless of task performance.

Approach

Speculative-decoding-style loop in which all models draft fixed-length segments in parallel, verify all candidates in one forward pass via a verify-in-line attention mask and reset position IDs, and update each model's voting weight multiplicatively by how often its candidates win peer comparisons.

Key results

• On English FuseEval with 7-9B base models, SpecEM reaches ROUGE-L 28.86 and ROUGE-2 14.40, above the best single model (Mistral-7B, ROUGE-L 25.57) and the best ensemble baseline (PairRank, 25.94).
• With four 24B-72B models, SpecEM achieves 51.32 LC win rate on AlpacaEval 2.0, versus 49.20 for UniTE and 48.46 for the best single model Mistral-24B.
• Removing online feedback reduces GPT-4o win-rate comparisons by an average 6.64 points; the paper reports first-token latency under 0.6s.

Measured effects reported in the paper

Surprising findings

• Adding a weaker model can reduce performance: on IFEval, SpecEM with Qwen2, GLM4, and Gemma2 underperforms the GLM4+Gemma2 ensemble.
• Selecting the top-win segment directly (without reward accumulation) hurts performance by about 5.8 points, so win counts work better as soft rewards.

Limitations

• Weaker models in the ensemble can degrade performance, sometimes below a smaller ensemble.
• Fixed-length segments can truncate semantic units.
• Evaluated on a limited set of models and benchmarks; the FuseEval test set is a self-constructed subset.

Experimental setup

Datasets: Dolly-15k Alpaca-GPT4 COIG-CQIA Human-Value COIG-CQIA Ruozhiba

Benchmarks: FuseEval IFEval AlpacaEval 2.0 MMLU ARC-C GSM8K

Metrics: ROUGE-1 ROUGE-2 ROUGE-L BLEU BARTScore BERTScore GPT4-Rank accuracy exact match IFEval prompt/instruction strict and loose

Baselines: PairRank MBR Generation Fusion MOA Majority Voting UniTE

Builds on: speculative decoding multiplicative weights update Mixture-of-Agents UniTE LLM-Blender PairRank

Closest papers by meaning

• Rethinking LLM Ensembling from the Perspective of Mixture Models (ICML 2026 · similarity 0.89)
• Multi-Branch Self-Drafting for LLM Inference Acceleration (AAAI 2025 · similarity 0.88)
• Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration (NeurIPS 2024 · similarity 0.88)
• Adaptive Draft-Verification for Efficient Large Language Model Decoding (AAAI 2025 · similarity 0.87)
• Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies (ICML 2025 · similarity 0.87)
• Efficiently Aligning Draft Models via Parameter- and Data-Efficient Adaptation (NeurIPS 2026 · similarity 0.87)