Research map / Architectures and efficiency / Mixture of experts
MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that speculative decoding can speed up sparse MoE LLMs at moderate batch sizes, proposes a target efficiency metric and a fitted analytical speedup model, validated on vLLM across several GPUs.
Problem
Speculative decoding is assumed to be ineffective for Mixture-of-Experts models because verifying multiple draft tokens activates more experts and increases memory access, and acceptance rate alone cannot explain system-level speedup degradation.
Approach
At moderate batch sizes where all experts are already activated, verifying multiple draft tokens adds little expert-loading cost, and sparser MoEs keep FFNs memory-bound longer, so SD gains more. The paper introduces target efficiency (T_T(B,1)/T_T(B,γ)) and a roofline-based parametric model of SD speedup fitted to measurements.
Key results
- Qwen2-57B-A14B-Instruct with Qwen2-0.5B draft reaches up to 2.29x speedup at medium batch sizes on 2xGPU-B (HumanEval, temperature 0, gamma=4).
- Target efficiency for MoE first increases then decreases with batch size, while the dense OPT model's decreases monotonically, explaining why SD favors MoE at moderate batches.
- The fitted analytical model with 21 GPU measurements tracks measured speedups across sparsity levels K and draft lengths gamma, though very sparse K=1,2 variants show monotonic decrease contrary to the theory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use speculative decoding at moderate batch sizes rather than autoregressive decoding | Qwen2-57B-A14B-Instruct · HumanEval, temperature 0.0, 2xGPU-B | speedup: 2.29x | Helped |
| Increase draft length from γ=2 to γ=4 | MoESD on Qwen2 · HumanEval, temperature 0.0, 2xGPU-B | speedup: 1.71 -> 2.29 | Helped |
| Increase draft length from γ=2 to γ=4 | MoESD on Qwen2 · MT-bench, temperature 1.0, 2xGPU-B | speedup: 1.27 -> 1.18 | Hurt |
| Increase MoE sparsity by reducing activated experts per token K (ablation) | MoESD on Qwen2-57B-A14B-Instruct · Varied batch sizes | speedup | Mixed |
| Reduce activated experts per token to K=1 or 2 (ablation) | MoESD on Qwen2-57B-A14B-Instruct · Varied batch sizes | speedup | Hurt |
| Increase batch size from small to moderate | MoESD on MoE models · Varied batch sizes | speedup | Helped |
| Increase batch size beyond the moderate-size peak | MoESD on MoE models · Varied batch sizes | speedup | Hurt |
| Use a sparser MoE target model | Speculative decoding on MoE · Broader batch-size range | speedup | Helped |
Surprising findings
- Under moderate batch sizes, SD benefits MoE more than dense models, contrary to the conventional belief that SD is ineffective for MoEs.
- Very sparse MoEs (K=1,2) show continuously decreasing SD speedup, which the authors attribute to attention dominating the artificially modified model.
Limitations
- Assumes KV-cache is small relative to parameters; long-sequence regimes are not covered and are deferred to MagicDec.
- Assumes uniform expert activation, which may not hold under load imbalance.
- Sparsity is varied by changing num_experts_per_token without retraining, requiring an adjustment factor for acceptance rate changes.
Experimental setup
Datasets: HumanEval MT-bench
Benchmarks: HumanEval MT-bench
Metrics: end-to-end speedup target efficiency acceptance rate MSE of speedup model fit
Baselines: autoregressive decoding OPT-30B dense target model with OPT-350m draft
Builds on: speculative decoding roofline model MagicDec EAGLE
Closest papers by meaning
- SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding (ICML 2026 · similarity 0.93)
- Mixture of Attentions For Speculative Decoding (ICLR 2025 · similarity 0.92)
- A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.92)
- MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding (ICLR 2025 · similarity 0.92)
- Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios (AAAI 2026 · similarity 0.91)
- Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.