Research map / Architectures and efficiency / Mixture of experts
Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models
Ekagra Ranjan, Komal Teru, Bharat Venkitesh, Acyr Locatelli
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Evaluates speculative decoding across 10 sparse MoE models (30B-1T parameters, 8-512 experts) to test the predicted concave speedup-batch size relationship.
Problem
Prior work predicts that speculative decoding speedup on MoE models is concave in batch size, peaking at intermediate batch sizes due to expert saturation; the paper examines whether this holds in practice.
Approach
Empirically measures speculative decoding speedup across many sparse MoE models and finds that real gains exceed the concave prediction, attributing this to the actual behavior of expert saturation under speculative verification.
Key results
- Evaluates 10 sparse MoE models spanning 30B-1T parameters and 8-512 experts; the abstract indicates the measured speedup-batch size relationship departs from the predicted concave shape.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use fine-grained MoE target (small experts, low k/N) instead of coarse MoE | speculative decoding on MoE targets · TPOT speedup vs batch size, MT-Bench | speedup curve shape | Improved |
| increase tensor parallelism from 1 to 2 (bf16) (ablation) | Qwen3-30B EAGLE-3 · batch size 1, MT-Bench | target efficiency: 0.630 -> 0.697 | Improved |
| quantize weights from bf16 to fp8 (TP1) (ablation) | Qwen3-30B EAGLE-3 · batch size 1, MT-Bench | target efficiency: 0.630 -> 0.818 | Improved |
| combine fp8 and TP2 (ablation) | Qwen3-30B EAGLE-3 · batch size 1, MT-Bench | target efficiency: 0.630 -> 0.860 | Improved |
| use empirical routing (temporal overlap) instead of independence baseline for unique expert count (ablation) | SD verification of 4 tokens, Qwen3-30B · MT-Bench routing | unique experts per layer: 26.9 -> 19.5 | Improved |
| model per-layer skewed expert probabilities in IPF instead of uniform (ablation) | IPF unique-expert predictor · Qwen3-30B, MT-Bench | prediction error: +3.1% -> +1.2% | Improved |
| replace uniform random routing with IPF synthetic routing in dummy-weight runs (ablation) | proxy target efficiency · Mixtral, Qwen3-30B, Qwen3-235B | proxy target efficiency vs real-model curve | Improved |
| correct MoE runtime fraction from weight share (67%) to measured GPU time share (33%) (ablation) | activated-expert-bytes model · Qwen3-30B batch size 1 H100 | predicted target efficiency: 0.511 -> 0.679 | Improved |
| correct MoE scaling S from 2.44 to 1.57 (L2 reuse) (ablation) | activated-expert-bytes model · Qwen3-30B batch size 1 H100 | predicted target efficiency: 0.679 -> 0.843 | Improved |
| correct non-MoE scaling from 1.0 to 0.94 (ablation) | activated-expert-bytes model · Qwen3-30B batch size 1 H100 | predicted target efficiency: 0.843 -> 0.872 | Improved |
| move from H100 to B200 (default MoE kernel) | Qwen3-30B FP8 SD verification · batch size 1 | MoE L2 hit rate under SD: 27.79% -> 2.21% | Hurt |
| increase speculative window K from 3 to 5 and 7 (ablation) | Mixtral, Qwen3-30B, Qwen3-235B · MT-Bench end-to-end sweep | speedup | No clear effect |
Surprising findings
- Speculative decoding gains on sparse MoE models are larger than the prior concave prediction suggests, per the title and abstract.
Limitations
- Full text unavailable; details of the mechanism and experimental setup are not verified.
Experimental setup
Metrics: speedup
Builds on: speculative decoding
Closest papers by meaning
- MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenario (NeurIPS 2026 · similarity 0.92)
- Speculative Speculative Decoding (ICLR 2026 · similarity 0.92)
- MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE (NeurIPS 2025 · similarity 0.91)
- Performance-Driven Policy Optimization for Speculative Decoding with Adaptive Windowing (NeurIPS 2026 · similarity 0.90)
- Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarios (AAAI 2026 · similarity 0.89)
- A Theoretical Perspective for Speculative Decoding Algorithm (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
