AI Paper Atlas

Research map / Architectures and efficiency / Mixture of experts

Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models

Ekagra Ranjan, Komal Teru, Bharat Venkitesh, Acyr Locatelli

NeurIPS 2026 · Architectures and efficiency › Mixture of experts · approach: Efficient MoE language model systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Evaluates speculative decoding across 10 sparse MoE models (30B-1T parameters, 8-512 experts) to test the predicted concave speedup-batch size relationship.

Problem

Prior work predicts that speculative decoding speedup on MoE models is concave in batch size, peaking at intermediate batch sizes due to expert saturation; the paper examines whether this holds in practice.

Approach

Empirically measures speculative decoding speedup across many sparse MoE models and finds that real gains exceed the concave prediction, attributing this to the actual behavior of expert saturation under speculative verification.

Key results

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use fine-grained MoE target (small experts, low k/N) instead of coarse MoEspeculative decoding on MoE targets · TPOT speedup vs batch size, MT-Benchspeedup curve shapeImproved
increase tensor parallelism from 1 to 2 (bf16) (ablation)Qwen3-30B EAGLE-3 · batch size 1, MT-Benchtarget efficiency: 0.630 -> 0.697Improved
quantize weights from bf16 to fp8 (TP1) (ablation)Qwen3-30B EAGLE-3 · batch size 1, MT-Benchtarget efficiency: 0.630 -> 0.818Improved
combine fp8 and TP2 (ablation)Qwen3-30B EAGLE-3 · batch size 1, MT-Benchtarget efficiency: 0.630 -> 0.860Improved
use empirical routing (temporal overlap) instead of independence baseline for unique expert count (ablation)SD verification of 4 tokens, Qwen3-30B · MT-Bench routingunique experts per layer: 26.9 -> 19.5Improved
model per-layer skewed expert probabilities in IPF instead of uniform (ablation)IPF unique-expert predictor · Qwen3-30B, MT-Benchprediction error: +3.1% -> +1.2%Improved
replace uniform random routing with IPF synthetic routing in dummy-weight runs (ablation)proxy target efficiency · Mixtral, Qwen3-30B, Qwen3-235Bproxy target efficiency vs real-model curveImproved
correct MoE runtime fraction from weight share (67%) to measured GPU time share (33%) (ablation)activated-expert-bytes model · Qwen3-30B batch size 1 H100predicted target efficiency: 0.511 -> 0.679Improved
correct MoE scaling S from 2.44 to 1.57 (L2 reuse) (ablation)activated-expert-bytes model · Qwen3-30B batch size 1 H100predicted target efficiency: 0.679 -> 0.843Improved
correct non-MoE scaling from 1.0 to 0.94 (ablation)activated-expert-bytes model · Qwen3-30B batch size 1 H100predicted target efficiency: 0.843 -> 0.872Improved
move from H100 to B200 (default MoE kernel)Qwen3-30B FP8 SD verification · batch size 1MoE L2 hit rate under SD: 27.79% -> 2.21%Hurt
increase speculative window K from 3 to 5 and 7 (ablation)Mixtral, Qwen3-30B, Qwen3-235B · MT-Bench end-to-end sweepspeedupNo clear effect

Surprising findings

Limitations

Experimental setup

Metrics: speedup

Builds on: speculative decoding

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy