Research map / Architectures and efficiency / Mixture of experts
Dense Backpropagation Improves Training for Sparse Mixture-of-Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Default MoE, which gives the MoE router a dense straight-through-style gradient by filling non-activated experts' outputs with EMA-maintained default vectors, so sparse forward passes are kept while the router receives signals from all experts.
Problem
Sparse Top-K MoE routers receive gradients only from the experts a token activates, which slows learning and can cause training instability; dense gradients require passing tokens through all experts, which removes sparsity.
Approach
Replace the outputs of non-activated experts in the straight-through dense router gradient with an exponential moving average of previously observed expert outputs (default vectors), giving a dense router gradient at near-zero extra compute and memory.
Key results
- On 8c1 MoE after 160B tokens, Default MoE averages 47.9 vs 46.9 for TopK across 13 benchmarks (+2.1%), and on 32c4 49.5 vs 49.4.
- Default MoE reaches target perplexity about 9% faster than TopK on FineWeb-Edu (8c1, 1.96B params).
- Loss-Free Balancing reaches validation perplexity 24.22 vs 23.48 for Default MoE after 10B tokens; SparseMixer reaches 19.77 at 24B tokens, worse than both TopK and Default MoE; ReMoE failed to converge.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EMA default outputs to provide dense router gradients instead of standard TopK routing | 1.96B MoE 8c1 · Average across pretraining benchmarks | benchmark score: 46.9$_{0.4}$ -> 47.9$_{0.4}$ | Helped |
| Use EMA default outputs to provide dense router gradients instead of standard TopK routing | 1.96B MoE 32c4 · Average across pretraining benchmarks | benchmark score: 49.4$_{0.4}$ -> 49.5$_{0.4}$ | No clear effect |
| Use Default MoE rather than TopK routing | 1.96B MoE 8c1 · LogiQA | score: 26.0$_{1.7}$ -> 26.9$_{1.7}$ | Helped |
| Use Default MoE rather than TopK routing | 1.96B MoE 32c4 · LogiQA | score: 29.0$_{1.8}$ -> 28.3$_{1.8}$ | Hurt |
| Use Default MoE rather than TopK routing | 1.96B MoE 8c1 · BoolQ | score: 58.5$_{0.9}$ -> 62.0$_{0.8}$ | Helped |
| Use Default MoE rather than TopK routing | 1.96B MoE 32c4 · MMLU | score: 33.7$_{0.4}$ -> 32.5$_{0.4}$ | Hurt |
| Use Default MoE rather than TopK routing (ablation) | MoE configurations 8c1, 8c2, 32c1, 32c2, and 32c4 · FineWeb training | perplexity | Helped |
| Weight EMA default-vector updates by router logits (ablation) | Default MoE · FineWeb training, varying EMA decay rate | perplexity | No clear effect |
| Set EMA decay β to 0.9 rather than 0.999 (ablation) | 8c1 Default MoE · FineWeb training | perplexity | Helped |
| Initialize the EMA buffer with zeros rather than random Gaussian values (ablation) | Default MoE · FineWeb training | perplexity | Helped |
| Pass the default vector forward as well as using it in the backward pass (ablation) | Default MoE · FineWeb training | perplexity | Helped |
| Increase the learning rate from the TopK-best 7×10⁻⁴ to 9×10⁻⁴ (ablation) | Default MoE · FineWeb training | perplexity | Helped |
Surprising findings
- ReMoE failed to converge in the authors' setting despite over 1000 GPU hours of attempts.
- Much of the reported gain of prior routing methods appears to be absorbed by globally reduced auxiliary load-balancing loss, which the authors use in all baselines.
Limitations
- Only dropless MoEs were trained; token-dropping MoEs are not studied.
- Evaluated only on pretraining with standard LM benchmarks; no reasoning, coding, or math benchmarks like GSM8K/HumanEval, and no finetuning studied.
- Benchmark gains are small and mixed across benchmarks and configurations; a single run per configuration was used.
Experimental setup
Datasets: FineWeb-Edu FineWeb
Benchmarks: LogiQA MathQA MMLU OpenBookQA LAMBADA SocialIQA HellaSwag ARC Winogrande PubMedQA BoolQ PIQA
Metrics: accuracy validation perplexity throughput (tokens/s) cosine similarity to dense router gradient
Baselines: TopK MoE SparseMixer ReMoE Loss-Free Balancing
Builds on: Top-K routing straight-through estimator Megablocks SparseMixer ReMoE
Closest papers by meaning
- A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts (ICML 2024 · similarity 0.90)
- Scaling Beyond the GPU Memory Limit for Large Mixture-of-Experts Model Training (ICML 2024 · similarity 0.89)
- From Sparse to Soft Mixtures of Experts (ICLR 2024 · similarity 0.89)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.89)
- Tight Clusters Make Specialized Experts (ICLR 2025 · similarity 0.89)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.