Research map / Architectures and efficiency / Mixture of experts
Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Drop-Upcycling, which builds an MoE from a pre-trained dense model by partially re-initializing expert FFN weights (ratio r=0.5) with statistics-matched normal noise, improving long-horizon training over naive upcycling and training from scratch.
Problem
Upcycling a dense model into an MoE gives fast initial gains but slow later convergence because copied experts are identical and fail to specialize, so MoE trained from upcycled weights is suboptimal in the long term.
Approach
Copy the dense FFN into all experts, then randomly select a proportion r of intermediate indices shared across the gate/up/down matrices and re-initialize those columns/rows from a normal distribution matched to the statistics of the selected original weights; this retains dense knowledge while diversifying experts.
Key results
- Drop-Upcycling (r=0.5) on 8x1.5B reaches average 40.3 vs 37.4 for naive upcycling and 34.5 for training from scratch after 500B tokens.
- The 8x3.7B MoE (5.9B active) with Drop-Upcycling reaches average 44.4, versus 44.5 for the dense 13B model, using about 1/4 of training FLOPs (2.70 vs 7.43 x10^22 for dense 3.7B; 1.98 x10^22 for MoE).
- Drop-Upcycling outperforms training from scratch on 8x3.7B (44.4 vs 40.1 average).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| partially reinitialize expert FFN weights at r=0.5 instead of copying identical FFNs | MoE initialized from a dense model · 12-task average, 8×1.5B | average score: 37.4 -> 40.3 | Helped |
| partially reinitialize expert FFN weights at r=0.5 instead of copying identical FFNs | MoE initialized from a dense model · 12-task average, 8×152M | average score: 17.9 -> 19.7 | Helped |
| partially reinitialize expert FFN weights at r=0.5 rather than train all parameters from scratch | 8×3.7B MoE · 12-task average, 500B tokens | average score: 40.1 -> 44.4 | Helped |
| increase re-initialization ratio from 0.5 to 1.0 (ablation) | Drop-Upcycling · 12-task average, 8×152M | average score: 19.7 -> 18.9 | Hurt |
| increase re-initialization ratio from 0.5 to 1.0 (ablation) | Drop-Upcycling · 12-task average, 8×1.5B | average score: 40.3 -> 38.9 | Hurt |
| increase re-initialization ratio from 0.5 to 0.75 (ablation) | Drop-Upcycling · 8×152M and 8×1.5B, training up to 150B tokens | average downstream task score | Hurt |
| increase re-initialization ratio from 0.0 to 0.5 (ablation) | Drop-Upcycling · 8×152M and 8×1.5B, training up to 150B tokens | average downstream task score | Helped |
| use global rather than layer-wise load-balancing loss (ablation) | MoE construction methods · 8×1.5B, 40B tokens | training loss and downstream task performance | No clear effect |
| use uniform gate initialization U(-0.0346, 0.0346) rather than N(0, 0.02) (ablation) | Naive Upcycling · 8×1.5B, 50B tokens | average score: 32.1 -> 32.3 | Helped |
| use gate initialization N(0, 0.2887) rather than N(0, 0.02) (ablation) | Naive Upcycling · 8×1.5B, 50B tokens | average score: 32.1 -> 31.3 | Hurt |
Surprising findings
- Setting r=1.0 (random re-initialization of all FFNs while reusing embeddings and attention) still outperforms naive upcycling in some settings.
- Naive upcycling with global load balancing collapses onto only two experts across all layers, while Drop-Upcycling shows domain-specialized routing.
Limitations
- Branch-Train-MiX comparison uses extra 300B tokens, and the convergence analysis lacks constant-LR control, so long-term advantages may partly reflect LR scheduling.
- Largest experiments (8x3.7B) were limited to training from scratch and Drop-Upcycling only due to compute constraints.
- Evaluation focuses on Japanese and English benchmarks with a single model family.
Experimental setup
Datasets: LLM-jp corpus v3 Dolma 1.6 The Stack Japanese Common Crawl NDL WARP
Benchmarks: JEMHQA NIILC JSQuAD XL-Sum WMT20 En-Ja WMT20 Ja-En OpenBookQA TriviaQA HellaSwag SQuAD2 XWinograd BBH
Metrics: training loss character F1 ROUGE-2 BLEU accuracy CoT accuracy average score across 12 tasks
Baselines: training from scratch naive upcycling Random Noise Upcycling Branch-Train-MiX dense 13B model dense 3.7B model
Builds on: Upcycling Mixtral Llama Branch-Train-MiX DeepSeekMoE
Closest papers by meaning
- Scaling Laws for Upcycling Mixture-of-Experts Language Models (ICML 2025 · similarity 0.91)
- DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Models (CVPR 2025 · similarity 0.90)
- BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts (NeurIPS 2024 · similarity 0.89)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.89)
- Unchosen Experts Can Contribute Too: Unleashing MoE Models’ Power by Self-Contrast (NeurIPS 2024 · similarity 0.88)
- Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Model (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.