Research map / Architectures and efficiency / Mixture of experts
Dirichlet-Prior Shaping: Guiding Expert Specialization in Upcycled MoEs
Leyla Mirvakhabova, Babak Ehteshami Bejnordi, Gaurav Kumar, Hanxue Liang, Wanru Zhao, Paul N. Whatmough
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add DPSL router regularization to sparse upcycling | Qwen2-1.5B upcycled MoE · VLM benchmarks avg (2in4) | average score: 36.17 -> 38.17 | Improved |
| add DPSL router regularization to sparse upcycling | Qwen2-1.5B upcycled MoE · VLM benchmarks avg (8in16) | average score: 36.72 -> 37.25 | Improved |
| add DPSL router regularization to sparse upcycling | Llama3.2-1B upcycled MoE · VLM benchmarks avg (2in4) | average score: 34.85 -> 35.92 | Improved |
| add DPSL router regularization to sparse upcycling | Llama3.2-1B upcycled MoE · VLM benchmarks avg (8in16) | average score: 34.92 -> 35.87 | Improved |
| add DPSL router regularization to sparse upcycling | Phi3-mini 3.8B upcycled MoE · VLM benchmarks avg (2in4) | average score: 38.98 -> 40.18 | Improved |
| use modality-aware Dirichlet priors instead of manual modality routing | Llama3.2-1B 2in4 · VLM benchmarks avg | average score: 34.65 -> 36.18 | Improved |
| use task-specific Dirichlet priors instead of BTX | Llama3.2-1B 2in4 · VLM benchmarks avg | average score: 35.31 -> 35.35 | No clear effect |
| use modality-specific prior instead of symmetric prior (ablation) | DPSL on Llama3.2-1B 2in4 · VLM benchmarks avg | average score: 35.92 -> 36.18 | Improved |
| lower symmetric alpha from 1.0 to 0.75 (ablation) | DPSL on Llama3.2-1B 2in4 · VLM benchmarks avg | average score: 34.86 -> 35.92 | Improved |
| lower symmetric alpha from 1.0 to 0.75 (ablation) | DPSL on Qwen2-1.5B 8in16 · VLM benchmarks avg | average score: 37.25 -> 37.03 | No clear effect |
| raise lambda from 0.001 to 0.01 (ablation) | DPSL on Llama3.2-1B 2in4 · VLM benchmarks avg | average score: 35.2 -> 35.9 | Improved |
| raise lambda from 0.01 to 0.1 (ablation) | DPSL on Llama3.2-1B 2in4 · VLM benchmarks avg | average score: 35.9 -> 34.8 | Hurt |
Closest papers by meaning
- Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization (NeurIPS 2024 · similarity 0.88)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.88)
- Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations (NeurIPS 2025 · similarity 0.87)
- Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Model (ICLR 2025 · similarity 0.87)
- A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts (ICML 2024 · similarity 0.87)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
