Research map / Architectures and efficiency / Mixture of experts
OLMoE: Open Mixture-of-Experts Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OLMoE-1B-7B, a fully open sparse MoE LM (1.3B active / 6.9B total) pretrained on 5T tokens, plus an instruct variant, releasing weights, data, code, and logs, with controlled experiments on MoE design choices and routing analysis.
Problem
Open language models with sparse Mixture-of-Experts architectures lack full openness (data, code, training recipes), and MoE design choices (granularity, routing, shared experts, upcycling, auxiliary losses) are under-documented, hindering cost-efficient open MoE development.
Approach
A fully open MoE LM with fine-grained experts (64 small experts, 8 active per token), dropless token-choice routing, load balancing and router z-loss, trained from scratch on 5T tokens, with controlled ablations over MoE and dense design choices and analysis of router saturation, co-activation, and domain/vocabulary specialization.
Key results
- OLMoE-1B-7B reaches 54.1 MMLU and 1.3B active parameters, the best among models with ~1B active parameters in the paper's comparison.
- OLMoE-1B-7B-Instruct (DPO) achieves 57.7 average across the adaptation suite and 84.0 AlpacaEval 1.0, outperforming Qwen1.5-MoE-A2.7B-Chat's 57.3 average.
- MoE trained with the same config reaches dense performance with ~3x fewer tokens, giving ~2x faster wall-clock training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-grained routing: use 32 experts with 4 activated instead of 8 experts with 1 activated (ablation) | OLMoE pretraining · HellaSwag and MMLU Var | downstream performance: around 10% | Helped |
| increase from 32 experts with 4 active to 64 experts with 8 active (ablation) | OLMoE pretraining · downstream tasks | downstream performance: 1--2% | Helped |
| add a shared expert rather than using routed experts only (ablation) | OLMoE MoE · MoE pretraining | performance | Hurt |
| use token-choice routing instead of expert-choice routing (ablation) | OLMoE MoE · pretraining tasks | downstream performance | Helped |
| use expert-choice routing instead of token-choice routing (ablation) | OLMoE MoE · routing throughput | tokens per second per device: 29,400 tokens per second per device versus 24,400 | Helped |
| train the MoE from scratch rather than sparsely upcycling OLMo-1B (ablation) | OLMoE MoE · pretraining after 500B tokens | performance: 500B tokens | Helped |
| add noise to upcycled weights (ablation) | upcycled OLMo-1B MoE · pretraining | performance | No clear effect |
| add load balancing loss during pretraining (ablation) | OLMoE MoE · pretraining | training and validation loss | Helped |
| add router z-loss with weight 0.001 (ablation) | OLMoE MoE · pretraining | downstream performance: ∼2% throughput reduction | Helped |
| use a truncated normal initialization instead of regular normal initialization (ablation) | OLMoE pretraining · pretraining | training stability and performance: around 450 billion tokens | Helped |
| replace non-parametric layer normalization with RMSNorm (ablation) | OLMoE pretraining · pretraining | performance: 15% throughput reduction | Helped |
| weight decay embedding parameters (ablation) | OLMoE pretraining · pretraining | performance | Helped |
Surprising findings
- Sparse upcycling only gave an advantage over training from scratch for about 25% of the dense model's compute budget, far less than the 120% reported in prior work.
- Shared experts slightly hurt performance by reducing expert combinations, and removing the load balancing loss during adaptation improved results.
Limitations
- Some design ablations may not transfer across configurations due to the large number of hyperparameters, as the authors acknowledge.
- Pretraining is for 5T tokens, far beyond compute-optimal, so granularity predictions from prior scaling laws may not apply.
- Adaptation uses mixes built with an improved pipeline, so comparisons to other chat models are not fully controlled.
Experimental setup
Datasets: DCLM-Baseline StarCoder peS2o arXiv OpenWebMath Algebraic Stack Wikipedia Tulu 2 SFT Mix No Robots CodeFeedback-Filtered-Instruction MetaMathQA Daring Anteater
Benchmarks: MMLU HellaSwag ARC-Challenge ARC-Easy PIQA WinoGrande GSM8k HumanEval BBH AlpacaEval 1.0 IFEval XSTest
Metrics: accuracy average score
Baselines: Llama2-7B Llama2-13B-Chat OLMo-7B (0724) DeepSeekMoE-16B Qwen1.5-MoE-A2.7B JetMoE-2B-9B Gemma2-9B Llama3.1-8B DCLM-7B
Builds on: OLMo DCLM Dolma 1.7 DPO Mixtral
Closest papers by meaning
- OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models (ICML 2024 · similarity 0.94)
- FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.89)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.89)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.89)
- dMoE: dLLMs with Learnable Block Experts (NeurIPS 2026 · similarity 0.88)
- LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.