Research map / Large language models / Language, safety and interpretability
OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Releases OpenMoE, a family of fully open decoder-only MoE LLMs (0.65B to 34B), and analyzes their routing to find token-ID-driven, early-fixed routing and drop-towards-the-end token dropping.
Problem
Open-source MoE LLMs trained on trillion-token diverse data were scarce, and the routing mechanisms of MoE LLMs (how tokens are assigned to experts and what this implies for performance) were poorly understood.
Approach
Train and fully release decoder-only MoE models (ST-MoE-style top-2 routing with residual MoE, UL2 objective, code-heavy data), then analyze routing behavior to identify context-independent token-ID specialization, early routing fixation, and later-position token dropping under expert capacity.
Key results
- OpenMoE-8B/32E reaches 32.7 exact match on TriviaQA versus 29.7 for OpenLLaMA-3B and 11.2 for TinyLLaMA-1.1B, with 2.1B activated parameters.
- OpenMoE-8B/32E scores 4.69 on MT-Bench first turn, above OpenLLaMA-3B (4.36) but below it on second turn (3.26 vs 3.62), giving an average of 3.98 vs 3.99.
- Routing is largely determined by token ID: expert preferences overlap across intermediate checkpoints, indicating routing is fixed early in pre-training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MoE rather than no MoE (ablation) | OpenMoE-Base/16E · TriviaQA zero-shot | EM: 0.1 -> 1.4 | Helped |
| Use UL2 rather than PrefixLM only (ablation) | OpenMoE-Base/16E · TriviaQA zero-shot | EM: 0.0 -> 1.4 | Helped |
| Add code data to pre-training (ablation) | OpenMoE-Base/16E · TriviaQA zero-shot | EM: 0.7 -> 1.4 | Helped |
| Replace umT5 tokenizer with LLaMA tokenizer (ablation) | OpenMoE-Base/16E · TriviaQA zero-shot | EM: 1.4 -> 2.2 | Helped |
| Train OpenMoE-8B/32E instead of OpenLLaMA-3B | raw language models · TriviaQA | Exact Match: 29.7 -> 32.7 | Helped |
| Train OpenMoE-8B/32E instead of TinyLLaMA-1.1B | raw language models · HumanEval | Pass@1: 9.1 -> 9.8 | Helped |
| Train OpenMoE-8B/32E instead of TinyLLaMA-1.1B | raw language models · WMT16 En-Ro | BLEU: 2.6 -> 3.1 | Helped |
| Train OpenMoE-8B/32E instead of GPT-3 6B | raw language models · BigBench-Lite | Average: 5.40 -> 6.93 | Helped |
| Train OpenMoE-8B/32E instead of OpenLLaMA-3B | raw language models · lm-evaluation-harness | Average: 48.7 -> 46.1 | Hurt |
| Use OpenMoE-8B/32E instead of OpenLLaMA-3B | chat models · MT-Bench | Average score: 3.99 -> 3.98 | No clear effect |
| Use OpenMoE-8B/32E instead of OpenLLaMA-3B | chat models · MT-Bench | First-turn score: 4.36 -> 4.69 | Helped |
| Use OpenMoE-8B/32E instead of OpenLLaMA-3B | chat models · MT-Bench | Second-turn score: 3.62 -> 3.26 | Hurt |
Surprising findings
- Routing decisions are predominantly context-independent, with tokens sharing a token ID routed to the same few experts regardless of context.
- Fine-tuning on instruction data does not alleviate the drop-towards-the-end token dropping issue, since routing is fixed early in pre-training.
Limitations
- Ablations only at small scale (OpenMoE-Base/16E), so generalizability to larger models is uncertain.
- Models underperform on MMLU (about 26.2%) and lm-evaluation-harness relative to OpenLLaMA-3B, and 34B model trained on only 200B tokens.
- Used Adafactor and Inverse Square Root schedule, with less rigorous comparisons to standard setups.
Experimental setup
Datasets: RedPajama The Stack (dedup) WildChat TED-Parallel-Corpus MT-Bench TriviaQA HumanEval WMT16 En-Ro BigBench-Lite
Benchmarks: TriviaQA HumanEval WMT16 En-Ro BigBench-Lite lm-evaluation-harness (13 tasks) MT-Bench MMLU
Metrics: exact match pass@1 BLEU accuracy average score routing decision analysis
Baselines: TinyLLaMA-1.1B OpenLLaMA-3B GPT-3 BIG-G BIG-G-sparse GPT-J-6B Mixtral DeepSeek-MoE
Builds on: ST-MoE UL2 Switch Transformer GShard
Closest papers by meaning
- OLMoE: Open Mixture-of-Experts Language Models (ICLR 2025 · similarity 0.94)
- $\textit{Read-ME}$: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design (NeurIPS 2024 · similarity 0.90)
- Your Mixture-of-Experts LLM Is Secretly an Embedding Model for Free (ICLR 2025 · similarity 0.90)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.90)
- Oracle-MoE: Locality-preserving Routing in the Oracle Space for Memory-constrained Large Language Model Inference (ICML 2025 · similarity 0.90)
- Ada-K Routing: Boosting the Efficiency of MoE-based LLMs (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.