Research map / Architectures and efficiency / Mixture of experts
Mixture of Parrots: Experts improve memorization more than reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically and empirically that increasing MoE expert count (fixed active parameters) improves memorization (phone-book, world knowledge) but saturates on reasoning (shortest path, commonsense, math), with dense width needed for reasoning.
Problem
Whether Mixture-of-Experts offer free performance gains over dense transformers, and how those tradeoffs depend on task type (memorization vs reasoning) at fixed active or total parameters.
Approach
Combines communication-complexity lower bounds (length-2 path requires width ~|V|/log N for MoEs regardless of expert count) with a memorization upper bound showing MoEs need ~sqrt(nm) active parameters versus Omega(n) for dense models, then validates on synthetic and pre-trained models that experts help memorization but not reasoning.
Key results
- On phone-book memorization, maximal phone-book size correlates with total parameters; an MoE with 42M active parameters outperforms a dense model with 10x as many parameters.
- On shortest-path reasoning (50-node graphs), increasing experts does not consistently improve accuracy; active parameters predict performance better.
- Pre-trained MoEs match dense models at equal total parameters on world-knowledge tasks but underperform dense models on commonsense and math benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase number of experts while fixing active parameters | MoE transformers on phone-book memorization · Phone-book memorization | maximal phone-book size with >90% accuracy | Helped |
| increase number of experts at fixed width | MoE transformers on shortest path task (50-node graphs) · Shortest path (50-node graphs) | test accuracy | No clear effect |
| increase number of experts at fixed active parameters | pre-trained MoEs on world-knowledge tasks · World-knowledge benchmarks (TriviaQA, NQ, etc.) | average F1 accuracy | Helped |
| increase number of experts at fixed active parameters | pre-trained MoEs on commonsense and math reasoning · Commonsense and math benchmarks | average accuracy | Hurt |
| compare MoEs to dense models at fixed validation perplexity (ablation) | MoEs vs dense on world knowledge · World-knowledge benchmarks | downstream accuracy | Helped |
| compare MoEs to dense models at fixed validation perplexity (ablation) | MoEs vs dense on reasoning · Commonsense and math benchmarks | downstream accuracy | No clear effect |
| MoE routing with more experts on math-trained models (train-test gap) | pre-trained MoEs trained on math data · OpenMathInstruct train vs GSM8k/Hendrycks-MATH test | generalization gap | Hurt |
Surprising findings
- MoEs show a larger train-test generalization gap on math than dense models beyond 159M parameters, suggesting memorization of pre-training problems.
- At fixed validation perplexity, MoEs outperform dense models on world-knowledge tasks but perform about the same on reasoning tasks.
Limitations
- Pre-trained models are limited to at most about 2.1B parameters, far below frontier MoEs like Mixtral or DeepSeek-V2.
- Only top-2 token-choice routing was studied; other routing mechanisms were left for future work.
- Tasks studied are stylized extremes of pure memorization or pure reasoning; mixed tasks are not analyzed.
Experimental setup
Datasets: FineWeb-edu Cosmopedia Wikipedia Proof-Pile 2 OpenMathInstruct MetaMathQA TriviaQA Natural Questions HellaSwag WinoGrande synthetic shortest path graphs synthetic phone-book
Benchmarks: TriviaQA Natural Questions HotpotQA WebQuestions ComplexWebQuestions ARC-C ARC-E CommonsenseQA HellaSwag OpenbookQA PIQA SciQ
Metrics: exact-match accuracy F1 validation perplexity maximal phone-book size at 90% accuracy generalization gap
Baselines: dense transformers at matched total parameters dense transformers at matched active parameters
Builds on: Mistral Mixtral OLMoE Megablocks Transformers as communication-complexity-limited models (Sanford et al.)
Closest papers by meaning
- Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient (ICML 2025 · similarity 0.90)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.89)
- On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks (NeurIPS 2025 · similarity 0.88)
- Toward Efficient Inference for Mixture of Experts (NeurIPS 2024 · similarity 0.88)
- BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts (NeurIPS 2024 · similarity 0.88)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.