Research map / Architectures and efficiency / Mixture of experts
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sparse MoE GLaM (64B/64E) instead of GPT-3 (175B) | Language modeling across 29 NLP tasks · Zero-shot average | accuracy average: 56.9 -> 62.7 | Helped |
| Use sparse MoE GLaM (64B/64E) instead of GPT-3 (175B) | Language modeling across 29 NLP tasks · One-shot average | accuracy average: 61.6 -> 65.5 | Helped |
| Use sparse MoE GLaM (64B/64E) instead of GPT-3 (175B) | Language modeling across 29 NLP tasks · Few-shot average | accuracy average: 65.2 -> 68.1 | Helped |
| Use sparse MoE GLaM (64B/64E) instead of GPT-3 (175B) | Inference · Token prediction | FLOPs / token (G): 350 -> 180 | Helped |
| Use sparse MoE GLaM (64B/64E) instead of GPT-3 (175B) | Training · Model training | Train energy (MWh): 1287 -> 456 | Helped |
| Use sparse MoE GLaM (64B/64E) instead of GPT-3 | TriviaQA · Open-Domain test, one-shot | performance: 68.0 -> 75.0 | Helped |
| Use sparse MoE GLaM (64B/64E) instead of GPT-3 | WinoGender · Full dataset, one-shot | accuracy: 64.2% -> 71.7% | Helped |
| Replace filtered webpages with unfiltered webpages in the training data (ablation) | GLaM (1.7B/64E) · NLG and NLU tasks, zero-, one-, and few-shot | task performance | Hurt |
| Add more experts per MoE layer while keeping prediction compute roughly fixed (ablation) | GLaM (1.7B) · Zero-, one-, and few-shot average performance | average score | Helped |
| Use MoE instead of dense layers at similar effective FLOPs (ablation) | GLaM family · NLG and NLU tasks, average zero-, one-, and few-shot performance | average score | Mixed |
Closest papers by meaning
- Scaling Laws for Fine-Grained Mixture of Experts (ICML 2024 · similarity 0.90)
- Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models (ICML 2025 · similarity 0.89)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.88)
- Scaling Laws for Upcycling Mixture-of-Experts Language Models (ICML 2025 · similarity 0.88)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.88)
- No Need to Talk: Asynchronous Mixture of Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.