Research map / Architectures and efficiency / Mixture of experts
BASE Layers: Simplifying Training of Large, Sparse Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BASE instead of data parallel training | Language model training · Higher compute budgets | validation perplexity | Helped |
| Use BASE instead of model parallel training | Language model training · Higher compute budgets | validation perplexity | Helped |
| Use BASE instead of data parallel training | Language model training · 8 GPUs | validation perplexity | No clear effect |
| Use BASE instead of Sparsely Gated MoE | Sparse language model training · Language modeling, validation | perplexity | No clear effect |
| Use BASE instead of Switch | Sparse language model training · Language modeling, validation | perplexity | Helped |
| Use BASE routing rather than Sparsely Gated MoE routing with a single midway expert layer (ablation) | Single-layer sparse Transformer · Language modeling, validation | perplexity | Helped |
| Vary the expert-to-shared parameter allocation across small, standard, and large experts (ablation) | BASE · Language modeling, validation | perplexity | No clear effect |
| Place the BASE layer at the top of the Transformer rather than midway (ablation) | BASE · Language modeling, validation | perplexity | No clear effect |
| Interleave three smaller BASE layers through the Transformer (ablation) | BASE · Language modeling, validation | perplexity | Helped |
| Use a single large BASE layer midway through the Transformer rather than interleaving expert layers (ablation) | BASE · Language modeling, validation | perplexity | Mixed |
| Use BASE instead of Sparsely Gated MoE | Sparse model training · Training, 128 GPUs | tokens per second: 292k -> 545k | Helped |
| Use BASE instead of Switch | Sparse model training · Training, 128 GPUs | tokens per second: 469k -> 545k | Helped |
Closest papers by meaning
- Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models (NeurIPS 2024 · similarity 0.88)
- Pruning Large Language Models with Semi-Structural Adaptive Sparse Training (AAAI 2025 · similarity 0.88)
- Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models (ICML 2025 · similarity 0.88)
- Sparse Layers are Critical to Scaling Looped Language Models (NeurIPS 2026 · similarity 0.88)
- Mixture-of-Experts Meets Instruction Tuning: A Winning Combination for Large Language Models (ICLR 2024 · similarity 0.88)
- FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.