Research map / Architectures and efficiency / Mixture of experts
On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical study proving shallow MoEs approximate functions on low-dimensional manifolds without the curse of dimensionality, and deep MoEs with depth O(L) and E experts per layer approximate compositionally sparse piecewise functions with E^L pieces, plus small synthetic validations.
Problem
Understanding the expressive power of mixture-of-experts networks for complex tasks with low-dimensional manifold structure or compositional sparsity, which existing theory does not explain.
Approach
Shows that MoE layers decompose approximation into localized subproblems on manifold charts (or per-coordinate subfunctions) with a gating network performing exact input-to-expert assignment; depth-2L MoEs compose L such blocks to realize E^L pieces, and derives approximation rates depending on intrinsic dimension rather than ambient dimension.
Key results
- Shallow 1-4-MoE on a circle embedded in R^D keeps test error ~3.2e-4 to 3.4e-4 for D from 16 to 128, showing no dimension-dependent degradation.
- On the 9-cube piecewise function, 2-3-MoE test error falls from 8.32e-5 (width 16) to 2.59e-6 (width 128), while the shallow 1-6-MoE plateaus and rises to 4.60e-5 at width 128.
- Main theorems: depth-2 MoE with E experts achieves rate m^{-kappa(f|U_i)/d ∧ 1/2}, versus m^{-kappa(f)/D ∧ 1/2} for dense networks; depth-2L MoE handles E^L pieces.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase input dimension from 16 to 128 for the 1-4-MoE | 1-4-MoE on the low-dimensional manifold function · Low-dimensional manifold function, D=16 to D=128 | test error: 3.40e-4 -> 3.42e-4 | No clear effect |
| Increase expert width from 16 to 128 in the 2-3-MoE | 2-3-MoE on the piecewise function with compositional sparsity · Piecewise function with compositional sparsity | test error: 8.32e-5 -> 2.59e-6 | Helped |
| Use a shallow 1-6-MoE instead of a depth-2 2-3-MoE | MoE on the piecewise function with compositional sparsity · Piecewise function with compositional sparsity | test error: At m=128: 2.59e-6 vs 4.60e-5 | Hurt |
| Increase expert width from 16 to 128 in the 1-6-MoE | 1-6-MoE on the piecewise function with compositional sparsity · Piecewise function with compositional sparsity | test error: 7.96e-5 -> 4.60e-5 | Mixed |
Surprising findings
- A shallow 1-6-MoE's test error stops improving and even worsens as width grows (2.17e-5 at m=32 to 4.60e-5 at m=128), unlike the deeper 2-3-MoE.
Limitations
- Results are approximation-theoretic only; no training dynamics or learnability guarantees.
- Analysis restricted to compositional sparsity and K=1 top routing; other sparsity forms left open.
- Experiments are small synthetic toy settings without real LLM tasks.
Experimental setup
Datasets: synthetic low-dimensional circle functions sin(5x1)+cos(3x2) synthetic piecewise compositionally sparse function on 9 unit cubes
Metrics: test error
Baselines: dense ReLU networks 1-6-MoE
Builds on: two-layer ReLU approximation theory manifold charts and partitions of unity Switch/GShard-style MoE
Closest papers by meaning
- Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient (ICML 2025 · similarity 0.91)
- Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learning (ICML 2025 · similarity 0.90)
- MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts (AAAI 2024 · similarity 0.90)
- Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization (NeurIPS 2024 · similarity 0.89)
- A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts (ICML 2024 · similarity 0.89)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.