Research map / Learning theory and algorithms / Sample complexity
Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical analysis showing sigmoid-gated mixture-of-experts yields faster least-squares expert estimation rates than softmax gating, with a small synthetic validation.
Problem
Whether sigmoid gating is more sample efficient than the widely used softmax gating for expert estimation in mixture-of-experts regression, which had lacked theoretical support.
Approach
Analyzes least squares estimation for sigmoid-gated MoE regression in the over-specified case (k > k*), defines strong and weak identifiability for expert functions and Voronoi losses, and shows sigmoid gating gives parametric or faster expert rates than softmax gating, especially for ReLU/GELU and polynomial experts.
Key results
- Under Regime 2 with ReLU experts, sigmoid gating attains parametric O_P(n^{-1/2}) expert estimation rates while softmax gating gives O_P(n^{-1/4}).
- Empirically, Voronoi loss for sigmoid gating decays as O(n^{-0.51}) with ReLU experts versus O(n^{-0.24}) for softmax.
- With linear experts, sigmoid decays as O(n^{-0.40}) versus O(n^{-0.11}) for softmax.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace softmax gating with sigmoid gating | MoE with ReLU/GELU experts · Expert estimation, Regime 1 | expert estimation rate: ${\mathcal{O}_{P}}(n^{-1/4})$ -> ${\mathcal{O}_{P}}(n^{-1/4})$ | No clear effect |
| replace softmax gating with sigmoid gating | MoE with ReLU/GELU experts · Expert estimation, Regime 2 | expert estimation rate: ${\mathcal{O}_{P}}(n^{-1/4})$ -> ${\mathcal{O}_{P}}(n^{-1/2})$ | Helped |
| replace softmax gating with sigmoid gating | MoE with polynomial experts · Expert estimation, Regime 1 | expert estimation rate: ${\mathcal{O}_{P}}(1/\log(n))$ -> ${\mathcal{O}_{P}}(1/\log(n))$ | No clear effect |
| replace softmax gating with sigmoid gating | MoE with polynomial experts · Expert estimation, Regime 2 | expert estimation rate: ${\mathcal{O}_{P}}(1/\log(n))$ -> ${\mathcal{O}_{P}}(n^{-1/2})$ | Helped |
| replace softmax gating with sigmoid gating | MoE with ReLU experts · Synthetic ReLU expert, Regime 2 | Voronoi loss convergence rate: \mathcal{O}(n^{-0.24}) -> \mathcal{O}(n^{-0.51}) | Helped |
| replace softmax gating with sigmoid gating | MoE with linear experts · Synthetic linear expert, Regime 2 | Voronoi loss convergence rate: \mathcal{O}(n^{-0.11}) -> \mathcal{O}(n^{-0.40}) | Helped |
Surprising findings
- Under Regime 1, sigmoid and softmax gating yield the same expert estimation rates for ReLU/GELU experts.
Limitations
- Ground-truth parameters assumed independent of sample size, so rates are pointwise rather than uniform.
- Assumes the true regression function lies in the sigmoid-gated MoE class (well-specified), which is restrictive.
Experimental setup
Datasets: synthetic regression data (Uniform([-1,1]^32) inputs, softmax-gated MoE ground truth)
Benchmarks: synthetic regression with Voronoi loss
Metrics: Voronoi loss empirical convergence rate exponent
Baselines: softmax gating
Builds on: softmax gating MoE convergence analysis (Nguyen et al.) Voronoi loss (Manole et al.)
Closest papers by meaning
- A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts (ICML 2024 · similarity 0.91)
- Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts (ICLR 2024 · similarity 0.91)
- On Least Square Estimation in Softmax Gating Mixture of Experts (ICML 2024 · similarity 0.91)
- On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts (NeurIPS 2025 · similarity 0.89)
- Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts? (ICML 2024 · similarity 0.89)
- Learning to Specialize: Joint Gating-Expert Training for Adaptive MoEs in Decentralized Settings (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.