Research map / Learning theory and algorithms / Sample complexity
On Least Square Estimation in Softmax Gating Mixture of Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes least squares estimation for a deterministic softmax-gated mixture-of-experts regression model, proving a parametric L2 rate for the regression function and expert-specific rates that depend on expert identifiability (fast for sigmoid/tanh FFNs, very slow for polynomial and input-independent experts).
Problem
Parameter and expert estimation for softmax-gated mixture of experts is theoretically understood only under well-specified probabilistic (Gaussian) MoE models; the deterministic regression setting with least squares estimation is unexplored.
Approach
Introduces a "strong identifiability" condition on expert functions (linear independence of the expert and its first- and second-order parameter derivatives multiplied by monomials) and a Voronoi-based loss, which together give L2 lower bounds that translate regression rates into expert parameter rates; also introduces "strong independence" for ridge activations and shows polynomial experts yield
Key results
- The LSE achieves a parametric L2 rate O_P(sqrt(log(n)/n)) for the regression function f_G under the deterministic softmax-gated MoE model.
- Under strong identifiability, expert parameters fitted by a single component converge at O_P(n^{-1/2}), while those fitted by multiple components converge at O_P(n^{-1/4}).
- For polynomial experts and input-independent ridge experts, minimax lower bounds show parameter rates slower than any polynomial rate, possibly as slow as O_P(1/log(n)).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use strongly identifiable expert functions instead of polynomial experts | Least squares estimation in deterministic softmax-gating MoE · Expert estimation | Expert estimation rate: \mathcal{O}_{P}(n^{-1/2}) or \mathcal{O}_{P}(n^{-1/4}) versus slower than any polynomial rates | Helped |
| Use strongly independent ridge activation with every expert input-dependent | Least squares estimation in deterministic softmax-gating MoE · Ridge expert estimation | Expert estimation rate: \mathcal{O}_P(n^{-1/2}) or \mathcal{O}_P(n^{-1/4}) versus slower than any polynomial rate | Helped |
| Allow at least one ridge expert input-weight parameter to be zero | Least squares estimation in deterministic softmax-gating MoE · Ridge expert estimation | Expert estimation rate: slower than any polynomial rate; could be as slow as \mathcal{O}_{P}(1/\log(n)) | Hurt |
| Use polynomial activation experts instead of strongly identifiable experts | Least squares estimation in deterministic softmax-gating MoE · Polynomial expert estimation | Expert estimation rate: slower than any polynomial rates; could be of order \mathcal{O}_{P}(1/\log(n)) | Hurt |
| Fit a strongly identifiable expert with more than one component rather than one component | Least squares estimator · Strongly identifiable expert estimation | Expert estimation rate: \mathcal{O}_{P}(n^{-1/2}) to \mathcal{O}_{P}(n^{-1/4}) | Hurt |
| Fit a ridge expert with strongly independent activation using more than one component rather than one component | Least squares estimator · Input-dependent ridge expert estimation | Expert estimation rate: \mathcal{O}_P(n^{-1/2}) to \mathcal{O}_P(n^{-1/4}) | Hurt |
Surprising findings
- Polynomial (including linear) experts have expert estimation rates slower than any polynomial in n in the deterministic setting, unlike the polynomial rates known for probabilistic Gaussian MoE with linear experts.
- Ridge experts with a zero expert parameter (input-independent expert) yield slow rates even when the activation is strongly independent.
Limitations
- Analysis assumes well-specified deterministic softmax MoE regression; misspecified settings are not covered.
- Depth of expert networks is not considered in the rate analysis.
- Results hold for fixed dimension and compact parameter spaces.
Experimental setup
Metrics: L2 estimation rate Voronoi loss convergence rate
Baselines: maximum likelihood estimation for Gaussian MoE
Builds on: Gaussian softmax gating MoE convergence theory (Ho and Nguyen) empirical process theory (van de Geer)
Closest papers by meaning
- A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts (ICML 2024 · similarity 0.93)
- Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts (NeurIPS 2024 · similarity 0.91)
- On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts (NeurIPS 2025 · similarity 0.91)
- Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts? (ICML 2024 · similarity 0.90)
- Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts (ICLR 2024 · similarity 0.89)
- Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learning (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.