Research map / Architectures and efficiency / Mixture of experts
Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical analysis of maximum likelihood estimation for dense-to-sparse softmax-gated Gaussian MoE, showing temperature causes parameter rates slower than any polynomial, and proposes activation dense-to-sparse gates (sigmoid/GELU) that restore polynomial/parametric rates.
Problem
Dense-to-sparse softmax gating in Gaussian MoE has a temperature that interacts with gating and expert parameters, and the statistical rates of MLE for these parameters are not well understood (possibly slower than any polynomial rate).
Approach
Identify PDE-based interactions between the softmax temperature and gating/expert parameters that slow MLE rates, and propose an activation dense-to-sparse gate that applies an activation (e.g., sigmoid, GELU) to the gating linear output, with first/second-order independence conditions that remove these interactions.
Key results
- Under dense-to-sparse gating, the density estimation rate is parametric (O~(n^{-1/2})) but temperature and gating parameter estimation rates can be slower than any polynomial rate, as slow as O(1/log n).
- With activation dense-to-sparse gating (sigmoid), exact-specified parameter rates for beta_1, tau, a, b, nu all improve to O~(n^{-1/2}); over-specified a and beta_1 rates improve from slower-than-polynomial to O~(n^{-1/4}) or better.
- Simulations with sigmoid activation show empirical convergence rates of O~(n^{-1/2}) for both exact- and over-specified settings, matching theory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a standard temperature-controlled dense-to-sparse softmax gate | Gaussian MoE parameter estimation · Exact-specified | parameter estimation rate for β₁ and τ: \widetilde{\mathcal{O}}(n^{-1/2}) -> slower than \widetilde{\mathcal{O}}(n^{-1/2r}), \forall r\geq 1 | Hurt |
| Add a first-order-independent activation before softmax | Gaussian MoE with a dense-to-sparse gate · Exact-specified | parameter estimation rate for β₁ and τ: Slower than \widetilde{\mathcal{O}}(n^{-1/2r}), \forall r\geq 1 -> \widetilde{\mathcal{O}}(n^{-1/2}) | Helped |
| Use a standard temperature-controlled dense-to-sparse softmax gate | Gaussian MoE parameter estimation · Over-specified | parameter estimation rate for β₁, τ, and a: Slower than \widetilde{\mathcal{O}}(n^{-1/2r}),\forall r\geq 1 | Hurt |
| Add a second-order-independent activation before softmax | Gaussian MoE with a dense-to-sparse gate · Over-specified | parameter estimation rate for β₁, τ, and a: Slower than \widetilde{\mathcal{O}}(n^{-1/2r}),\forall r\geq 1 -> \widetilde{\mathcal{O}}(n^{-1/4}) | Helped |
| Use sigmoid activation in the softmax gate | Gaussian MoE MLE · Exact-specified and over-specified | empirical convergence rate: \widetilde{\mathcal{O}}(n^{-1/2}) | Helped |
Surprising findings
- Temperature-gating interactions make parameter estimation rates slower than any polynomial rate, which the authors say was not observed in prior Gaussian MoE work.
Limitations
- Analysis assumes well-specified data generated from the (activation) dense-to-sparse Gaussian MoE; misspecified case is left open.
- Results are for probabilistic Gaussian experts with linear experts; deterministic settings with general expert functions are not covered.
- Experiments are simulation-only with a small synthetic setup (k*=2).
Experimental setup
Datasets: synthetic data from Gaussian MoE models
Metrics: convergence rate of Voronoi losses (log-log slope) total variation distance Hellinger distance
Baselines: dense-to-sparse gating Gaussian MoE top-K sparse softmax gating MoE
Builds on: dense-to-sparse gate top-K sparse softmax gating Voronoi loss
Closest papers by meaning
- Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts (ICLR 2024 · similarity 0.92)
- A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts (ICML 2024 · similarity 0.91)
- On Least Square Estimation in Softmax Gating Mixture of Experts (ICML 2024 · similarity 0.90)
- Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts (NeurIPS 2024 · similarity 0.89)
- On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts (NeurIPS 2025 · similarity 0.88)
- A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.