Research map / Probabilistic and scientific modeling / Variational and generative sampling
A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Establishes convergence rates for MLE in softmax-gated multinomial logistic mixture of experts, showing slow (sub-polynomial) rates when expert parameters collapse, and proposes modified softmax gating with input transform M(X) that restores polynomial rates.
Problem
Theoretical understanding of maximum likelihood estimation convergence in mixture-of-experts models for classification (multinomial logistic experts) with softmax gating, which has been studied only for Gaussian (regression) MoE.
Approach
Introduces a Voronoi-based loss D_r for parameter estimation in softmax-gated multinomial logistic MoE, identifies a PDE-induced interaction between gating and expert parameters when expert parameters vanish, and proposes a modified softmax gate that transforms the input via M(X) to remove this interaction.
Key results
- Under standard softmax gating with some expert parameters vanishing, minimax lower bound of order n^{-1/2} holds for any polynomial-rate comparison, so rates can be slower than any polynomial (e.g. O(1/log n))
- Under regime 1 (no expert collapse), exact-specified parameters converge at O~(n^{-1/2}) and over-specified at O~(n^{-1/4})
- Modified softmax gating restores O~(n^{-1/4}) over-specified and O~(n^{-1/2}) exact-specified rates regardless of expert parameter values
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the standard softmax gate with a modified softmax gate using M(X) | Multinomial logistic MoE with collapsed expert parameters (Regime 2) · Regime 2 | over-specified parameter estimation rate: slower than $\widetilde{\mathcal{O}}(n^{-1/2r})$ for any $ r\geq 1$ -> $\widetilde{\mathcal{O}}(n^{-1/4})$ | Helped |
| Use the standard softmax gate when expert parameters satisfy Regime 1 rather than Regime 2 | Standard softmax-gating multinomial logistic MoE · Over-specified parameters | parameter estimation rate: slower than $\widetilde{\mathcal{O}}(n^{-1/2r})$ for any $ r\geq 1$ -> $\widetilde{\mathcal{O}}(n^{-1/4})$ | Helped |
Surprising findings
- Standard softmax gating can hurt parameter estimation rates, becoming slower than any polynomial rate when expert parameters collapse
Limitations
- Analysis restricted to well-specified setting; misspecified case left open
- Non-convex parameter space complicates MLE analysis under misspecification
- Effect of temperature parameter not studied
Experimental setup
Metrics: Hellinger distance Voronoi loss convergence rate
Baselines: standard softmax gating Gaussian MoE softmax gating rates
Builds on: Gaussian MoE convergence analysis Voronoi loss generalized Wasserstein loss
Closest papers by meaning
- On Least Square Estimation in Softmax Gating Mixture of Experts (ICML 2024 · similarity 0.93)
- Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts (NeurIPS 2024 · similarity 0.91)
- Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts? (ICML 2024 · similarity 0.91)
- Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts (ICLR 2024 · similarity 0.91)
- On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts (NeurIPS 2025 · similarity 0.90)
- Learning Mixtures of Experts with EM: A Mirror Descent Perspective (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.