Research map / Learning theory and algorithms / Sample complexity
Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives statistical convergence rates for MLE in top-K sparse softmax gating Gaussian mixture of experts, under exact- and over-specified settings, via novel Voronoi loss functions.
Problem
Theoretical understanding of top-K sparse softmax gating in mixture of experts is missing: how density and parameter estimation converge when the gating partitions input space and the number of experts is known or over-specified.
Approach
Analyzes MLE for a Gaussian MoE with top-K sparse softmax gating by partitioning input space into regions induced by top-K selection and defining novel Voronoi-type loss functions (D1, D2) that capture region-specific behavior; shows parametric rates in exact-specified case and slowed parameter rates in over-specified case due to gating-expert interaction.
Key results
- Under exact-specified settings, density estimation converges under Hellinger distance at rate O~(n^{-1/2}) and parameter estimation at the same parametric rate.
- Under over-specified settings, density estimation remains O~(n^{-1/2}) provided the top-K' count exceeds the total cardinality of K Voronoi cells, but parameter rates slow to n^{-1/(2 r(|C|))} type orders (e.g., n^{-1/12} for beta1 and b when |C|=3).
- The solvability index r-bar(2)=4 and r-bar(3)=6 is cited from prior work and conjectured to be 2m for m>=2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the exact-specified model, with the true number of experts known | Top-K sparse softmax gating Gaussian MoE MLE · Exact-specified | density and parameter estimation rates: both are \mathcal{O}(n^{-1/2}) | Helped |
| Over-specify the model with more experts than the true number | Top-K sparse softmax gating Gaussian MoE MLE · Over-specified | density estimation rate: \mathcal{O}(n^{-1/2}) in both exact-specified and over-specified settings | No clear effect |
| Over-specify the model so a true component is fitted by multiple components | Top-K sparse softmax gating Gaussian MoE MLE · Over-specified; |\mathcal{C}_{j}|>1 | \beta_{1j}^{*}, b_{j}^{*} estimation rate: \mathcal{O}(n^{-1/2}) -> \mathcal{O}(n^{-1/2\bar{r}(|\mathcal{C}_{j}|)}) | Hurt |
| Over-specify the model so a true component is fitted by multiple components | Top-K sparse softmax gating Gaussian MoE MLE · Over-specified; |\mathcal{C}_{j}|>1 | a_{j}^{*}, \sigma_{j}^{*} estimation rate: \mathcal{O}(n^{-1/2}) -> \mathcal{O}(n^{-1/\bar{r}(|\mathcal{C}_{j}|)}) | Hurt |
| Increase the selected expert count to at least the maximum total cardinality of the relevant Voronoi cells | Over-specified density estimation with top-\overline{K} gating · Over-specified | Hellinger density convergence | Helped |
| Set the selected expert count below the maximum total cardinality of the relevant Voronoi cells | Over-specified density estimation with top-\overline{K} gating · Over-specified | Hellinger density distance: >0 | Hurt |
| Use top-1 sparse softmax gating instead of selecting multiple experts | Over-specified top-K sparse softmax gating Gaussian MoE · Over-specified parameter estimation | parameter estimation rate | Helped |
Surprising findings
- Parameter estimation rates under top-K sparse gating match those of dense softmax gating in the stated settings, implying no statistical cost for sparsity
- Parameter estimation rates slow sharply in over-specified settings even though density estimation remains parametric
Limitations
- Analysis restricted to Gaussian experts with linear expert functions and bounded parameter space and input space
- Theory assumes data generated from the model (well-specified); misspecified case left open
- EM convergence guarantees not established
Experimental setup
Metrics: Hellinger distance Voronoi loss
Builds on: Voronoi loss functions for softmax-gated MoE bracketing entropy density estimation (M-estimators)
Closest papers by meaning
- Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts? (ICML 2024 · similarity 0.92)
- Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts (NeurIPS 2024 · similarity 0.91)
- A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts (ICML 2024 · similarity 0.91)
- On Least Square Estimation in Softmax Gating Mixture of Experts (ICML 2024 · similarity 0.89)
- On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts (NeurIPS 2025 · similarity 0.87)
- From Sparse to Soft Mixtures of Experts (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.