Research map / Architectures and efficiency / Mixture of experts
On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies maximum likelihood estimation in a softmax-gated contaminated mixture of experts (frozen pre-trained expert plus trainable Gaussian prompt expert), deriving parametric and slower-than-parametric minimax-optimal rates depending on a novel distinguishability condition.
Problem
Theoretical properties of parameter estimation in softmax-contaminated mixture of experts, where a frozen pre-trained model is fine-tuned by a new trainable prompt expert; estimability degrades when the prompt overlaps with the pre-trained model.
Approach
Introduces an analytic distinguishability condition between pre-trained and prompt experts and a strong identifiability condition on expert functions; under distinguishability MLE rates are parametric and minimax optimal, while under non-distinguishability rates slow according to the prompt's convergence rate to the pre-trained model.
Key results
- Under distinguishability, empirical MLE rates were O(n^-0.45), O(n^-0.52), O(n^-0.50), O(n^-0.54) for beta, tau, eta, nu, matching the theoretical O(n^-1/2).
- In the non-distinguishable case with eta* at n^-1/8 perturbation, exp(tau) error decayed as O(n^-0.23), consistent with the predicted n^-1/4.
- Softmax gating yields parametric O(n^-1/2) rates for expert and variance parameters under distinguishability, versus slower rates depending on the gating parameter for input-free gating.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use softmax gating instead of input-free gating | Contaminated MoE in the distinguishable setting · Distinguishable setting | expert parameter and variance estimation rate: $\widetilde{\mathcal{O}}(n^{-1/2}(\lambda^*)^{-1})$ -> $\widetilde{\mathcal{O}}(n^{-1/2})$ | Helped |
| Use softmax gating instead of input-free gating | Contaminated MoE in the non-distinguishable setting · Non-distinguishable setting | expert parameter and variance estimation rate: $\widetilde{\mathcal{O}} (n^{-\frac{1}{2}}\cdot\Vert(\Delta\eta^*,\Delta\nu^*)\Vert^{-1}(\lambda^*)^{-1})$ -> $\widetild | Helped |
| Allow the prompt model to overlap in expertise with the pre-trained model | Softmax-contaminated MoE MLE · Distinguishable versus non-distinguishable settings | parameter estimation rate: $\widetilde{\mathcal{O}} (n^{-\frac{1}{2}})$ -> rates depend on $\Vert(\Delta\eta^*,\Delta\nu^*)\Vert$ | Hurt |
Surprising findings
- Under non-distinguishability, the gating parameter rate degrades to n^-1/4 when the prompt parameters converge at n^-1/8, slower than the parametric rate.
Limitations
- Prompt model restricted to Gaussian densities
- Single prompt expert only; multiple prompts left for future work
- Non-distinguishable results require restricting the parameter space to Xi(l_n) for consistency
Experimental setup
Datasets: synthetic Gaussian covariates with softmax-contaminated MoE responses
Benchmarks: synthetic simulations
Metrics: parameter estimation error empirical convergence rate (log-log slope)
Baselines: input-free-gated contaminated MoE
Builds on: prefix tuning Nguyen et al. contaminated mixture of experts Yan et al. contaminated mixture of experts
Closest papers by meaning
- On Least Square Estimation in Softmax Gating Mixture of Experts (ICML 2024 · similarity 0.91)
- A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts (ICML 2024 · similarity 0.90)
- Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts (NeurIPS 2024 · similarity 0.89)
- Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts? (ICML 2024 · similarity 0.88)
- Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts (ICLR 2024 · similarity 0.87)
- Theory on Mixture-of-Experts in Continual Learning (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.