Research map / Optimization / Optimizers
Learning Mixtures of Experts with EM: A Mirror Descent Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that EM for mixtures of experts (general exponential-family MoE, plus symmetric 2-expert linear and logistic MoE) is equivalent to projected mirror descent with unit step size and KL regularizer, yielding convergence guarantees and experiments showing EM beats gradient descent on synthetic and Fashion-MNIST data.
Problem
Training mixtures of experts by maximum likelihood is non-convex and usually done with gradient descent; the theoretical convergence behavior of EM for MoE is not well understood.
Approach
EM for MoE with exponential-family conditionals is shown to equal projected mirror descent on the conditional log-likelihood with unit step size and a KL-divergence mirror map, generalizing prior EM-as-MD results beyond complete-data exponential families; this yields sublinear and linear convergence conditions and SNR-based guarantees for symmetric 2-expert linear/logistic MoE.
Key results
- On altered Fashion-MNIST, EM reaches 78.5% test accuracy vs 66.0% for Gradient EM and 62.4% for gradient descent with a 2-component logistic MoE.
- Paired t-test on synthetic and Fashion-MNIST final accuracy gives T-statistic >= 22 and >= 17 for EM vs GD (p ~ 0.000).
- On synthetic SymMoLinE data, EM needs fewer iterations than Gradient EM and GD to reduce objective error and recover true parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EM instead of Gradient Descent | 2-component MoLogE · Altered Fashion MNIST test | accuracy: 62.4% -> 78.5% | Helped |
| Use EM instead of Gradient Descent | 2-component MoLogE · Altered Fashion MNIST test | cross entropy: 1.30 -> 0.827 | Helped |
| Use Gradient EM instead of Gradient Descent | 2-component MoLogE · Altered Fashion MNIST test | accuracy: 62.4% -> 66.0% | Helped |
| Use Gradient EM instead of Gradient Descent | 2-component MoLogE · Altered Fashion MNIST test | cross entropy: 1.30 -> 1.29 | Helped |
| Use a 2-component MoLogE instead of a single expert | Altered Fashion MNIST · Altered Fashion MNIST test | accuracy: 10.2% -> 78.5% | Helped |
| Use EM instead of Gradient EM | 2-component MoLogE · Altered Fashion MNIST test | accuracy: 66.0% -> 78.5% | Helped |
Limitations
- Stochastic and mini-batch EM are not covered by the theory; the authors leave this open.
- Real-world experiments are small-scale and use a synthetically altered Fashion-MNIST.
- Sufficient conditions for linear convergence depend on the SNR and local average-convexity assumptions that are hard to verify in general.
Experimental setup
Datasets: Fashion-MNIST
Benchmarks: Fashion-MNIST (randomly inverted)
Metrics: test accuracy cross-entropy loss objective error relative parameter error
Baselines: Gradient Descent Gradient EM
Builds on: Kunstner et al. EM as mirror descent Balakrishnan et al. EM local convergence framework Mixtures of linear regression EM analysis
Closest papers by meaning
- Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learning (ICML 2025 · similarity 0.90)
- On Least Square Estimation in Softmax Gating Mixture of Experts (ICML 2024 · similarity 0.88)
- Big Learning Expectation Maximization (AAAI 2024 · similarity 0.88)
- Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient (ICML 2025 · similarity 0.87)
- A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts (ICML 2024 · similarity 0.87)
- Dense Backpropagation Improves Training for Sparse Mixture-of-Experts (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.