Research map / Learning theory and algorithms / Clustering
Big Learning Expectation Maximization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BigLearn-EM, an EM upgrade that mixes joint, marginal, and randomly orthogonally-transformed marginal matching updates (all with closed-form EM-type updates) to escape bad local optima of GMM training.
Problem
EM for Gaussian mixture models is highly sensitive to initialization and frequently converges to bad local optima that can be arbitrarily worse than the optimum.
Approach
Borrowing the big learning principle from foundation-model pretraining, randomly alternate between joint EM, marginal EM over random index subsets, and marginal EM in randomly rotated (orthogonally transformed) domains, all with analytical EM-type updates, plus a MAP prior on mixture weights to avoid zeroed components.
Key results
- On the simulated 25-GMM, the full BigLearn-EM reduces test joint KL divergence to 0.030 (std 0.006) versus 0.263 (std 0.035) for Joint-EM.
- Across nine real-world clustering datasets, BigLearn-EM attains the best NMI or ARI on most datasets, e.g., Letter NMI 0.532 vs 0.492 for Joint-EM and Pendigits NMI 0.823 vs 0.771.
- On FashionMNIST, BigLearn-EM reaches NMI 0.681, the highest reported among compared deep clustering methods, with ACC 0.624 below VaGAN-GMM's 0.638.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MAP estimate for mixture weights (ablation) | Joint-EM on 25-GMM simulated datasets · 25-GMM simulated datasets | Test Joint KL Divergence: 0.263 -> 0.225 | Helped |
| Add marginal matching after MAP mixture-weight estimation (ablation) | Joint-EM on 25-GMM simulated datasets · 25-GMM simulated datasets | Test Joint KL Divergence: 0.225 -> 0.141 | Helped |
| Add conditional matching after marginal matching (ablation) | Joint-EM on 25-GMM simulated datasets · 25-GMM simulated datasets | Test Joint KL Divergence: 0.141 -> 0.124 | Helped |
| Add randomly transformed marginal matching (ablation) | BigLearn-EM on 25-GMM simulated datasets · 25-GMM simulated datasets | Test Joint KL Divergence: 0.124 -> 0.077 | Helped |
| Increase local updates from W=1 to W=5 (ablation) | BigLearn-EM on 25-GMM simulated datasets · 25-GMM simulated datasets | Test Joint KL Divergence: 0.077 -> 0.030 | Helped |
| Increase local updates from W=5 to W=10 (ablation) | BigLearn-EM on 25-GMM simulated datasets · 25-GMM simulated datasets | Test Joint KL Divergence: 0.030 -> 0.031 | No clear effect |
| Use BigLearn-EM instead of Joint-EM | GMM clustering · real-world benchmark datasets | NMI and ARI | Mixed |
| Use BigLearn-EM instead of Joint-EM | GMM clustering · real-world benchmark datasets | Joint-LL | Mixed |
| Use BigLearn-EM for clustering without deep neural networks | FashionMNIST clustering · FashionMNIST | NMI: 0.681 vs 0.670 | Helped |
| Use BigLearn-EM for clustering without deep neural networks | FashionMNIST clustering · FashionMNIST | ACC: 0.624 vs 0.638 | Mixed |
| Use BigLearn-EM instead of Joint-EM as training data become scarce | GMM clustering · Pendigits training-data scarcity experiment | NMI and ARI | Helped |
Surprising findings
- Joint matching can remain stuck in a bad local optimum while marginal matching on the same model continues to improve, so the consistency among matchings is not automatically implied.
Limitations
- No theoretical global convergence guarantee; the authors leave it as future work.
- Exploration may be weak; the algorithm can wander around certain states for many iterations.
- Uses a fixed iteration count rather than a principled stopping criterion.
Experimental setup
Datasets: Connect-4 Covtype Glass Letter Pendigits Satimage Seismic Svmguide2 Vehicle FashionMNIST simulated 25-GMM data
Benchmarks: Connect-4 Covtype Glass Letter Pendigits Satimage Seismic Svmguide2 Vehicle FashionMNIST
Metrics: NMI ARI Joint log-likelihood clustering accuracy (ACC) joint KL divergence
Baselines: K-Means SW-GMM WM-GMM Joint-EM DEC IDEC VaDE JULE DAC VaGAN-GMM EDESC
Builds on: Expectation Maximization big learning principle Sliced Wasserstein GMM (SW-GMM)
Closest papers by meaning
- Toward Global Convergence of Gradient EM for Over-Paramterized Gaussian Mixture Models (NeurIPS 2024 · similarity 0.89)
- Learning Mixtures of Experts with EM: A Mirror Descent Perspective (ICML 2025 · similarity 0.88)
- Local Maxima in the Likelihood of Gaussian Mixture Models: Structural Results and Algorithmic Consequences (NeurIPS 2016 · similarity 0.88)
- Convergence of Gradient EM on Multi-component Mixture of Gaussians (NeurIPS 2017 · similarity 0.88)
- Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework (NeurIPS 2025 · similarity 0.86)
- Is $\sqrt{d}$ Separation Necessary for Gradient EM to Learn Gaussian Mixtures in High Dimensions? (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.