Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes

Sublinear Variational Optimization of Gaussian Mixture Models with Millions to Billions of Parameters

NeurIPS 2026 · Probabilistic and scientific modeling › Uncertainty and Gaussian processes · approach: Sparse variational Gaussian processes · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives a truncated variational EM (v-MFA) for mixtures of factor analyzers with a guided search-space construction, cutting per-iteration cost to sublinear in the number of components, and trains MFAs with up to ~10.8B parameters on YFCC100M on one CPU.

Problem

Training general Gaussian mixture models (arbitrary covariances) is computationally prohibitive for large numbers of data points N, high dimensionality D and many components C, since standard EM costs O(NCD^2) per iteration and scales linearly with NC.

Approach

Combines truncated variational EM with mixtures of factor analyzers, and replaces Euclidean component-similarity heuristics with a KL-divergence estimate computed from data assigned to each component, so partial E-steps only evaluate joints for a small guided search space per data point.

Key results

• v-MFA achieved speed-ups of 3.3x to 25.6x over em-MFA across hyperparameter settings on four benchmarks, with relative NLL deviations below 0.32%.
• On YFCC100M, v-MFA trained an MFA with 10.8 billion parameters (C=500000) in under nine hours on a single 64-core CPU; for C=500 em-MFA needed about two days vs about 5 hours for v-MFA.
• In zero-shot denoising, v-MFA is reported as about twice as fast as N2F and over 2000x faster than S2S, with the highest PSNR/SSIM on the Confocal data set.

Surprising findings

• Joint evaluations per data point scale sublinearly (exponents below 1/3) with the number of components C.
• The only method faster than v-MFA in denoising was the non-blind filter-based BM3D.

Limitations

• v-MFA requires more iterations than em-MFA, and its NLL is slightly worse than em-MFA on EMNIST and SVHN.
• Empty components occurred for large C (under 0.2% of components) and were ignored.
• Denoising PSNR/SSIM of v-MFA is often behind S2S on artificial Gaussian noise benchmarks.

Experimental setup

Datasets: CIFAR-10 CelebA EMNIST SVHN YFCC100M BSD68 Set12 Confocal (Fluorescence Microscopy Data set) Div2K

Benchmarks: CIFAR-10 CelebA EMNIST SVHN YFCC100M BSD68 Set12 Confocal Div2K Tiger

Metrics: negative log-likelihood (relative) wall-clock runtime number of joint evaluations PSNR SSIM processing speed (kilopixels/s)

Baselines: em-MFA k-means+FA Noise2Self Noise2Void Self2Self Noise2Fast Deep Image Prior DivNoising Pixel2Pixel BM3D

Builds on: mixtures of factor analyzers truncated variational EM AFK-MC2 k-means++

Closest papers by meaning

• Toward Global Convergence of Gradient EM for Over-Paramterized Gaussian Mixture Models (NeurIPS 2024 · similarity 0.90)
• End-to-end Learning of Gaussian Mixture Priors for Diffusion Sampler (ICLR 2025 · similarity 0.88)
• Is $\sqrt{d}$ Separation Necessary for Gradient EM to Learn Gaussian Mixtures in High Dimensions? (NeurIPS 2026 · similarity 0.87)
• Soft Mixture Denoising: Beyond the Expressive Bottleneck of Diffusion Models (ICLR 2024 · similarity 0.87)
• Minimax Optimal Early-Stopped Gradient Descent for Gaussian Mixture Classification (NeurIPS 2026 · similarity 0.87)
• Gaussian Mixture Flow Matching Models (ICML 2025 · similarity 0.86)