Research map / Probabilistic and scientific modeling / Variational and generative sampling
Efficient Mixture Learning in Black-Box Variational Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MISVAE, a mixture VAE whose encoder amortizes all mixture components via one-hot encodings, plus two cheaper unbiased/lower-bound MISELBO estimators (Some-to-All and Some-to-Some) that scale to hundreds of mixture components.
Problem
Mixture variational distributions in black-box variational inference scale poorly: learnable parameters grow linearly with the number of components A and ELBO evaluation (MISELBO) cost grows quadratically with A.
Approach
An encoder shares all weights across mixture components, conditioning on one-hot component encodings as biases (MISVAE), and two subset-sampling estimators of MISELBO (S2A, unbiased for S<A; S2S, a lower bound) reduce cost from A×A to S×A or S×S.
Key results
- On MNIST, MISVAE S2A with A=800, S=1 reaches NLL 74.07 with 798,781 parameters, beating prior SOTA composite SEMVAE (77.23).
- MISVAE S2A (A=200, S=1) trains in 73.06 s/epoch versus 261.71 s/epoch for A=800, showing marginal time growth with A.
- On eight phylogenetic datasets, S2A and S2S with fewer likelihood and variational evaluations match the A2A VBPI-mixture NLL values within reported standard deviations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of mixture components A while holding S fixed in S2S (ablation) | MISVAE · MNIST | NLL | Helped |
| Increase A while holding S fixed in S2S (ablation) | MISVAE · CIFAR-10 | NLL | Mixed |
| Vary S while holding A fixed in S2A (ablation) | MISVAE · MNIST and CIFAR-10 | NLL | No clear effect |
| Increase S from 1 to 4 in S2A (ablation) | MISVAE · MNIST test set | FID: 10.87 -> 9.70 | Helped |
| Use S2A instead of A2A with fewer sampled components | VBPI with NFs and Mixtures · Bayesian phylogenetics | NLL | No clear effect |
| Increase S in S2S toward A (ablation) | MISVAE · MNIST | NLL | Mixed |
| Use MISVAE with S2S (A=50, S=20) instead of the cited MNIST VAE baselines | MNIST VAE models · MNIST | NLL: 77.23 -> 76.67 | Helped |
| Use MISVAE with S2A (A=800, S=1) instead of the cited MNIST VAE baselines | MNIST VAE models · MNIST | NLL: 76.93 -> 74.07 | Helped |
Surprising findings
- S2S with large fixed A and small S can outperform A2A with equal number of components in NLL, which the authors attribute to ensemble effects.
- For CIFAR-10, MISVAE (NLL/BPD 3.19-3.23) is reported well behind the best cited CR-NVAE (2.51).
Limitations
- Restricted to uniformly weighted mixtures; weighted mixtures tend to mode-collapse per cited analyses.
- Gap to the best CIFAR-10 results reported in the appendix.
- Theoretical convergence results for mixture BBVI are left to future work.
Experimental setup
Datasets: MNIST FashionMNIST CIFAR-10 synthetic toy posterior (Neal's funnel with Bernoulli likelihood) eight phylogenetic datasets (DS1-DS8)
Benchmarks: MNIST FashionMNIST CIFAR-10 eight phylogenetic datasets (DS1-DS8)
Metrics: negative log-likelihood bits per dimension FID training time per epoch number of network parameters MISELBO
Baselines: SEMVAE IWAE NVAE VampPrior hierarchical VAE MAE CR-NVAE A2A estimator MrBayes GGNN EDGE
Builds on: MISELBO IWAE VBPI with normalizing flows and mixtures SEMVAE VIMCO
Closest papers by meaning
- Variational Inference with Mixtures of Isotropic Gaussians (NeurIPS 2025 · similarity 0.89)
- Batch and match: black-box variational inference with a score-based divergence (ICML 2024 · similarity 0.89)
- End-to-end Learning of Gaussian Mixture Priors for Diffusion Sampler (ICLR 2025 · similarity 0.89)
- EigenVI: score-based variational inference with orthogonal function expansions (NeurIPS 2024 · similarity 0.88)
- Theoretical Guarantees for Variational Inference with Fixed-Variance Mixture of Gaussians (ICML 2024 · similarity 0.88)
- Denoising Diffusion Variational Inference: Diffusion Models as Expressive Variational Posteriors (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.