Research map / Probabilistic and scientific modeling / Variational and generative sampling
Nearly Dimension-Independent Convergence of Mean-Field Black-Box Variational Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves that BBVI with the reparametrization gradient on mean-field location-scale families converges with nearly dimension-independent iteration complexity (O(log d) for sub-Gaussian families, O(d^{2/k}) for heavy-tailed families with k moments).
Problem
Known iteration complexity of BBVI on mean-field location-scale families scales as O(sqrt(d)), while full-rank families scale as O(d); whether mean-field BBVI can achieve a better dimension dependence is an open conjecture.
Approach
A probabilistic analysis of the reparametrization gradient variance for mean-field families shows that the heavy-tailed term from the maximal coordinate only affects one random coordinate, so its effect averages out over d, yielding a bound with max-of-d-coordinates (log d) dependence instead of sqrt(d). A matching lower bound shows the spectral Hessian bounds alone cannot improve this.
Key results
- Mean-field BBVI with sub-Gaussian variational family achieves O(log d) explicit dimension dependence in iteration complexity, versus O(sqrt(d)) prior bound and O(d) for full-rank.
- Heavy-tailed families (Student-t with nu>4) achieve O(d^{2/(nu-2)}) dependence.
- If the target Hessian is constant, complexity has no explicit dependence on d; lower bound shows Omega(L^2 log d) for gradient variance under spectral Hessian bounds alone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the full-rank location-scale family with a mean-field location-scale family with sub-Gaussian tails | BBVI with the reparametrization gradient · Strongly log-concave, log-smooth targets | iteration complexity dimension dependence: O(d) -> O(\log d) | Helped |
| Use a mean-field family instead of the previously analyzed mean-field guarantee | BBVI with the reparametrization gradient · Strongly log-concave, log-smooth targets | iteration complexity dimension dependence: O(\sqrt{d}) -> O(\log d) | Helped |
| Use a heavy-tailed mean-field family with k finite moments instead of a sub-Gaussian family | Mean-field BBVI · Strongly log-concave, log-smooth targets | iteration complexity dimension dependence: O(\log d) -> O(d^{2/k}) | Mixed |
| Use a target with constant Hessian | Mean-field BBVI · Quadratic targets | iteration complexity dimension dependence: O(\log d) -> no explicit dependence on d | Helped |
| Use a standard Gaussian base distribution | Mean-field BBVI · Strongly log-concave, log-smooth targets | iteration complexity dimension dependence: O(\log d) | Helped |
| Use a Student-t family with ν > 4 degrees of freedom | Mean-field BBVI · Strongly log-concave, log-smooth targets | iteration complexity dimension dependence: O(d^{2/(\nu - 2)}) | Mixed |
| Bound the mean-field scale-gradient variance using the maximum-coordinate decomposition | Mean-field reparametrization gradient · Strongly log-concave, log-smooth targets | gradient variance dimension dependence: O(r_4 \sqrt{d}) -> dependence through \mathbb{E}\max_{j=1,\ldots,d} \mathsfit{u}_j^2 | Helped |
| Rely only on spectral bounds on the target log-density Hessian | Mean-field BBVI gradient-variance analysis · Mean-field location-scale families | gradient variance bound | No clear effect |
Surprising findings
- Spectral bounds on the target Hessian alone cannot yield a dimension-independent gradient variance bound; a worst-case Hessian construction gives an Omega(log d) lower bound.
Limitations
- Results are theoretical bounds under strong log-concavity and log-smoothness of the target; no experiments reported.
- Heavy-tailed families give only polynomial O(d^{2/k}) dependence.
- Analysis uses linear parametrization with a proximal (log-barrier) step rather than common non-linear parametrizations.
Experimental setup
Metrics: iteration complexity expected squared parameter distance
Baselines: full-rank location-scale BBVI mean-field BBVI O(sqrt(d)) analysis
Builds on: black-box variational inference reparametrization gradient stochastic proximal gradient descent variance transfer
Closest papers by meaning
- On the Convergence of Black-Box Variational Inference (NeurIPS 2023 · similarity 0.93)
- Provably Scalable Black-Box Variational Inference with Structured Variational Families (ICML 2024 · similarity 0.93)
- Batch and match: black-box variational inference with a score-based divergence (ICML 2024 · similarity 0.89)
- EigenVI: score-based variational inference with orthogonal function expansions (NeurIPS 2024 · similarity 0.89)
- Least squares variational inference (NeurIPS 2025 · similarity 0.89)
- VIKING: Deep variational inference with stochastic projections (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.