Research map / Probabilistic and scientific modeling / Variational and generative sampling
On Variational Bounds of Mutual Information
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, George Tucker
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Interpolate between the batch mixture and learned marginal in the denominator | MI lower-bound estimation · Gaussian toy problem | MI estimate: At MI 10.0: I_alpha 8.9; CPC 6.0; NWJ 8.0; JS 8.1 | Improved |
| Interpolate between the batch mixture and learned marginal in the denominator | MI lower-bound estimation · Cubic toy problem | MI estimate: At MI 10.0: I_alpha 8.2; CPC 4.8; NWJ 6.9; JS 7.6 | Improved |
| Use linear interpolation between CPC and NWJ instead of the proposed denominator interpolation (ablation) | I_alpha · 20-dimensional Gaussian toy problem | bias-variance tradeoff | Hurt |
| Use product interpolation between the batch mixture and learned marginal instead of the proposed denominator interpolation (ablation) | I_alpha · 20-dimensional Gaussian toy problem | bias-variance tradeoff | Hurt |
| Increase alpha toward 1 (CPC-like bound) (ablation) | I_alpha gradient estimation · Correlated Gaussian | gradient MSE | Improved |
| Decrease alpha toward 0 (NWJ-like bound) (ablation) | I_alpha gradient estimation · Correlated Gaussian | gradient MSE | Improved |
| Use a joint critic rather than a separable critic (ablation) | NWJ and JS estimators · Gaussian and Cubic toy problems | MI estimate variance and performance | Improved |
| Use a joint critic rather than a separable critic (ablation) | Multi-sample bounds · Gaussian and Cubic toy problems | robustness to critic architecture | No clear effect |
| Use a known conditional p(y|x) to parameterize the critic, learning only q(y) (ablation) | Structured MI bounds · Gaussian toy problem with known p(y|x) | MI estimate: NWJ estimates: 2.0, 4.0, 6.0, 8.0, 10.0 | Improved |
| Use leave-one-out upper bound with a small batch size (ablation) | MI estimation · Correlated Gaussian | bias and variance | Hurt |
| Use CPC with a small batch size when the true MI is large (ablation) | MI estimation · Correlated Gaussian and Cubic toy problems | MI estimate | Hurt |
| Optimize the critic using JS gradients rather than NWJ gradients (ablation) | JS density-ratio estimator · MI estimation | critic optimization stability | Improved |
Closest papers by meaning
- Understanding the Limitations of Variational Mutual Information Estimators (ICLR 2020 · similarity 0.93)
- Mutual Information Estimation via $f$-Divergence and Data Derangements (NeurIPS 2024 · similarity 0.90)
- Connecting Jensen–Shannon and Kullback–Leibler Divergences: A New Bound for Representation Learning (NeurIPS 2025 · similarity 0.89)
- Exactly Tight Information-theoretic Generalization Bounds via Binary Jensen-Shannon Divergence (ICML 2025 · similarity 0.88)
- Slicing Mutual Information Generalization Bounds for Neural Networks (ICML 2024 · similarity 0.88)
- Towards Generalization beyond Pointwise Learning: A Unified Information-theoretic Perspective (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
