AI Paper Atlas

Research map / Probabilistic and scientific modeling / Variational and generative sampling

On Variational Bounds of Mutual Information

Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, George Tucker

ICML 2019 · Probabilistic and scientific modeling › Variational and generative sampling · approach: Amortized and online variational inference · 143 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Interpolate between the batch mixture and learned marginal in the denominatorMI lower-bound estimation · Gaussian toy problemMI estimate: At MI 10.0: I_alpha 8.9; CPC 6.0; NWJ 8.0; JS 8.1Improved
Interpolate between the batch mixture and learned marginal in the denominatorMI lower-bound estimation · Cubic toy problemMI estimate: At MI 10.0: I_alpha 8.2; CPC 4.8; NWJ 6.9; JS 7.6Improved
Use linear interpolation between CPC and NWJ instead of the proposed denominator interpolation (ablation)I_alpha · 20-dimensional Gaussian toy problembias-variance tradeoffHurt
Use product interpolation between the batch mixture and learned marginal instead of the proposed denominator interpolation (ablation)I_alpha · 20-dimensional Gaussian toy problembias-variance tradeoffHurt
Increase alpha toward 1 (CPC-like bound) (ablation)I_alpha gradient estimation · Correlated Gaussiangradient MSEImproved
Decrease alpha toward 0 (NWJ-like bound) (ablation)I_alpha gradient estimation · Correlated Gaussiangradient MSEImproved
Use a joint critic rather than a separable critic (ablation)NWJ and JS estimators · Gaussian and Cubic toy problemsMI estimate variance and performanceImproved
Use a joint critic rather than a separable critic (ablation)Multi-sample bounds · Gaussian and Cubic toy problemsrobustness to critic architectureNo clear effect
Use a known conditional p(y|x) to parameterize the critic, learning only q(y) (ablation)Structured MI bounds · Gaussian toy problem with known p(y|x)MI estimate: NWJ estimates: 2.0, 4.0, 6.0, 8.0, 10.0Improved
Use leave-one-out upper bound with a small batch size (ablation)MI estimation · Correlated Gaussianbias and varianceHurt
Use CPC with a small batch size when the true MI is large (ablation)MI estimation · Correlated Gaussian and Cubic toy problemsMI estimateHurt
Optimize the critic using JS gradients rather than NWJ gradients (ablation)JS density-ratio estimator · MI estimationcritic optimization stabilityImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy