Research map / Label-efficient and robust learning / Self-supervised and contrastive
On Mutual Information Maximization for Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Maximize InfoNCE or NWJ with an invertible RealNVP encoder | MNIST top/bottom-view representation learning · MNIST test, linear evaluation | accuracy | Helped |
| Use an invertible encoder adversarially trained to frustrate a linear classifier | RealNVP representation · MNIST test, linear evaluation | accuracy | Hurt |
| Allow the estimator to favor ill-conditioned, hard-to-invert encoders | MLP representation initialized near identity · MNIST with coupled input noise | Jacobian condition number | Hurt |
| Replace the bilinear critic with a higher-capacity MLP critic (ablation) | InfoNCE and NWJ representation learning · MNIST test, linear evaluation | accuracy | Hurt |
| Use a bilinear critic instead of a higher-capacity MLP critic (ablation) | NWJ representation learning · MNIST test, linear evaluation | accuracy | Helped |
| Use a separable critic instead of an MLP critic (ablation) | InfoNCE and NWJ representation learning · MNIST test, linear evaluation | accuracy | Helped |
| Use a ConvNet encoder instead of an MLP encoder at the same target MI lower bound (ablation) | InfoNCE and NWJ representation learning · MNIST test, linear evaluation | accuracy | Helped |
| Change the MI estimator between InfoNCE and NWJ while matching the target lower bound (ablation) | MLP and ConvNet encoders · MNIST test, linear evaluation | accuracy | No clear effect |
| Draw negative samples dependently rather than i.i.d. (ablation) | InfoNCE and NWJ estimation · Synthetic Gaussian experiment | MI estimate relative to true MI | Hurt |
| Draw negative samples i.i.d. (ablation) | InfoNCE and NWJ estimation · Synthetic Gaussian experiment | MI estimate relative to true MI | Helped |
| Maximize a tighter MI lower bound using a higher-capacity critic (ablation) | MI-based representation learning · MNIST test, linear evaluation | accuracy | Hurt |
| Increase the number of negative samples | Contrastive representation learning · Unspecified | downstream performance | Mixed |
Closest papers by meaning
- Mutual Information Gradient Estimation for Representation Learning (ICLR 2020 · similarity 0.91)
- Learning deep representations by mutual information estimation and maximization (ICLR 2019 · similarity 0.90)
- Improved Mutual Information Estimation (AAAI 2021 · similarity 0.89)
- Connecting Jensen–Shannon and Kullback–Leibler Divergences: A New Bound for Representation Learning (NeurIPS 2025 · similarity 0.88)
- Mutual Information Estimation via $f$-Divergence and Data Derangements (NeurIPS 2024 · similarity 0.87)
- InfoNet: Neural Estimation of Mutual Information without Test-Time Optimization (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.