Research map / Label-efficient and robust learning / Self-supervised and contrastive
Contrastive Predictive Coding Done Right for Mutual Information Estimation
Jongha Jon Ryu, Pavan Yeddanapudi, Xiangxiang Xu, Gregory W. Wornell
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add an anchor class (nu=1) to InfoNCE (ablation) | InfoNCE for SSL · CIFAR-100 linear probe, ResNet-18 | top-1 accuracy: 65.98 -> 65.74 | No clear effect |
| add an anchor class to InfoNCE for MI estimation | InfoNCE · MI benchmark (Gaussian, MNIST, BERT text) | MI estimation bias and variance | Improved |
| use large K (K=B-1) instead of K=1 (JS plug-in) (ablation) | InfoNCE-anchor · MNIST and text MI estimation | MI estimation accuracy | Improved |
| use large K instead of K=1 (JS) in SSL pretraining (ablation) | contrastive objective · CIFAR-100 linear probe, top-1 | top-1 accuracy: 61.69 -> 65.74 | Improved |
| replace log score with spherical proper scoring rule (ablation) | InfoNCE-anchor · MI estimation | MI estimation accuracy | Hurt |
| replace log score with spherical proper scoring rule (ablation) | InfoNCE-anchor · CIFAR-100 linear probe, top-1 | top-1 accuracy: 65.74 -> 4.33 | Hurt |
| replace log score with chi-squared scoring rule (ablation) | InfoNCE-anchor · CIFAR-100 linear probe, top-1 | top-1 accuracy: 65.74 -> 65.59 | No clear effect |
| use InfoNCE-anchor plug-in estimator for protein interaction prediction instead of other variational estimators | PMI estimator on ProtTrans5 embeddings · OmniPath kinase-target and ligand-receptor | ROC prediction | Improved |
Closest papers by meaning
- Decomposed Mutual Information Estimation for Contrastive Representation Learning (ICML 2021 · similarity 0.88)
- InfoNet: Neural Estimation of Mutual Information without Test-Time Optimization (ICML 2024 · similarity 0.85)
- Connecting Jensen–Shannon and Kullback–Leibler Divergences: A New Bound for Representation Learning (NeurIPS 2025 · similarity 0.85)
- Mutual Information Estimation via $f$-Divergence and Data Derangements (NeurIPS 2024 · similarity 0.85)
- A Benchmark Suite for Evaluating Neural Mutual Information Estimators on Unstructured Datasets (NeurIPS 2024 · similarity 0.85)
- On Discriminative Probabilistic Modeling for Self-Supervised Representation Learning (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
