Research map / Learning theory and algorithms / Sample complexity
Connecting Jensen–Shannon and Kullback–Leibler Divergences: A New Bound for Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a tight lower bound on KL divergence as a function of JSD, shows that minimizing discriminator cross-entropy yields a provable lower bound on mutual information, and evaluates it as an MI estimator and in the Information Bottleneck.
Problem
Maximizing discriminator-based Jensen-Shannon divergence between joint and marginal distributions is widely used for representation learning, but its relationship to mutual information (defined via KL divergence) is unclear, and standard variational MI lower bounds are unstable.
Approach
Uses the joint range of (JSD, KLD) for Bernoulli pairs to derive an optimal lower bound Xi(JSD) <= KLD, and shows that the cross-entropy of a binary joint-vs-marginal discriminator gives log2 - CE <= JSD, so minimizing CE increases a provable lower bound on MI.
Key results
- In the discrete setting, the JSD-based bound Xi(JSD) nearly coincides with true MI across alpha and category counts k from 2 to 500.
- On MNIST IB generalization, JSD-LB reaches 98.8% test accuracy vs 98.2% for DisenIB and 97.6% for VIB.
- Under adversarial attack at epsilon=0.3 on MNIST training, JSD-LB achieves 91.4% vs 68.4% for DisenIB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the Information Bottleneck MI estimator with JSD-LB | Information Bottleneck · MNIST test | classification accuracy: 98.2 -> 98.8 | Helped |
| Replace the Information Bottleneck MI estimator with JSD-LB | Information Bottleneck · MNIST adversarial training, ε = 0.1 | accuracy: 94.3 -> 99.5 | Helped |
| Replace the Information Bottleneck MI estimator with JSD-LB | Information Bottleneck · MNIST adversarial training, ε = 0.2 | accuracy: 81.5 -> 96.0 | Helped |
| Replace the Information Bottleneck MI estimator with JSD-LB | Information Bottleneck · MNIST adversarial training, ε = 0.3 | accuracy: 68.4 -> 91.4 | Helped |
| Replace the Information Bottleneck MI estimator with JSD-LB | Information Bottleneck · MNIST adversarial testing, ε = 0.1 | accuracy: 90.2 -> 94.6 | Helped |
| Replace the Information Bottleneck MI estimator with JSD-LB | Information Bottleneck · MNIST adversarial testing, ε = 0.2 | accuracy: 80.0 -> 89.6 | Helped |
| Replace the Information Bottleneck MI estimator with JSD-LB | Information Bottleneck · MNIST adversarial testing, ε = 0.3 | accuracy: 67.8 -> 86.1 | Helped |
| Replace the Information Bottleneck MI estimator with JSD-LB | Information Bottleneck · MNIST OOD detection | TPR95: 0.0 -> 0.00 | No clear effect |
| Replace the Information Bottleneck MI estimator with JSD-LB | Information Bottleneck · MNIST OOD detection | AUROC: 99.4 -> 100.0 | Helped |
| Replace the Information Bottleneck MI estimator with JSD-LB | Information Bottleneck · MNIST OOD detection | AUPR In: 99.6 -> 99.9 | Helped |
| Replace the Information Bottleneck MI estimator with JSD-LB | Information Bottleneck · MNIST OOD detection | AUPR Out: 98.9 -> 99.9 | Helped |
| Replace the Information Bottleneck MI estimator with JSD-LB | Information Bottleneck · MNIST OOD detection | Detection Error: 1.7 -> 0.1 | Helped |
Surprising findings
- The MI-specialized JSD bound stays tight across discrete joint-marginal families, suggesting the general optimal bound is not loosened when restricted to MI.
Limitations
- Xi has no closed form and is approximated numerically or via a logit approximation.
- The CE-based bound is tight only in the nonparametric limit of infinite data and model capacity.
Experimental setup
Datasets: MNIST synthetic Gaussian and non-Gaussian MI benchmarks Gaussian noise (OOD)
Benchmarks: MNIST synthetic Gaussian/non-Gaussian MI estimation benchmarks
Metrics: mutual information estimate classification accuracy adversarial accuracy TPR95 AUROC AUPR detection error
Baselines: MINE NWJ CPC VIB NIB squared-VIB squared-NIB DisenIB
Builds on: MINE NWJ InfoNCE GAN-DIME Deep InfoMax
Closest papers by meaning
- Improved Mutual Information Estimation (AAAI 2021 · similarity 0.91)
- Mutual Information Estimation via $f$-Divergence and Data Derangements (NeurIPS 2024 · similarity 0.89)
- Statistical and Geometrical properties of the Kernel Kullback-Leibler divergence (NeurIPS 2024 · similarity 0.87)
- Exactly Tight Information-theoretic Generalization Bounds via Binary Jensen-Shannon Divergence (ICML 2025 · similarity 0.87)
- Generalization Guarantees for Representation Learning via Data-Dependent Gaussian Mixture Priors (ICLR 2025 · similarity 0.86)
- A Benchmark Suite for Evaluating Neural Mutual Information Estimators on Unstructured Datasets (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.