Research map / Probabilistic and scientific modeling / Variational and generative sampling
Mutual Information Estimation via $f$-Divergence and Data Derangements
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes f-DIME, a family of discriminative mutual information estimators built from the variational f-divergence representation, plus a derangement-based marginal sampling strategy that avoids the log(N) upper bound of naive permutation.
Problem
Neural mutual information estimators based on variational lower bounds suffer from high bias or high variance, and naive shuffling to get marginal samples caps the estimate at log N.
Approach
Use the variational representation of any f-divergence (not just KL) as the training objective for a discriminator, recovering the density ratio without a partition-function term, and sample marginals via derangements of the batch so the estimate is not bounded by log N.
Key results
- GAN-DIME attains the lowest bias among the tested estimators in the Gaussian and cubic staircase settings, with variance comparable to HD-DIME.
- CPC is upper-bounded by log N, so it is unreliable unless the batch size is large; the derangement strategy avoids this bound.
- For d=20 and N=30k, GAN-DIME with the shift-based deranged architecture takes under 9 minutes versus about 55 minutes for the separable architecture.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace naive random permutation with derangement sampling | f-DIME · MI estimation | bias | Helped |
| Use the deranged architecture instead of the joint architecture | MI estimators · Large batch sizes | time: about 55 minutes -> 15 minutes | Helped |
| Use shift-based rather than random-based derangement (ablation) | GAN-DIME with deranged architecture · 5-step MI staircase, d=20, N=30k | time: 15 minutes -> less than 9 minutes | Helped |
| Use shift-based rather than random-based derangement (ablation) | MI estimators · MI estimation | MI estimate | No clear effect |
| Use GAN divergence rather than KL divergence (ablation) | f-DIME · High-MI Gaussian and cubic scenarios | bias | Helped |
| Use Hellinger distance squared rather than KL divergence (ablation) | f-DIME · High-MI Gaussian and cubic scenarios | bias | Helped |
| Use KL divergence rather than GAN divergence (ablation) | f-DIME · High-MI Gaussian and cubic scenarios | variance | Mixed |
| Remove partition-function estimation | f-DIME · MI estimation | variance | Helped |
| Use f-DIME instead of MINE, NWJ, or SMILE with τ = ∞ | MI estimators · High-MI estimation | variance | Helped |
| Use GAN-DIME rather than CPC | MI estimation · Varying batch size N | bias | Helped |
| Use f-DIME on image datasets | f-DIME · Image self-consistency tests | number of tests satisfied: two out of the three tests | Mixed |
Surprising findings
- Naive random permutation of marginal samples yields an estimator upper bounded by log N, which the authors prove and observe empirically.
- SMILE and GAN-DIME behave almost identically despite structurally different value functions, since both minimize the same f-divergence (JS-related).
Limitations
- Discriminative estimators are less precise when the true MI is high, passing only two of three self-consistency tests.
- NJEE fails in memory for d>30 and was excluded from image tests.
- Variance analysis assumes the optimal discriminator is known, an idealization of the practical training.
Experimental setup
Datasets: synthetic Gaussian samples synthetic cubic, half-cube, asinh, swiss roll, uniform, student distributions image datasets for self-consistency tests
Benchmarks: staircase MI estimation (Gaussian, cubic, half-cube, asinh, swiss roll, uniform, student) self-consistency tests on image datasets computational time analysis
Metrics: bias variance runtime
Baselines: MINE NWJ SMILE CPC NJEE
Builds on: f-GAN MINE NWJ SMILE CPC
Closest papers by meaning
- Connecting Jensen–Shannon and Kullback–Leibler Divergences: A New Bound for Representation Learning (NeurIPS 2025 · similarity 0.89)
- InfoNet: Neural Estimation of Mutual Information without Test-Time Optimization (ICML 2024 · similarity 0.88)
- Neural Mutual Information Estimation with Vector Copulas (NeurIPS 2025 · similarity 0.88)
- A Benchmark Suite for Evaluating Neural Mutual Information Estimators on Unstructured Datasets (NeurIPS 2024 · similarity 0.88)
- MINDE: Mutual Information Neural Diffusion Estimation (ICLR 2024 · similarity 0.87)
- Mutual Information Estimation via Normalizing Flows (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.