Research map / Probabilistic and scientific modeling / Variational and generative sampling
MINDE: Mutual Information Neural Diffusion Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MINDE, a mutual information estimator built on score-based diffusion models via a Girsanov-based identity expressing KL divergence as a difference of score functions, with conditional and joint diffusion variants.
Problem
Estimating mutual information between high-dimensional random variables, especially for challenging distributions (long-tailed, sparse, transformed, high MI) and under self-consistency tests (data processing, additivity).
Approach
Use score-based diffusion models to estimate KL divergence as the expected squared difference of score functions (via Girsanov theorem and time reversal), yielding entropy and MI estimators; introduce a joint diffusion process with coefficients alpha, beta that trains one score network for joint and conditional scores.
Key results
- MINDE-C achieves an error within 0.1 nats of ground truth on 35/40 benchmark tasks, versus 24/40 for MINE, the second-best method.
- MINDE variants are reported to pass all self-consistency tests (independence, data processing, additivity) on MNIST, where competing neural estimators do not pass all simultaneously.
- On high-MI transformed distributions (Half-cube, Spiral), MINDE outperforms neural baselines according to the authors' high MI benchmark.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use conditional diffusion variant (MINDE-C) instead of neural baselines | MI estimation · MI benchmark (40 tasks) | tasks within 0.1 nats of ground truth: 35/40 vs MINE 24/40 | Helped |
Surprising findings
- The estimator is neither an upper nor a lower bound on the true KL; errors from the score network can cancel out when the same error appears in both score functions.
Limitations
- Estimator error depends on score approximation quality and finite diffusion time T, with no tight quantified guarantees.
- Latent-space application for MNIST relies on autoencoders trained per consistency setting.
Experimental setup
Datasets: MNIST synthetic MI benchmark (Uniform, Normal, Student-t, transformations)
Benchmarks: MI estimation benchmark (40 tasks) high MI benchmark (sparse multinormal with Half-cube and Spiral) MNIST self-consistency tests (independence, data processing, additivity)
Metrics: estimated MI in nats vs ground truth bias self-consistency test results
Baselines: MINE InfoNCE NWJ DoE D-V
Builds on: score-based diffusion models (VP-SDE) MINE InfoNCE NWJ DoE
Closest papers by meaning
- A Benchmark Suite for Evaluating Neural Mutual Information Estimators on Unstructured Datasets (NeurIPS 2024 · similarity 0.88)
- InfoNet: Neural Estimation of Mutual Information without Test-Time Optimization (ICML 2024 · similarity 0.88)
- Mutual Information Estimation via $f$-Divergence and Data Derangements (NeurIPS 2024 · similarity 0.87)
- Information-Theoretic Discrete Diffusion (NeurIPS 2025 · similarity 0.87)
- Neural Mutual Information Estimation with Vector Copulas (NeurIPS 2025 · similarity 0.87)
- Neural Entropy (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.