Research map / Probabilistic and scientific modeling / Variational and generative sampling
Neural Mutual Information Estimation with Vector Copulas
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a mutual information estimator (VCE) based on vector copula theory: flows model marginals, and the vector copula is a mixture of vector Gaussian copulas fit by MLE, with MI read off as the mean log copula density.
Problem
Estimating mutual information from limited samples is hard: flexible neural estimators need lots of data, while simple Gaussian-assumption estimators fail on complex non-Gaussian distributions.
Approach
Extends classic copula theory to vector marginals so that MI equals the negative entropy of the vector copula density; marginals are learned separately with flow matching, and the copula is modeled as a mixture of vector Gaussian copulas whose complexity is chosen by NLL cross-validation. The paper also reinterprets critic-based and flow-based estimators as particular copula parameterizations.
Key results
- On the Gaussian plates benchmark at I=7, VCE reached 6.53 ± 0.36 versus 5.91 ± 0.28 for MRE and 3.85 ± 0.22 for N-MIENF.
- On the IMDB Llama-3 13B embeddings at I≈2.1, VCE obtained 2.01 ± 0.04, beating MINE (1.83) and MRE (1.72).
- On the Rectangles benchmark VCE underperformed discriminative methods, e.g. 5.02 ± 0.14 versus 5.45 ± 0.31 for MRE at I=7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use vector-copula estimation (VCE) instead of N-MIENF | MI estimation · Image dataset with known MI | MI estimation accuracy | Helped |
| Use VCE instead of discriminative estimators | MI estimation · Image dataset with known MI, Rectangles | MI estimation accuracy | Hurt |
| Use VCE instead of discriminative estimators | MI estimation · Language-model embeddings | MI estimation accuracy | No clear effect |
| Use VCE instead of N-MIENF | MI estimation · Language-model embeddings | MI estimation accuracy | Helped |
| Use VCE instead of MINE and InfoNCE | MI estimation · Synthetic distributions with high MI or high dimensionality | MI estimation accuracy | Helped |
| Use VCE instead of N-MIENF | MI estimation · Synthetic cases with non-Gaussian dependence, including MoG and 64D t-distribution | MI estimation accuracy | Helped |
| Use VCE instead of MINDE | MI estimation · Spiral transformation and multivariate t-distribution with ν = 1 | MI estimation accuracy | Mixed |
Surprising findings
- Joint learning of marginals and copula sometimes failed badly (about 2 of 10 runs on a 64-D Student-t with nu=1), while separate learning was stable.
Limitations
- Relies on the marginal distributions being well modeled, which is hard for high-dimensional data such as images.
- The model-based Gaussian-mixture copula is less flexible than neural critics, causing weaker results on the Rectangles case.
- Gains are small or absent on low-MI text embeddings relative to discriminative methods.
Experimental setup
Datasets: Gaussian plates and Rectangles image benchmark IMDB movie reviews synthetic MI benchmark distributions (Student-t, MoG, spiral, mixing transformations)
Benchmarks: Gaussian plates image benchmark Rectangles image benchmark IMDB text embeddings with Llama-3 13B and BERT synthetic non-Gaussian distributions
Metrics: estimated mutual information vs ground truth standard deviation over 8 runs negative log-likelihood for model selection
Baselines: MINE InfoNCE MRE N-MIENF MINDE
Builds on: vector copula theory N-MIENF MINE MRE MINDE
Closest papers by meaning
- A Benchmark Suite for Evaluating Neural Mutual Information Estimators on Unstructured Datasets (NeurIPS 2024 · similarity 0.89)
- InfoNet: Neural Estimation of Mutual Information without Test-Time Optimization (ICML 2024 · similarity 0.88)
- Mutual Information Estimation via $f$-Divergence and Data Derangements (NeurIPS 2024 · similarity 0.88)
- MINDE: Mutual Information Neural Diffusion Estimation (ICLR 2024 · similarity 0.87)
- Mutual Information Estimation via Normalizing Flows (NeurIPS 2024 · similarity 0.86)
- FMMI: Flow Matching Mutual Information Estimation (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.