Research map / Probabilistic and scientific modeling / Variational and generative sampling
A Benchmark Suite for Evaluating Neural Mutual Information Estimators on Unstructured Datasets
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a benchmark suite that gives known true mutual information for image (MNIST) and text (BERT-embedded IMDB) datasets via same-class sampling and a binary symmetric channel trick, and uses it to evaluate neural MI estimators across seven aspects.
Problem
Neural MI estimators are usually evaluated only on Gaussian datasets with analytic MI, which do not reflect the complexity of real unstructured data like images and text, where true MI is unavailable.
Approach
Use same-class sampling for positive pairs so that MI equals the class entropy H(C), then scale MI to non-integer values with a binary symmetric channel on one view, enabling known ground-truth MI for image and sentence-embedding data.
Key results
- Joint critic gives reliable estimates across estimators and domains, while bilinear critic is notably inaccurate on unstructured data.
- Estimation becomes unreliable beyond d_s of about 4 for Gaussian, 36 for vision, and 64 for NLP.
- Pearson correlation between critic depth and estimation accuracy was near zero (-0.007 Gaussian, 0.059 vision, -0.001 NLP), so larger critics did not help.
- No single MI estimator wins across all three domains; MINE is relatively robust to nuisance, and lower-layer ResNet-50 representations give inaccurate MI estimates.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a deeper critic rather than depth 1 (ablation) | SMILE-inf with joint critic · Gaussian, true MI 2 bits | MSE: 0.125 (depth 1); 0.117 (depth 2); 0.114 (depth 3); 0.114 (depth 4); 0.117 (depth 5) | Mixed |
| Use a deeper critic rather than depth 1 (ablation) | SMILE-inf with joint critic · Images, true MI 2 bits | MSE: 0.194 (depth 1); 0.189 (depth 2); 0.194 (depth 3); 0.195 (depth 4); 0.202 (depth 5) | Hurt |
| Use a deeper critic rather than depth 1 (ablation) | SMILE-inf with joint critic · Sentence embeddings, true MI 2 bits | MSE: 0.173 (depth 1); 0.153 (depth 2); 0.150 (depth 3); 0.151 (depth 4); 0.158 (depth 5) | Mixed |
| Increase the number of information sources | Neural MI estimators · Gaussian, images, and sentence embeddings | estimation accuracy | Mixed |
| Increase representation dimension while holding information sources and true MI fixed | Neural MI estimators on images · Images, d_s=1 and MI=1 | estimated MI: 100; 400; 2500; 10000 | No clear effect |
| Increase nuisance strength | Neural MI estimators on images · Images, true MI 2 bits and larger MI values | estimation accuracy: 0.4; 0.7; 1 | Hurt |
| Use MINE instead of other estimators under maximum nuisance | MI estimation on images · Images, nuisance strength η=1 | estimated MI: η=1 | Mixed |
| Estimate MI from upper rather than lower ResNet-50 layers | MI estimation on image representations · ResNet-50 layers | estimation accuracy | Helped |
| Replace BERT with DistilBERT or RoBERTa embeddings | Neural MI estimators · Sentence embeddings | estimation accuracy | No clear effect |
| Use a joint critic rather than a bilinear critic | Neural MI estimators on unstructured data · Images and sentence embeddings | estimation accuracy | Helped |
| Use a joint critic rather than a separable critic | Neural MI estimators · Unstructured datasets | estimation accuracy | Mixed |
| Use SMILE rather than DV or MINE | MI estimation · Images and sentence embeddings | estimation accuracy | No clear effect |
Surprising findings
- Larger critic capacity does not improve estimation accuracy, contradicting prior assumptions.
- Estimated MI for lower ResNet-50 layers violates the data processing inequality, indicating unreliable estimates at lower layers.
Limitations
- Only MNIST images and BERT-embedded IMDB text are used, so coverage of real-world data is limited.
- Theoretical equality I=H(C) relies on assumptions (e.g., error-free classifier) that do not strictly hold in practice.
- Nuisance is applied only to image datasets.
Experimental setup
Datasets: MNIST CIFAR-10 IMDB
Benchmarks: Gaussian (analytic MI) MNIST-based image MI benchmark IMDB sentence-embedding MI benchmark
Metrics: bias variance mean squared error estimated MI
Baselines: NWJ DV InfoNCE MINE JS SMILE
Builds on: InfoNCE DV NWJ MINE SMILE
Closest papers by meaning
- InfoNet: Neural Estimation of Mutual Information without Test-Time Optimization (ICML 2024 · similarity 0.89)
- Neural Mutual Information Estimation with Vector Copulas (NeurIPS 2025 · similarity 0.89)
- MINDE: Mutual Information Neural Diffusion Estimation (ICLR 2024 · similarity 0.88)
- Mutual Information Estimation via $f$-Divergence and Data Derangements (NeurIPS 2024 · similarity 0.88)
- Connecting Jensen–Shannon and Kullback–Leibler Divergences: A New Bound for Representation Learning (NeurIPS 2025 · similarity 0.86)
- FMMI: Flow Matching Mutual Information Estimation (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.