Research map / Probabilistic and scientific modeling / Molecules and proteins
InfoNet: Neural Estimation of Mutual Information without Test-Time Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InfoNet, an attention-based network trained on simulated GMM data to directly output a discretized optimal Donsker-Varadhan discriminant, giving mutual information estimates without test-time optimization.
Problem
Estimating mutual information from sample sequences is slow for real-time use when neural estimators like MINE must be re-optimized for each new distribution, and histogram-based methods are not differentiable.
Approach
Train an attention-based network on many simulated GMM joint distributions so it predicts the optimal Donsker-Varadhan discriminant as a 2D lookup table in a single feed-forward pass, with a copula (empirical CDF) preprocessing step; MI is then obtained by summing over the table.
Key results
- InfoNet runs about 0.011 s on GMM sequences of length 2000 versus 0.991 s for MINE-500 and 0.098 s for KSG-1 (Table 1, GPU/CPU timing).
- On GMM correlation-order prediction across K=1..10 components, InfoNet achieves 99.0-99.8% accuracy while MINE-500 ranges 78.1-98.5%.
- On Gaussian MI-error evaluation, InfoNet's mean error at MI 0.9 is -0.138 versus -0.485 for MINE-500, with lower variance than MINE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace test-time optimization with InfoNet feed-forward estimation | MI estimation on GMM distributions · GMM, sequence lengths 200–5000 | runtime: 0.821 -> 0.010 (sequence length 200); 1.235 -> 0.003 (sequence length 5000) | Helped |
| Use InfoNet instead of MINE-500 | MI estimation on multi-component GMMs · GMM, MI bins 0.0–0.9 | mean error: -0.058 -> 0.004 (MI 0.1); -0.485 -> -0.138 (MI 0.9) | Helped |
| Use InfoNet instead of MINE | Correlation-order prediction · GMM, K=1–10 | order accuracy: 98.5 -> 99.8 (K=1); 78.1 -> 99.5 (K=10) | Helped |
| Add a Gaussian smoothing kernel to the lookup table (ablation) | InfoNet lookup table · Lookup-table visualization | smoothness | Helped |
Surprising findings
- Test-time-optimized MINE with only 10 steps performs near chance (about 50-61% order accuracy), showing sensitivity to optimization budget.
Limitations
- Focuses on low-dimensional (1D/2D) variables; high-dimensional use relies on sliced MI, which estimates a different quantity.
- Training distributions are GMMs, so generalization to other families depends on the GMM prior.
- Paper states code and models will be released but gives no link.
Experimental setup
Datasets: Synthetic Gaussian mixture model (GMM) distributions PointOdyssey
Benchmarks: Gaussian sanity check GMM multi-component MI error correlation order accuracy high-dimensional independence testing PointOdyssey motion segmentation
Metrics: runtime (seconds) mean error error variance order accuracy (%) AUC of ROC precision-recall
Baselines: KSG KDE MINE Energy Distance
Builds on: MINE Donsker-Varadhan representation sliced mutual information Kraskov-Stoegbauer-Grassberger (KSG) estimator
Closest papers by meaning
- A Benchmark Suite for Evaluating Neural Mutual Information Estimators on Unstructured Datasets (NeurIPS 2024 · similarity 0.89)
- Neural Mutual Information Estimation with Vector Copulas (NeurIPS 2025 · similarity 0.88)
- Mutual Information Estimation via $f$-Divergence and Data Derangements (NeurIPS 2024 · similarity 0.88)
- MINDE: Mutual Information Neural Diffusion Estimation (ICLR 2024 · similarity 0.88)
- FMMI: Flow Matching Mutual Information Estimation (NeurIPS 2026 · similarity 0.86)
- Mutual Information Estimation via Normalizing Flows (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.