Research map / Probabilistic and scientific modeling / Variational and generative sampling

Mutual Information Estimation via Normalizing Flows

NeurIPS 2024 · Probabilistic and scientific modeling › Variational and generative sampling · approach: Normalizing flow training and design · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes mutual information estimators that Gaussianize data with Cartesian products of normalizing flows, yielding closed-form MI (N-MIENF and tridiagonal-N-MIENF) with non-asymptotic error bounds and consistency guarantees.

Problem

Estimating mutual information between high-dimensional random vectors is hard; existing neural and generative estimators are costly, biased, or lack error guarantees.

Approach

Train normalizing flows f_X and f_Y so that the joint latent distribution is close to a Gaussian with tridiagonal (canonical-correlation) covariance, making MI available in closed form as -1/2 sum log(1-rho_j^2); MI invariance under injective maps guarantees the estimate transfers to the original data.

Key results

• The 5-NN weighted KL estimator (WKL) on uniform data has RMSE rising from 2.2 at d=2 to 522.4 at d=32, showing it degrades rapidly with dimension.
• The proposed N-MIENF and tridiag-N-MIENF yield nearly identical results with similar bias across the incompressible high-dimensional tests, with reasonable estimates.
• The estimator gives closed-form MI -1/2 sum log(1-rho_j^2) with only d' = min(d_X, d_Y) extra parameters, and asymptotic variance O(d^2/N) (O(d/N) for Gaussian data).

Measured effects reported in the paper

Surprising findings

• Authors note that the independency, self-consistency, and data-processing tests cannot easily detect bias of the estimator toward binormalizable data.
• The tractable KL-based error bound is trivial (zero) in the Gaussian-based setup.

Limitations

• Consistency proven only for distributions equivalent to a Gaussian via a Cartesian product of diffeomorphisms.
• Tractable KL-based error bound is trivial in the Gaussian-based setup.
• General method depends on the choice of latent PDF family Q, trading off bias and cost.

Experimental setup

Datasets: synthetic correlated Gaussian mapped to images of geometric shapes synthetic uniform, smoothed uniform, and multivariate Student distributions with known MI

Benchmarks: synthetic high-dimensional images (16x16, 32x32 Gaussians and rectangles) incompressible high-dimensional uniform, smoothed uniform, and Student data

Metrics: estimated MI vs ground truth MI RMSE 99.9% confidence intervals

Baselines: MINE NWJ Nishiyama estimator KSG WKL (5-NN weighted Kozachenko-Leonenko with CNN autoencoder)

Builds on: normalizing flows GLOW Real NVP MINE KSG

Closest papers by meaning

• FMMI: Flow Matching Mutual Information Estimation (NeurIPS 2026 · similarity 0.90)
• Mutual Information Estimation via $f$-Divergence and Data Derangements (NeurIPS 2024 · similarity 0.87)
• Neural Mutual Information Estimation with Vector Copulas (NeurIPS 2025 · similarity 0.86)
• InfoNet: Neural Estimation of Mutual Information without Test-Time Optimization (ICML 2024 · similarity 0.86)
• A Benchmark Suite for Evaluating Neural Mutual Information Estimators on Unstructured Datasets (NeurIPS 2024 · similarity 0.85)
• MINDE: Mutual Information Neural Diffusion Estimation (ICLR 2024 · similarity 0.84)