Research map / Probabilistic and scientific modeling / Variational and generative sampling
Partial Information Decomposition via Normalizing Flows in Latent Gaussian Distributions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves joint Gaussian solutions are optimal for Gaussian PID, gives a faster projected-gradient algorithm (Thin-PID), and extends PID to arbitrary modalities via normalizing-flow latent Gaussian encoders (Flow-PID).
Problem
Partial information decomposition (PID) estimation is computationally expensive and inaccurate for continuous, high-dimensional multimodal data; existing Gaussian PID solvers scale poorly and their joint-Gaussian optimality was an open question.
Approach
Show the joint Gaussian solution is optimal for GPID, reformulate GPID as minimizing a log-determinant ratio over the off-diagonal noise cross-covariance solved by projected gradient descent with an SVD-based spectral-norm projection (Thin-PID), and learn invertible normalizing flows that map each modality-target pair to approximately Gaussian marginals while preserving total MI (Flow-PID).
Key results
- Thin-PID achieves absolute error below 1e-12 on Gaussian examples while Tilde-PID error exceeds 1e-8, with about 10x speedup over Tilde-PID when min dimension exceeds 100.
- On non-Gaussian synthetic examples, Flow-PID matches the exact truth in relative PID values where Tilde-PID on transformed data distorts them (e.g., (100,60,2) dims: Flow-PID R=4.34 vs truth 5.71; Tilde-PID R=1.48).
- Model selection using PID similarity over 10 synthetic datasets selects top-3 models achieving over 96% of best accuracy on new datasets (Flow-PID 96.72% to 100%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Thin-PID instead of Tilde-PID for Gaussian PID | GPID · High-dimensional synthetic Gaussian examples | absolute error: <10^{-12} vs >10^{-8} | Helped |
| Use Thin-PID instead of Tilde-PID for Gaussian PID | GPID · Feature dimensions with min(d_X1, d_X2) > 100 | execution speed: more than 10× | Helped |
| Transform non-Gaussian inputs into latent Gaussian representations with normalizing flows | Flow-PID · Non-Gaussian synthetic examples | PID estimate agreement with truth | Helped |
| Use Flow-PID instead of BATCH | PID estimation · Discrete-target synthetic synergy task | synergy: 0.15 vs 0.30; truth 0.56 | Mixed |
| Use Flow-PID instead of CVX/BATCH | PID estimation · UR-FUNNY (vision, text) | unique information U2: 0.04 vs 1.19 | Helped |
| Use Flow-PID for model selection instead of BATCH | Multimodal model selection · Model selection (Synthetic) | accuracy relative to best-performing model: 99.91% vs 99.76% | Hurt |
| Use Flow-PID for model selection instead of BATCH | Multimodal model selection · Model selection (UR-FUNNY) | accuracy relative to best-performing model: 98.58% vs 96.72% | Hurt |
Limitations
- Latent Gaussian encoders only approximate marginals through bijections, introducing bias when the approximated Gaussian diverges from true marginals.
- Flow-PID accuracy depends on encoder quality and can degrade with complex features or few samples.
- Ground truth for real-world multimodal datasets is unknown, so quantification on them cannot be rigorously validated.
Experimental setup
Datasets: MultiBench AV-MNIST UR-FUNNY MOSI MOSEI MUStARD ENRICO TCGA-BRCA VQA2.0
Benchmarks: MultiBench synthetic GPID examples with ground truth
Metrics: PID values (R, U1, U2, S) absolute error vs ground truth execution time model selection accuracy relative to best model
Baselines: Tilde-PID CVX BATCH MMI-PID
Builds on: Tilde-PID BATCH PID MMI-PID normalizing flows RProp
Closest papers by meaning
- Analytically deriving Partial Information Decomposition for affine systems of stable and convolution-closed distributions (NeurIPS 2024 · similarity 0.91)
- Diffusion PID: Interpreting Diffusion via Partial Information Decomposition (NeurIPS 2024 · similarity 0.85)
- Interpretable Diffusion via Information Decomposition (ICLR 2024 · similarity 0.84)
- Learning Local Neighborhoods of Non-Gaussian Graphical Models (AAAI 2025 · similarity 0.84)
- BGDB: Bernoulli-Gaussian Decision Block with Improved Denoising Diffusion Probabilistic Models (AAAI 2025 · similarity 0.83)
- Annealing in variational inference mitigates mode collapse: a theoretical study on Gaussian mixtures (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.