Research map / Vision-language and multimodal / Human and embodied modeling
Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework
Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard J. Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, Ruslan Salakhutdinov, Louis-Philippe Morency
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Compute PID in polar coordinates instead of Cartesian | CVX PID estimator · Gaussian mixture model | PID redundancy: redundancy is 0 under Polar | Mixed |
| Use CVX versus Batch estimator | PID estimators · Synthetic D_R | redundancy R: 0.16 vs 0.29 (truth 0.58) | No clear effect |
| Increase ratio of z1 in label generation | PID estimates · Synthetic generative | U1: 0.01 -> 0.06 | Improved |
| Use synergistic higher-order fusion (Tensor) rather than additive fusion | Multimodal models · D_S accuracy | accuracy: 0.66 -> 0.74 | Improved |
| Use early fusion (EF) on synergy dataset | EF · D_S accuracy | accuracy: 0.56 | Hurt |
| Use uniqueness-oriented methods (Lower, Rec) vs additive/agree | Multimodal models · D_U | uniqueness U: 0.19 -> 0.55 | Improved |
| Use PID agreement between dataset and model as proxy | PID agreement · 10 synthetic datasets | correlation with accuracy: rho = 0.81 | Improved |
| Missing modality: relate drop to U_i | Multimodal models · Robustness | correlation of performance drop: rho=0.8 | Improved |
| For low U_i, relate S to drop vs R | PID · Robustness (U_i<=0.05) | correlation of performance drop: 0.73 vs 0.01 | Mixed |
| Select models via nearest synthetic dataset by PID | Model selection · Real-world datasets | % of best performance: 95.15% -> 100% | Improved |
| Compare PID with human annotation | CVX/Batch · MultiBench datasets | agreement of dominant interaction: 4 datasets agree | Improved |
| Add unimodal-genomic info on TCGA-LGG | PID · TCGA-LGG | U2: 0.06 | No clear effect |
Closest papers by meaning
- Efficient Quantification of Multimodal Interaction at Sample Level (ICML 2025 · similarity 0.92)
- Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications (ICLR 2024 · similarity 0.90)
- Measuring Cross-Modal Interactions in Multimodal Models (AAAI 2025 · similarity 0.87)
- $\texttt{I$^2$MoE}$: Interpretable Multimodal Interaction-aware Mixture-of-Experts (ICML 2025 · similarity 0.87)
- Partial Information Decomposition via Normalizing Flows in Latent Gaussian Distributions (NeurIPS 2025 · similarity 0.86)
- Balancing Multimodal Training Through Game-Theoretic Regularization (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
