Research map / Vision-language and multimodal / Human and embodied modeling
Efficient Quantification of Multimodal Interaction at Sample Level
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LSMI, a lightweight sample-wise estimator of redundancy, uniqueness, and synergy between two modalities and a target, built on pointwise information decomposition and KNIFE-based entropy estimation.
Problem
Quantifying multimodal interactions (redundancy, uniqueness, synergy) at the individual sample level for continuous, real-world data, where existing PID methods are distribution-level, expensive, or fail with negative pointwise mutual information.
Approach
Decomposes pointwise mutual information into positive and negative parts (self-information h(x) and conditional h(x|y)), defines pointwise redundancy as the min-based redundancy of each part, and estimates entropies with lightweight unimodal KNIFE-style entropy networks instead of modeling the joint distribution.
Key results
- On XOR, OR and XOR+NOT circuit logic data, LSMI's interaction estimates closely match ground truth (e.g., synergy 0.691 vs GT 0.693 on XOR), comparable to or better than PID-CVX and PID-Batch.
- LSMI runtime on real datasets is substantially lower than PID-Batch, e.g., on Food-101 504.0 s vs 59679.5 s and on UCF-101 678.9 s vs 48576.6 s.
- LSMI redundancy correlates with human judgments with Pearson 0.98 for redundancy and 0.95 for text uniqueness on Food-101.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LSMI instead of PID-Batch for interaction estimation | interaction estimation · KS | time (s): 21928.0 -> 501.5 | Helped |
| use LSMI instead of PID-Batch for interaction estimation | interaction estimation · Food-101 | time (s): 59679.5 -> 504.0 | Helped |
| fine-tune ImageBind on the high-redundancy subset instead of the complete dataset | ImageBind · KS | multimodal accuracy: 0.854 -> 0.877 | Helped |
| fine-tune ImageBind on the low-redundancy subset instead of the complete dataset | ImageBind · CREMA-D | visual accuracy: 0.684 -> 0.702 | Helped |
| fine-tune ImageBind on the high-redundancy subset instead of the complete dataset | ImageBind · KS | visual accuracy: 0.818 -> 0.824 | Helped |
| fine-tune ImageBind on the high-redundancy subset instead of the complete dataset | ImageBind · CREMA-D | multimodal accuracy: 0.795 -> 0.801 | Helped |
| fine-tune ImageBind on the low-redundancy subset instead of the complete dataset | ImageBind · KS | audio accuracy: 0.727 -> 0.729 | No clear effect |
| move fusion earlier, from after 4 unimodal layers to the input (ablation) | Hierarchical Multimodal Transformer · KS | synergy: 0.181 -> 1.445 | Helped |
| move fusion later, from the input to after 4 unimodal layers (ablation) | Hierarchical Multimodal Transformer · KS | redundancy: 1.238 -> 2.335 | Helped |
| replace direct feature distillation with interaction-guided distillation | student model · multimodal distillation | performance | Helped |
| replace weighted ensemble with LSMI-based ensemble | multimodal model ensemble · multiple datasets | performance | Helped |
Surprising findings
- Pointwise mutual information can be negative, so directly applying a pointwise measure breaks the redundancy lattice monotonicity, unlike the averaged mutual information case.
- Early fusion in a hierarchical multimodal transformer yields more synergy, while later fusion yields more redundancy.
Limitations
- Real-world interaction values lack ground truth; validation relies on human scores on a 0-5 scale that are not direct information measures.
- Extension beyond two modalities relies on pairwise analysis since PID does not uniquely extend to more sources.
- Estimates depend on the quality of the trained unimodal and multimodal discriminative models used as proxies for the data distribution.
Experimental setup
Datasets: Food-101 CREMA-D Kinetics-Sounds (KS) UCF-101 CMU-MOSEI UR-Funny synthetic circuit logic data synthetic mixture of Gaussians
Benchmarks: Food-101 CREMA-D KS UCF-101 CMU-MOSEI UR-Funny
Metrics: estimated interaction values R/U1/U2/S runtime (s) classification accuracy Pearson correlation with human judgment
Baselines: PID-CVX PID-Batch Joint learning MMIB Bilinear Additive ensemble Weighted ensemble QMF Mod-drop Alignment Recon
Builds on: Partial Information Decomposition KNIFE PID-Batch PID-CVX ImageBind
Closest papers by meaning
- Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications (ICLR 2024 · similarity 0.89)
- InfMasking: Unleashing Synergistic Information by Contrastive Multimodal Interactions (NeurIPS 2025 · similarity 0.85)
- $\texttt{I$^2$MoE}$: Interpretable Multimodal Interaction-aware Mixture-of-Experts (ICML 2025 · similarity 0.85)
- Balancing Multimodal Training Through Game-Theoretic Regularization (NeurIPS 2025 · similarity 0.84)
- Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approach (ICML 2025 · similarity 0.83)
- Enhancing Multimodal Cooperation via Sample-level Modality Valuation (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.