Research map / Vision-language and multimodal / Cross-modal learning
Information-Theoretic Decomposition for Multimodal Interaction Learning
Zequn Yang, Yake Wei, Haotian Ni, Zhihao Xu, Di Hu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add variational layers instead of fully connected layers (ablation) | DMIL · CREMA-D and Kinetic-Sounds | accuracy | Improved |
| add consistency decomposition module to intra-modality decomposition (ablation) | DMIL-ID · CREMA-D and Kinetic-Sounds | accuracy | Improved |
| add intra-modality decomposition module to consistency decomposition (ablation) | DMIL-CD · CREMA-D and Kinetic-Sounds | accuracy | Improved |
| replace Joint training with DMIL | audio-visual model · CREMA-D | accuracy: 72.55 -> 77.02 | Improved |
| replace Ensemble with DMIL | audio-visual model · UCF101 (2 modalities) | accuracy: 83.63 -> 85.01 | Improved |
| replace Joint with DMIL for ViT backbone | ViT encoders · Kinetic-Sounds (ViT) | accuracy: 67.81 -> 74.20 | Improved |
| replace Joint with DMIL on synergy-dominated data | multimodal model · VQAv2 | Synergy: 9.48 -> 11.44 | Improved |
| replace Joint with DMIL on synergy-dominated data | multimodal model · CLEVR | Synergy: 7.53 -> 9.69 | Improved |
| extend DMIL to three modalities | DMIL · CMU-MOSEI (trimodal) | accuracy: 80.31 -> 82.13 | Improved |
| use ViT backbone instead of CNN for baselines | Joint · Kinetic-Sounds | accuracy: 85.07 -> 67.81 | Hurt |
| Joint training vs Ensemble on UCF101 | multimodal baseline · UCF101 (2 modalities) | accuracy: 83.63 -> 79.33 | Hurt |
Closest papers by meaning
- Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications (ICLR 2024 · similarity 0.91)
- Efficient Quantification of Multimodal Interaction at Sample Level (ICML 2025 · similarity 0.91)
- InfMasking: Unleashing Synergistic Information by Contrastive Multimodal Interactions (NeurIPS 2025 · similarity 0.89)
- $\texttt{I$^2$MoE}$: Interpretable Multimodal Interaction-aware Mixture-of-Experts (ICML 2025 · similarity 0.89)
- What to align in multimodal contrastive learning? (ICLR 2025 · similarity 0.88)
- Balancing Multimodal Training Through Game-Theoretic Regularization (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
