Research map / Vision-language and multimodal / Cross-modal learning
Jointly Modeling Inter- & Intra-Modality Dependencies for Multi-modal Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes I2M2, a product-of-experts framework that combines unimodal experts (intra-modality) with a joint multimodal predictor (inter-modality) derived from a generative view of multimodal data, and evaluates it on healthcare and vision-language benchmarks.
Problem
Supervised multi-modal learning often fails to outperform uni-modal or ensemble baselines because prior methods capture only inter-modality or only intra-modality dependencies, while the relative importance of these dependencies is unknown a priori.
Approach
Derive from a label-generating graphical model with a selection variable that the posterior factorizes into unimodal predictors times a multimodal predictor, and implement this as a product of experts (log-space additive ensemble) of per-modality classifiers and a joint classifier.
Key results
- On AV-MNIST, I2M2 with LF reaches 72.34% accuracy vs 71.68% for LF inter-modality and 68.63% for intra-modality.
- On VQA-VS, I2M2 reaches 68.63 IID VQA score vs 68.04 for inter-modality, and 48.74 OOD vs 46.04 for inter-modality.
- On NLVR2, I2M2 (85.36%) is comparable to inter-modality modeling (85.29%), while intra-modality reaches about 52-54%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine unimodal predictors with inter-modality fusion | AV-MNIST · AV-MNIST | accuracy: 1% | Helped |
| Combine unimodal predictors with inter-modality modeling | MIMIC-III · Mortality and ICD-9 prediction | accuracy | Helped |
| Add unimodal predictors to inter-modality modeling | NLVR2 · NLVR2 | accuracy | No clear effect |
| Combine unimodal predictors with inter-modality modeling | VQA-VS · VQA-VS IID | VQA score | Helped |
| Combine unimodal predictors with inter-modality modeling | VQA-VS · VQA-VS OOD | VQA score: 5.86% | Helped |
| Add unimodal predictors to inter-modality modeling | fastMRI · fastMRI knee pathologies | AUROC | Mixed |
| Pre-train modality models separately before joint fine-tuning (ablation) | I2M2 on fastMRI · fastMRI ACL, Meniscus, Cartilage, and Others | AUROC | Helped |
Surprising findings
- On fastMRI, inter-modality modeling degraded performance relative to intra-modality modeling, the opposite of the AV-MNIST trend.
- Training unimodal models separately before joint fine-tuning outperformed training jointly from scratch across all datasets.
Limitations
- Model size and compute grow linearly with the number of modalities since separate experts are needed per modality and combination.
- Joint end-to-end training from scratch underperforms staged training, indicating unresolved optimization challenges.
Experimental setup
Datasets: AV-MNIST fastMRI FastMRI+ MIMIC-III NLVR2 VQA-VS
Benchmarks: AV-MNIST fastMRI knee pathologies MIMIC-III NLVR2 VQA-VS
Metrics: accuracy AUROC balanced accuracy VQA score
Baselines: Late fusion (LF) Low-rank multimodal fusion (LRTF) Image-only Audio-only Intra-modality modeling Inter-modality modeling WideResNet-20-3 Root-sum-of-squares (RSS) input
Builds on: late fusion low-rank tensor fusion FIBER
Closest papers by meaning
- Enhancing Multimodal Cooperation via Sample-level Modality Valuation (CVPR 2024 · similarity 0.88)
- MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data (NeurIPS 2026 · similarity 0.88)
- Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness (NeurIPS 2026 · similarity 0.88)
- Modalities Contribute Unequally: Enhancing Medical Multi-modal Learning through Adaptive Modality Token Re-balancing (ICML 2025 · similarity 0.88)
- $\texttt{I$^2$MoE}$: Interpretable Multimodal Interaction-aware Mixture-of-Experts (ICML 2025 · similarity 0.87)
- Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.