Research map / Vision-language and multimodal / Cross-modal learning
Multimodal Representation Learning by Alternating Unimodal Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MLA, which trains multimodal models by alternating single-modality updates with a shared head protected by gradient orthogonalization, and fuses predictions at test time by entropy-based uncertainty weights.
Problem
Multimodal learning suffers from "modality laziness": dominant modalities dominate joint optimization, leaving subordinate modalities underfit, and missing modalities worsen this.
Approach
Decompose joint multimodal training into alternating unimodal steps (one modality per iteration) with a shared head whose gradients are projected via a recursive-least-squares matrix to avoid forgetting earlier modalities; at inference, fuse per-modality predictions weighted by inverse predictive entropy.
Key results
- On CREMA-D, MLA reaches 79.70% multimodal accuracy versus 68.14% for OGM-GE and 63.71% for QMF.
- On Food-101, MLA achieves 93.33% multimodal accuracy, above QMF's 92.87%.
- On IEMOCAP with 10% missing rate, MLA gets 75.07% vs 72.95% for late fusion.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use alternating unimodal adaptation, head gradient modification, and dynamic modality fusion | multimodal learning · CREMA-D | accuracy: 68.14 -> 79.70 | Helped |
| use alternating unimodal adaptation, head gradient modification, and dynamic modality fusion | multimodal learning · KS | accuracy: 65.78 -> 71.35 | Helped |
| use alternating unimodal adaptation, head gradient modification, and dynamic modality fusion | multimodal learning · Food-101 | accuracy: 92.87 -> 93.33 | Helped |
| use alternating unimodal adaptation, head gradient modification, and dynamic modality fusion | multimodal learning · MVSA | accuracy: 77.96 -> 79.94 | Helped |
| use alternating unimodal adaptation, head gradient modification, and dynamic modality fusion | multimodal learning · IEMOCAP | accuracy: 76.17 -> 78.92 | Helped |
| use MLA rather than the strongest baseline | IEMOCAP · IEMOCAP with modality missing rates from 10% to 70% | accuracy: 75.07 vs. 73.49; 72.33 vs. 71.33; 68.47 vs. 66.34; 67.00 vs. 63.30; 63.48 vs. 60.54; 59.17 vs. 57.52; 55.89 vs. 55.44 | Helped |
| add head gradient modification (ablation) | MLA · CREMA-D | accuracy: 66.32 -> 74.51 | Helped |
| add test-time dynamic modality fusion (ablation) | MLA · CREMA-D | accuracy: 66.32 -> 72.79 | Helped |
| add both head gradient modification and test-time dynamic modality fusion (ablation) | MLA · CREMA-D | accuracy: 66.32 -> 79.70 | Helped |
Surprising findings
- Late fusion, which trains each modality encoder on its own data, mitigates modality laziness in unimodal performance but underperforms when combining modalities.
Limitations
- Gains over QMF on Food-101 multimodal accuracy are small and the method is evaluated on only five datasets.
- Hyperparameters (including the RLS regularizer alpha) are set via cross-validation, with no sensitivity analysis in the main text.
- Under the CLIP backbone, the image-only accuracy is improved but text-only performance is not.
Experimental setup
Datasets: CREMA-D Kinetic-Sound Food-101 MVSA IEMOCAP
Benchmarks: CREMA-D Kinetic-Sound Food-101 MVSA IEMOCAP
Metrics: test accuracy
Baselines: Sum Concat Late Fusion FiLM BiGated OGM-GE QMF CCA DCCA DCCAE AE CRA
Builds on: OGM-GE QMF recursive least squares gradient projection
Closest papers by meaning
- MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance (ICML 2024 · similarity 0.90)
- Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.89)
- Improving Multimodal Learning Balance and Sufficiency through Data Remixing (ICML 2025 · similarity 0.88)
- Embracing Unimodal Aleatoric Uncertainty for Robust Multimodal Fusion (CVPR 2024 · similarity 0.88)
- Continual Multimodal Contrastive Learning (NeurIPS 2025 · similarity 0.87)
- With Limited Data for Multimodal Alignment, Let the STRUCTURE Guide You (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.