Research map / Vision-language and multimodal / Cross-modal learning
Multimodal fusion with missing modalities
Learns aligned multimodal representations robust to missing or imbalanced modalities.
208 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Cross-modal learning, keyword cluster “fusion · missing · multi modal”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Leveling off Several papers from 2025–26 still report on 6 tracked benchmarks, but the best result has barely moved since 2024.
52 papers in 2023–24 and 126 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 17.9% to 16.0%.
What looks settled
- CREMA-D (accuracy): best up to 2024 79.82, best in 2025–26 75.71 from 5 papers, at most 0.2% better, if at all.
- CMU-MOSI (F1): best up to 2024 88.6, best in 2025–26 87.1 from 3 papers, at most 0.2% better, if at all.
- CMU-MOSI (Corr): best up to 2024 0.852, best in 2025–26 0.802 from 2 papers, at most 0.2% better, if at all.
- CMU-MOSI (MAE): best up to 2024 0.643, best in 2025–26 0.709 from 2 papers, at most 0.2% better, if at all.
- Food101 (accuracy): best up to 2024 93.33, best in 2025–26 93.46 from 2 papers, at most 0.2% better, if at all.
- MIMIC (accuracy (%)): best up to 2024 78.2, best in 2025–26 69.78 from 3 papers, at most 0.2% better, if at all.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · MDL-CW: A Multimodal Deep Learning Framework With Cross Weights (CVPR 2016)
- 2017 · Missing Modalities Imputation via Cascaded Residual Autoencoder (CVPR 2017 · 241 citations)
- 2018 · Efficient Large-Scale Multi-Modal Classification (AAAI 2018 · 161 citations)
- 2019 · Learning Factorized Multimodal Representations (ICLR 2019 · 234 citations)
- 2020 · What Makes Training Multi-Modal Classification Networks Hard? (CVPR 2020 · 532 citations)
- 2021 · SMIL: Multimodal Learning with Severely Missing Modality (AAAI 2021 · 297 citations)
- 2022 · Are Multimodal Transformers Robust to Missing Modality? (CVPR 2022 · 183 citations)
- 2023 · Multi-Modal Learning With Missing Modality via Shared-Specific Feature Modelling (CVPR 2023 · 210 citations)
- 2024 · Binding Touch to Everything: Learning Unified Multimodal Tactile Representations (CVPR 2024 · 57 citations)
- 2025 · Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisition (CVPR 2025 · 13 citations)
- 2026 · MCMoE: Completing Missing Modalities with Mixture of Experts for Incomplete Multimodal Action Quality Assessment (AAAI 2026 · 6 citations)
Earliest papers here
- MDL-CW: A Multimodal Deep Learning Framework With Cross Weights (CVPR 2016)
- Missing Modalities Imputation via Cascaded Residual Autoencoder (CVPR 2017 · 241 citations)
- Hierarchical Multimodal Metric Learning for Multimodal Classification (CVPR 2017 · 39 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
CREMA-D · accuracy (higher is better)
8 papers, 2023–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 61.8 | PMR: Prototypical Modal Rebalance for Multimodal Learning (CVPR 2023 · 136 citations) |
| 2024 | 79.82 | ReconBoost: Boosting Can Achieve Modality Reconcilement (ICML 2024 · 2 citations) |
CMU-MOSI · F1 (higher is better)
7 papers, 2024–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 88.6 | Neuro-Inspired Information-Theoretic Hierarchical Perception for Multimodal Learning (ICLR 2024 · 5 citations) |
CMU-MOSI · Corr (higher is better)
6 papers, 2024–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 0.852 | Neuro-Inspired Information-Theoretic Hierarchical Perception for Multimodal Learning (ICLR 2024 · 5 citations) |
CMU-MOSI · MAE (lower is better)
5 papers, 2024–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 0.643 | Neuro-Inspired Information-Theoretic Hierarchical Perception for Multimodal Learning (ICLR 2024 · 5 citations) |
Food101 · accuracy (higher is better)
5 papers, 2018–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2018 | 90.8 | Efficient Large-Scale Multi-Modal Classification (AAAI 2018 · 161 citations) |
| 2024 | 93.33 | Multimodal Representation Learning by Alternating Unimodal Adaptation (CVPR 2024 · 54 citations) |
| 2025 | 93.46 | MIDAS: Misalignment-based Data Augmentation Strategy for Imbalanced Multimodal Learning (NeurIPS 2025 · 0 citations) |
MIMIC · accuracy (%) (higher is better)
5 papers, 2021–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2021 | 78.2 | MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (NeurIPS 2021 · 22 citations) |
What has been tried
114 papers here report 1,252 measured changes (a component added, removed or swapped, compared with a baseline): helped 74%, no clear effect 5%, hurt 16%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Train on multiple datasets with sensor-specific tokens and batch sampling rather than the baseline setup | Touch and Go · accuracy (%) | 21.4 -> 52.7 | Helped | Binding Touch to Everything: Learning Unified Multimodal Tactile Representations (CVPR 2024 · 57 citations) |
| add head gradient modification | CREMA-D · accuracy | 66.32 -> 74.51 | Helped | Multimodal Representation Learning by Alternating Unimodal Adaptation (CVPR 2024 · 54 citations) |
| Add SUFA to RMFI | Noisy MVSA-Single, Salt-Pepper ϵ = 10 · Acc | 59.11 ± 1.92 -> 61.04 ± 1.31 | Helped | Embracing Unimodal Aleatoric Uncertainty for Robust Multimodal Fusion (CVPR 2024 · 50 citations) |
| use Laplace rather than Softmax gating | MIMIC-IV 25-PHE · AUROC | 70.54 ± 0.47 -> 71.23 ± 0.53 | Helped | FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion (NeurIPS 2024 · 43 citations) |
| Randomly re-sample all modalities at the same frequency instead of contribution-targeted re-sampling | Kinetics Sounds · Acc | 66.92 -> 64.69 | Hurt | Enhancing Multimodal Cooperation via Sample-level Modality Valuation (CVPR 2024 · 37 citations) |
| use multimodal multitask pretraining instead of unimodal pretraining | iNaturalist-2018 · top-1 accuracy | 74.2 -> 90.3 | Helped | OmniVec2 - A Novel Transformer based Network for Large Scale Multimodal and Multitask Learning (CVPR 2024 · 37 citations) |
| add similarity-based modality alignment instead of temporal CTC alignment | MIntRec · ACC (%) | 73.15 -> 73.62 | Helped | Token-Level Contrastive Learning with Modality-Aware Prompting for Multimodal Intent Recognition (AAAI 2024 · 36 citations) |
| Add GMD to DS | BraTS, average across missing-modality counts · DICE | 61.5 -> 70.7 | Helped | Gradient-Guided Modality Decoupling for Missing-Modality Robustness (AAAI 2024 · 22 citations) |
| Add expert specialization | ADNI, IGC B (full modalities) · ACC | 62.75 -> 66.11 | Helped | Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts (NeurIPS 2024 · 16 citations) |
| add relative advantage aware supervision regulation | IEMOCAP, imbalanced missing rates · F1 | 0.5696 -> 0.5999; 0.5599 -> 0.6109 | Helped | RedCore: Relative Advantage Aware Cross-Modal Representation Learning for Missing Modalities with Imbalanced Missing Rates (AAAI 2024 · 15 citations) |
| extend Joint training from 50 epochs to 100 epochs | CREMA-D · Accuracy | 35.14 -> 35.36 | No clear effect | Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisition (CVPR 2025 · 13 citations) |
| Use multi-channel retrieval rather than cross-modal retrieval | MM-IMDb, 70% text missing · F1-M | 52.37 -> 55.16 | Helped | Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learning (AAAI 2025 · 10 citations) |
What reviewers will expect
Datasets and baselines named in the 62 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CREMA-D · 12 of 62 papers
- CMU-MOSI · 10 of 62 papers
- CMU-MOSEI · 6 of 62 papers
- MM-IMDb · 6 of 62 papers
- MSR-VTT · 5 of 62 papers
- AV-MNIST · 4 of 62 papers
- Kinetics Sounds · 4 of 62 papers
- UR-FUNNY · 4 of 62 papers
Baselines
- CLIP · 7 of 62 papers
- OGM-GE · 6 of 62 papers
- PMR · 5 of 62 papers
- OGM · 5 of 62 papers
- MMPareto · 5 of 62 papers
- MulT · 4 of 62 papers
- AGM · 4 of 62 papers
- LanguageBind · 4 of 62 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 178 papers; from OpenAlex and the papers' affiliations).
- University of Electronic Science and Technology of China · 7 papers
- Tsinghua University · 6 papers
- Zhejiang University · 6 papers
- Tianjin University · 6 papers
- Chinese Academy of Sciences · 6 papers
- Renmin University of China · 5 papers
- Tencent · 5 papers
- The Chinese University of Hong Kong · 4 papers
- Massachusetts Institute of Technology · 4 papers
- University of Pennsylvania · 4 papers
Use it
Download BibTeX of 39 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Multi-modality image fusion (194 papers)
- Medical multimodal fusion (166 papers)
- Multimodal sentiment analysis (106 papers)
- Multi-modal object re-identification (40 papers)
Elsewhere in the topic
- Unimodal-to-multimodal representation transfer (145 papers)
- Probabilistic embeddings for cross-modal retrieval (108 papers)
- Contrastive vision-language alignment (104 papers)
- Audio-visual speech separation (76 papers)
- Audio-visual source localization (67 papers)
- Cross-modal hashing (45 papers)
- Self-supervised audio-visual representation learning (44 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).