Research map / Vision-language and multimodal / Cross-modal learning
Are Multimodal Transformers Robust to Missing Modality?
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-task optimization and a learned fusion policy instead of the ViLT baseline | ViLT · MM-IMDb, 100% image + 10% text at test | F1 Macro: 23.1 -> 37.3 | Helped |
| Use multi-task optimization and a learned fusion policy instead of the ViLT baseline | ViLT · UPMC Food-101, 100% image + 10% text at test | Accuracy: 58.4 -> 73.3 | Helped |
| Use multi-task optimization and a learned fusion policy instead of the ViLT baseline | ViLT · Hateful Memes, 100% image + 10% text at test | AUROC: 58.0 -> 59.6 | Helped |
| Use late rather than early fusion | ViLT · UPMC Food-101, 100% image + 30% text at test | Accuracy: 65.9 -> 69.1 | Helped |
| Use late rather than early fusion | ViLT · MM-IMDb, 100% image + 30% text at test | F1 Macro: 31.2 -> 31.0 | No clear effect |
| Use late rather than early fusion | ViLT · Hateful Memes, 100% image + 30% text at test | AUROC: 60.2 -> 57.8 | Hurt |
| Train with missing modalities using the simple missing-modality baseline | ViLT · UPMC Food-101, 100% image + 30% text at train and test | Accuracy: 65.9 -> 44.3 | Hurt |
| Train with missing modalities using the simple missing-modality baseline | ViLT · Hateful Memes, 100% image + 30% text at train and test | AUROC: 60.2 -> 59.7 | Hurt |
| Add the learned fusion policy to multi-task learning (ablation) | ViLT on MM-IMDb · MM-IMDb, 30% text at test | F1 Macro: 31.2 -> 41.8 | Helped |
| Add multi-task learning to the learned fusion policy (ablation) | ViLT on MM-IMDb · MM-IMDb, 30% text at test | F1 Macro: 28.6 -> 41.8 | Helped |
| Add the learned fusion policy to multi-task learning (ablation) | ViLT on MM-IMDb · MM-IMDb, 10% text at test | F1 Macro: 22.6 -> 37.3 | Helped |
| Add multi-task learning to the learned fusion policy (ablation) | ViLT on MM-IMDb · MM-IMDb, 10% text at test | F1 Macro: 17.3 -> 37.3 | Helped |
Closest papers by meaning
- Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities (CVPR 2024 · similarity 0.85)
- RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer (ICML 2025 · similarity 0.84)
- Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness (NeurIPS 2026 · similarity 0.84)
- Quantifying and Enhancing Multi-modal Robustness with Modality Preference (ICLR 2024 · similarity 0.83)
- Probabilistic Conformal Distillation for Enhancing Missing Modality Robustness (NeurIPS 2024 · similarity 0.83)
- Embracing Unimodal Aleatoric Uncertainty for Robust Multimodal Fusion (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.