Research map / Vision-language and multimodal / Cross-modal learning
Robustness in Multimodal Learning under Train-Test Modality Mismatch
Brandon McKinzie, Vaishaal Shankar, Joseph Yitan Cheng, Yinfei Yang, Jonathon Shlens, Alexander T Toshev
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MASD (modality augmented self-distillation) to contrastive pretraining (ablation) | Contrastive learning · AudioSet | Overall robustness R: 20.8 -> 24.1 (vs fine-tune baseline) | Improved |
| Add MASD to MAE pretraining (ablation) | MAE · AudioSet | Overall robustness R: 3.8 -> 15.1 | Improved |
| Add WiseFT weight interpolation on top of MASD (ablation) | Contrastive + MASD · AudioSet | Overall robustness R: 24.1 -> 24.8 (overall); mainly fixes Missing at Test | Mixed |
| Replace downstream fine-tuning on AS-20K with fine-tuning on 2M labeled examples (ablation) | Contrastive · AudioSet | Overall robustness R: 20.8 -> 21.8 (still below MASD 24.1) | Improved |
| Use Kinetics data (instead of AudioSet) as self-distillation data for MASD (ablation) | Contrastive MASD on Kinetics-400 · Kinetics-400 | Overall robustness R: 23.5 (distill-on-AS) -> 26.9 (distill-on-Kinetics) | Improved |
| Add MASD to contrastive learning | Contrastive learning · ImageNet-Captions | Overall robustness R: 66.7 -> 80.8 | Improved |
| Add text as an extra pretraining/training modality | Contrastive + MASD+WiseFT · AudioSet 20K | mAP: 44.2 (AV) -> 49.4 (AVT) | Improved |
| Adding MASD to MAE on AudioSet (ablation) | MAE · AudioSet | Performance P (overall): 28.9 -> 30.6 (Pbest) | Improved |
| Adding MASD to contrastive on AudioSet reduces robustness drop on Transferability and Added at Test setups (ablation) | Contrastive · AudioSet | Transferability robustness: 13.0 -> 24.1 | Improved |
| Use MASD on MAE with Kinetics: distill-on-Kinetics vs fine-tune | MAE · Kinetics-400 | Overall robustness R: 17.1 -> 27.6 | Improved |
Closest papers by meaning
- Fusion or Confusion? Multimodal Complexity Is Not All You Need (NeurIPS 2026 · similarity 0.88)
- Multimodal Negative Learning (NeurIPS 2025 · similarity 0.88)
- Quantifying and Enhancing Multi-modal Robustness with Modality Preference (ICLR 2024 · similarity 0.88)
- Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.88)
- Improving Multimodal Learning Balance and Sufficiency through Data Remixing (ICML 2025 · similarity 0.87)
- Embracing Unimodal Aleatoric Uncertainty for Robust Multimodal Fusion (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
