Research map / Vision-language and multimodal / Cross-modal learning
Quantifying and Enhancing Multi-modal Robustness with Modality Preference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically derives certified robustness bounds for late-fusion multi-modal models, shows modality preference limits robustness, and proposes CRMT, an orthogonal-classifier two-step training procedure that improves robustness to multi-modal and uni-modal attacks.
Problem
Multi-modal models are vulnerable to perturbations such as uni-modal attacks and missing modalities, and the vulnerability is especially pronounced for the preferred modality; existing robust training lacks theoretical explanation of this.
Approach
Derives a certified lower bound on perturbation radius for joint multi-modal models based on uni-modal representation margins, Lipschitz constants and integration factors; shows modality preference shrinks the weaker modality's margin and inflates the preferred modality's vulnerability indicator; proposes orthogonal uni-modal classifiers with a LogSumExp margin regularizer (step 1) and learned mod
Key results
- On Kinetics-Sounds under l2-PGD (eps=0.5), CRMT-AT reaches 0.602 accuracy versus 0.268 for Joint Training and 0.507 for MMAT.
- On KS, CRMT-AT attains 0.762 clean accuracy, the best among compared methods, versus 0.643 for JT.
- Under uni-modal attacks on KS, CRMT-AT gets 0.488 under FGM on the preferred audio modality versus 0.143 for JT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CRMT margin regularization and integration adjustment to joint training | JT · Kinetics-Sounds, FGM | adversarial accuracy: 0.288 -> 0.491 | Helped |
| Add CRMT margin regularization and integration adjustment to joint training | JT · Kinetics-Sounds, ℓ2-PGD | adversarial accuracy: 0.268 -> 0.468 | Helped |
| Add CRMT with adversarial training | MMAT · Kinetics-Sounds, FGM | adversarial accuracy: 0.514 -> 0.608 | Helped |
| Add CRMT with adversarial training | MMAT · Kinetics-Sounds, ℓ2-PGD | adversarial accuracy: 0.507 -> 0.602 | Helped |
| Add CRMT with adversarial training | JT · UCF101, FGM | adversarial accuracy: 0.506 -> 0.515 | Helped |
| Add CRMT with adversarial training | JT · UCF101, ℓ2-PGD | adversarial accuracy: 0.431 -> 0.433 | No clear effect |
| Add CRMT with adversarial training | MMAT · UCF101, FGM | adversarial accuracy: 0.609 -> 0.614 | No clear effect |
| Add CRMT with adversarial training | MMAT · UCF101, ℓ2-PGD | adversarial accuracy: 0.598 -> 0.602 | No clear effect |
| Add CRMT with adversarial training | JT · VGGSound, FGM | adversarial accuracy: 0.157 -> 0.248 | No clear effect |
| Add CRMT with adversarial training | JT · VGGSound, ℓ2-PGD | adversarial accuracy: 0.056 -> 0.124 | No clear effect |
Surprising findings
- Applying the orthogonal framework alone (OJT) does not improve robustness; gains come from the two-step training procedure.
- Gradient Blending, an imbalance-focused method, sometimes outperforms dedicated robustness methods such as MSEFM.
Limitations
- Theory covers only late fusion with two modalities under linear classifiers; extension to intermediate fusion and more modalities is sketched rather than fully validated.
- Certified bound relies on Lipschitz constants that are hard to compute, so the method uses an orthogonal-classifier surrogate.
- Transformer experiments only apply step 1 because cross-modal token interaction violates the analysis assumptions.
Experimental setup
Datasets: Kinetics-Sounds UCF101 VGGSound Food101
Benchmarks: Kinetics-Sounds UCF101 VGGSound Food101
Metrics: clean accuracy adversarial accuracy missing-modality accuracy
Baselines: Joint Training Gradient Blending OGM PMR MSEFM MMAT Mixup RMTL UME-MMA MMTM
Builds on: Gradient Blending On-the-fly Gradient Modulation Prototypical Modal Rebalance Multi-modal adversarial training Multi-modal mixup
Closest papers by meaning
- On the Comparison between Multi-modal and Single-modal Contrastive Learning (NeurIPS 2024 · similarity 0.87)
- Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models? (CVPR 2025 · similarity 0.87)
- Towards the Causal Complete Cause of Multi-Modal Representation Learning (ICML 2025 · similarity 0.87)
- Multimodal Negative Learning (NeurIPS 2025 · similarity 0.86)
- Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness (NeurIPS 2026 · similarity 0.86)
- RedCore: Relative Advantage Aware Cross-Modal Representation Learning for Missing Modalities with Imbalanced Missing Rates (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.