Research map / Vision-language and multimodal / Cross-modal learning
Gradient-Guided Modality Decoupling for Missing-Modality Robustness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Gradient-guided Modality Decoupling (GMD), which removes conflicting gradient components across modal-incomplete cases, plus a Dynamic Sharing (DS) architecture that switches off parameters of absent modalities, to improve missing-modality robustness on BraTS 2018 and CMU-MOSI/MOSEI.
Problem
Multimodal models trained with missing modalities suffer from modality dominance, where one modality dominates training and other modalities are under-optimized, degrading performance when modalities are missing.
Approach
Uses gradients (inspired by Grad-CAM) from modal-incomplete subsets as indicators of modality dominance, framing it as gradient conflicts; GMD projects out the conflicting components between pairs of gradients with negative cosine similarity, and DS uses modality-specific encoders with a shared backbone and average-pooling fusion so absent modalities are skipped rather than zero-filled.
Key results
- On BraTS 2018 the method reports average DSC of 70.7 for the ablation setting with DS+GMD versus 56.2 baseline; full method ET average 69.7 vs 62.3 for SMU-Net.
- On CMU-MOSI the method reports average accuracy 73.6 vs 72.2 for MMIN (next best) and 70.5 for the baseline.
- On CMU-MOSEI average accuracy 80.4 vs 78.6 for the baseline and 78.2 for MMIN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add GMD to DS (ablation) | DS framework · BraTS, average across missing-modality counts | DICE: 61.5 -> 70.7 | Helped |
| Add DS without GMD (ablation) | BraTS framework without DS or GMD · BraTS, average across missing-modality counts | DICE: 56.2 -> 61.5 | Helped |
| Train GMD using only cases with 1 missing modality rather than all cases (ablation) | GMD · BraTS | DSC: 79.5 -> 74.7 | Hurt |
| Train GMD using only cases with 2 missing modalities rather than all cases (ablation) | GMD · BraTS | DSC: 79.5 -> 78.4 | Hurt |
| Train GMD using only cases with 3 missing modalities rather than all cases (ablation) | GMD · BraTS | DSC: 79.5 -> 75.6 | Hurt |
| Use GMD with DS rather than the strongest listed baseline | BraTS 2018 WT · BraTS 2018 WT, all modality cases | DSC: 86.5 -> 87.9 | Helped |
| Use GMD with DS rather than the strongest listed baseline | BraTS 2018 TC · BraTS 2018 TC, all modality cases | DSC: 77.7 -> 80.4 | Helped |
| Use GMD with DS rather than the strongest listed baseline | BraTS 2018 ET · BraTS 2018 ET, all modality cases | DSC: 62.3 -> 69.7 | Helped |
| Use GMD with DS rather than the strongest listed baseline | CMU-MOSI · CMU-MOSI, all available-modality scenarios | accuracy: 72.2 -> 73.6 | Helped |
| Use GMD with DS rather than the strongest listed baseline | CMU-MOSEI · CMU-MOSEI, all available-modality scenarios | accuracy: 78.6 -> 80.4 | Helped |
Surprising findings
- Existing methods such as ACN and SMU-Net perform worse with multiple modalities than with a single modality, indicating modality dominance.
- Using only missing-1 or missing-3 cases for GMD training gives weaker results than missing-2 cases, suggesting balanced gradient conflicts are most informative.
Limitations
- Complexity grows exponentially with number of modalities if all modal-incomplete cases are used, requiring sampling approximation.
- Evaluated only on BraTS 2018 and two sentiment datasets; theoretical analysis assumes pairwise conflicts and ||G_j|| much larger than ||G_k||.
Experimental setup
Datasets: BraTS 2018 CMU-MOSI CMU-MOSEI
Benchmarks: BraTS 2018 CMU-MOSI CMU-MOSEI
Metrics: DSC accuracy F1
Baselines: U-HeMIS HVED ACN RFNet SMU-Net OGM-GE CPM GMC MMIN GCN
Builds on: Grad-CAM OGM-GE GMC
Closest papers by meaning
- Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness (NeurIPS 2026 · similarity 0.89)
- Towards Robust Multimodal Sentiment Analysis with Incomplete Data (NeurIPS 2024 · similarity 0.89)
- Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learning (NeurIPS 2025 · similarity 0.88)
- Knowledge Bridger: Towards Training-Free Missing Modality Completion (CVPR 2025 · similarity 0.88)
- A Unified Self-Distillation Framework for Multimodal Sentiment Analysis with Uncertain Missing Modalities (AAAI 2024 · similarity 0.87)
- Probabilistic Conformal Distillation for Enhancing Missing Modality Robustness (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.