Research map / Vision-language and multimodal / Cross-modal learning
Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes M-SAM, a modality-aware variant of Sharpness-Aware Minimization that uses Shapley-based per-batch modality contributions to apply the SAM perturbation only to the dominant modality's loss, improving multimodal accuracy on AV-MNIST, CREMA-D, UR-Funny and AVE.
Problem
Dominant modalities overshadow weaker ones in multimodal learning, causing overfitting and sharp minima that limit generalization and balanced modality learning.
Approach
Use Shapley values to identify the dominant modality each iteration, apply the SAM sharpness perturbation only to the dominant modality's loss while adding unperturbed gradients from the other modalities, so the optimizer favors flat minima for the dominant modality and lets others learn freely.
Key results
- On late fusion, M-SAM reaches 74.08% Acc_mm on AV-MNIST and 68.56% on CREMA-D, versus 73.59% and 67.47% for the strongest baseline Recon-Boost.
- On early fusion, M-SAM reaches 74.48% Acc_mm on AV-MNIST and 68.43% on CREMA-D, versus 73.11% and 66.98% for CGGM.
- The paper reports gains of +1.4% to +2.3% over the closest competitors (CGGM and Recon-Boost) in multimodal accuracy in the late fusion setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use M-SAM instead of CGGM | Multimodal learning · AV-MNIST, late fusion | multimodal accuracy: 73.42 -> 74.08 | Helped |
| Use M-SAM instead of Recon-Boost | Multimodal learning · CREMA-D, late fusion | multimodal accuracy: 67.47 -> 68.56 | Helped |
| Use M-SAM instead of Recon-Boost | Multimodal learning · UR-Funny, late fusion | multimodal accuracy: 67.61 -> 68.31 | Helped |
| Use M-SAM instead of Recon-Boost | Multimodal learning · AVE, late fusion | multimodal accuracy: 78.02 -> 79.67 | Helped |
| Use M-SAM instead of CGGM | Multimodal learning · AV-MNIST, early fusion | multimodal accuracy: 73.11 -> 74.48 | Helped |
| Use M-SAM instead of CGGM | Multimodal learning · CREMA-D, early fusion | multimodal accuracy: 66.90 -> 68.43 | Helped |
| Use M-SAM instead of AGM | Multimodal learning · UR-Funny, early fusion | multimodal accuracy: 66.07 -> 67.92 | Helped |
| Use M-SAM instead of CGGM | Multimodal learning · AVE, early fusion | multimodal accuracy: 77.32 -> 78.23 | Helped |
Surprising findings
- M-SAM has lower single-modality accuracies than some baselines despite higher multimodal accuracy, which the authors attribute to features that are informative only through cross-modal interaction.
- Plain SAM underperforms several gradient modulation baselines (e.g., on UR-Funny late fusion Acc_mm 65.95 vs. CGGM 67.43).
Limitations
- Relies on a Shapley-based per-batch estimate of modality contribution, with approximation of the loss decomposition not fully justified.
- Convergence analysis assumes smoothness and decaying schedules, while experiments use step-wise decay, a mismatch the authors acknowledge only briefly.
- Evaluated on only four datasets with small encoder architectures (ResNet18, Transformer).
Experimental setup
Datasets: AV-MNIST CREMA-D UR-Funny AVE
Benchmarks: AV-MNIST CREMA-D UR-Funny AVE
Metrics: multimodal accuracy unimodal accuracy normalized overfitting gap relative improvement over Joint-Train
Baselines: Joint-Train SAM MSES MSLR OGM-GE AGM MM-Pareto CGGM Recon-Boost
Builds on: Sharpness-Aware Minimization AGM OGM-GE MAXOUT fusion
Closest papers by meaning
- Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.90)
- Multimodal Negative Learning (NeurIPS 2025 · similarity 0.88)
- ReconBoost: Boosting Can Achieve Modality Reconcilement (ICML 2024 · similarity 0.88)
- MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data (NeurIPS 2026 · similarity 0.88)
- Cross-modal Representation Flattening for Multi-modal Domain Generalization (NeurIPS 2024 · similarity 0.88)
- Gradient-Guided Modality Decoupling for Missing-Modality Robustness (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.