Research map / Vision-language and multimodal / Cross-modal learning
Classifier-guided Gradient Modulation for Enhanced Multimodal Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CGGM, a classifier-guided gradient modulation method that balances multimodal training by modulating both gradient magnitudes and directions, evaluated on four datasets spanning classification, regression and segmentation.
Problem
Multimodal models over-rely on a single dominant modality during joint training (the greedy learner / modality competition problem), and existing gradient-balancing methods are limited to cross-entropy loss, two modalities, specific optimizers, and only modulate gradient magnitude.
Approach
Attach lightweight modality-specific classifiers whose per-iteration metric improvements define a utilization-based balancing term; use that term to scale each encoder's gradient magnitude and to add a loss that aligns the fusion gradient direction with the weighted average of the classifiers' (unimodal) gradient directions.
Key results
- On UPMC-Food 101, CGGM reaches 92.94 accuracy and 92.90 F1, above the strongest baseline QMF (92.87 / 92.85).
- On BraTS 2021, CGGM averages 73.94 Dice, versus 71.80 for ReconBoost.
- On IEMOCAP, CGGM reaches 75.38 accuracy versus 73.14 for ReconBoost (second best).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CGGM instead of the strongest reported baseline | Multimodal model · UPMC-Food 101 | Acc: 92.87 -> 92.94 | Helped |
| Apply CGGM instead of the strongest reported baseline | Multimodal model · BraTS 2021 | Avg. Dice: 71.80 -> 73.94 | Helped |
| Apply CGGM instead of the strongest reported baseline | Multimodal model · CMU-MOSI | Acc-2: 81.78 -> 82.84 | Helped |
| Apply CGGM instead of the strongest reported baseline | Multimodal model · IEMOCAP | Acc: 73.14 -> 75.38 | Helped |
| Add gradient direction modulation (ablation) | CGGM on IEMOCAP · IEMOCAP | Acc: 72.35 -> 73.74 | Helped |
| Add gradient direction modulation (ablation) | CGGM on IEMOCAP · IEMOCAP | F1: 71.56 -> 73.18 | Helped |
| Add gradient magnitude modulation (ablation) | CGGM on IEMOCAP · IEMOCAP | Acc: 72.41 -> 73.74 | Helped |
| Add gradient magnitude modulation (ablation) | CGGM on IEMOCAP · IEMOCAP | F1: 72.07 -> 73.18 | Helped |
| Increase rho beyond its accuracy-optimal value (ablation) | CGGM on IEMOCAP · IEMOCAP | accuracy | Hurt |
| Set lambda to 0.01 or 0.25 (ablation) | CGGM on IEMOCAP · IEMOCAP | accuracy | Hurt |
Surprising findings
- Modulating only gradient magnitude (rho=1.0) gives most of the gain over baseline on IEMOCAP, while the direction term alone (lambda=0.1) gives a smaller accuracy gain but a larger F1 gain than magnitude alone in the reported ablation.
Limitations
- Requires extra lightweight classifiers during training, which add computational cost.
- Unimodal gradients are approximated by classifier gradients rather than computed by dropping modalities.
- Sensitive to hyperparameters rho and lambda, which must be tuned per dataset.
Experimental setup
Datasets: UPMC-Food 101 CMU-MOSI IEMOCAP BraTS 2021
Benchmarks: UPMC-Food 101 CMU-MOSI IEMOCAP BraTS 2021
Metrics: accuracy F1 ACC-2 ACC-7 MAE Pearson correlation Dice score
Baselines: Baseline (joint training) Modality Random Dropout Modality-specific Learning Rate G-Blending Greedy OGM AGM PMR UMT UME QMF ReconBoost
Builds on: OGM-GE Greedy learner hypothesis MSLR ReconBoost QMF
Closest papers by meaning
- Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportion (NeurIPS 2025 · similarity 0.89)
- Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learning (NeurIPS 2025 · similarity 0.88)
- Facilitating Multimodal Classification via Dynamically Learning Modality Gap (NeurIPS 2024 · similarity 0.87)
- Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.87)
- MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance (ICML 2024 · similarity 0.87)
- Improving Multimodal Learning Balance and Sufficiency through Data Remixing (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.