Research map / Vision-language and multimodal / Cross-modal learning
Probe-Guided Gradient Balancing for Multimodal Learning
Vu Vo, Haytham Fayek, Thuy T Nguyen
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a probe-guided gradient balancing method for jointly trained multimodal networks that boosts underutilized weaker modalities rather than throttling the dominant one.
Problem
Jointly trained multimodal networks underutilize weaker modalities because faster-learning modalities dominate the shared fusion objective and suppress gradients to slower modalities; monitoring signals from the joint loss are entangled with the dominant modality.
Approach
Instead of throttling the dominant modality (the common approach) or directly boosting the weak modality using joint-loss signals (which fail due to entanglement), the method uses probes to obtain modality-specific monitoring signals that guide gradient balancing toward the weaker modality.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add PGGB boosting on top of OGM-GE throttling | OGM-GE · CREMA-D (3-frame) | accuracy (%): 69.14 -> 71.45 | Improved |
| use PGGB alone (boost only) (ablation) | joint training · CREMA-D (3-frame) | accuracy (%): 61.59 -> 62.72 | Improved |
| use PGGB alone on low-imbalance datasets | joint training · AVE, Kinetics-Sounds, Twitter15, Sarcasm | accuracy (%): AVE 86.54 -> 87.41; KS 79.05 -> 79.17 | Improved |
| add OGM-GE throttling | joint training · Kinetics-Sounds | accuracy (%): 79.05 -> 77.25 | Hurt |
| add PGGB to OGM-GE on text-dominant sentiment | OGM-GE · CMU-MOSI and CH-SIMS | accuracy (%): MOSI 72.68 -> 72.60; CH-SIMS 72.47 -> 72.43 | No clear effect |
| add PGGB+OGM-GE for segmentation | joint training · BraTS 2021 | Dice (%): 85.77 -> 86.49 | Improved |
| compose PGGB with OGM-GE (alpha 0.75) versus OGM-GE alone (ablation) | OGM-GE · CREMA-D (1-frame) | accuracy (%): 62.47 -> 62.69; std 1.42 -> 0.22 | Improved |
| set boost strength alpha to 0 with probes active (ablation) | PGGB+OGM-GE · CREMA-D (3-frame) | accuracy (%): 71.45 -> 69.14 | Hurt |
| train from scratch instead of pretrained encoders (ablation) | PGGB+OGM-GE · AVE | accuracy change vs baseline (pp): +0.69 -> -4.56 | Hurt |
| train from scratch instead of pretrained encoders (ablation) | PGGB alone · AVE | accuracy change vs baseline (pp): +0.87 / +0.42 | No clear effect |
| apply PGGB+OGM-GE with frozen encoders (ablation) | joint training · Food101 | accuracy change vs baseline (pp): -1.64 | Hurt |
| use CGGM auxiliary-classifier modulation | joint training · CREMA-D (3-frame) | accuracy (%): 61.59 -> 50.22 | Hurt |
Closest papers by meaning
- Classifier-guided Gradient Modulation for Enhanced Multimodal Learning (NeurIPS 2024 · similarity 0.91)
- Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.90)
- Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learning (NeurIPS 2025 · similarity 0.90)
- Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportion (NeurIPS 2025 · similarity 0.89)
- Improving Multimodal Learning Balance and Sufficiency through Data Remixing (ICML 2025 · similarity 0.88)
- Beyond Modality Fusion: Deep Ensembles for Multimodal Classification (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
