AI Paper Atlas

Research map / Vision-language and multimodal / Cross-modal learning

Probe-Guided Gradient Balancing for Multimodal Learning

Vu Vo, Haytham Fayek, Thuy T Nguyen

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Multimodal fusion with missing modalities · 0 citations (OpenAlex)

Read the paper

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a probe-guided gradient balancing method for jointly trained multimodal networks that boosts underutilized weaker modalities rather than throttling the dominant one.

Problem

Jointly trained multimodal networks underutilize weaker modalities because faster-learning modalities dominate the shared fusion objective and suppress gradients to slower modalities; monitoring signals from the joint loss are entangled with the dominant modality.

Approach

Instead of throttling the dominant modality (the common approach) or directly boosting the weak modality using joint-loss signals (which fail due to entanglement), the method uses probes to obtain modality-specific monitoring signals that guide gradient balancing toward the weaker modality.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add PGGB boosting on top of OGM-GE throttlingOGM-GE · CREMA-D (3-frame)accuracy (%): 69.14 -> 71.45Improved
use PGGB alone (boost only) (ablation)joint training · CREMA-D (3-frame)accuracy (%): 61.59 -> 62.72Improved
use PGGB alone on low-imbalance datasetsjoint training · AVE, Kinetics-Sounds, Twitter15, Sarcasmaccuracy (%): AVE 86.54 -> 87.41; KS 79.05 -> 79.17Improved
add OGM-GE throttlingjoint training · Kinetics-Soundsaccuracy (%): 79.05 -> 77.25Hurt
add PGGB to OGM-GE on text-dominant sentimentOGM-GE · CMU-MOSI and CH-SIMSaccuracy (%): MOSI 72.68 -> 72.60; CH-SIMS 72.47 -> 72.43No clear effect
add PGGB+OGM-GE for segmentationjoint training · BraTS 2021Dice (%): 85.77 -> 86.49Improved
compose PGGB with OGM-GE (alpha 0.75) versus OGM-GE alone (ablation)OGM-GE · CREMA-D (1-frame)accuracy (%): 62.47 -> 62.69; std 1.42 -> 0.22Improved
set boost strength alpha to 0 with probes active (ablation)PGGB+OGM-GE · CREMA-D (3-frame)accuracy (%): 71.45 -> 69.14Hurt
train from scratch instead of pretrained encoders (ablation)PGGB+OGM-GE · AVEaccuracy change vs baseline (pp): +0.69 -> -4.56Hurt
train from scratch instead of pretrained encoders (ablation)PGGB alone · AVEaccuracy change vs baseline (pp): +0.87 / +0.42No clear effect
apply PGGB+OGM-GE with frozen encoders (ablation)joint training · Food101accuracy change vs baseline (pp): -1.64Hurt
use CGGM auxiliary-classifier modulationjoint training · CREMA-D (3-frame)accuracy (%): 61.59 -> 50.22Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy