Research map / Label-efficient and robust learning / Recognition and long-tail
Facilitating Multimodal Classification via Dynamically Learning Modality Gap
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes dynamically integrating unsupervised contrastive modality matching with supervised classification (heuristic decay or bi-level learned weight) to reduce modality imbalance in multimodal classification.
Problem
Modality imbalance in multimodal learning, where dominant modalities are optimized faster and non-dominant modalities are neglected, causing multimodal models to underperform or fall below unimodal ones.
Approach
Attributes modality imbalance to inconsistent difficulty of fitting one-hot category labels across modalities; mixes a cross-modal contrastive (modality matching) loss with the classification loss, weighting it either by a decreasing heuristic schedule or by a bi-level optimization learned alpha.
Key results
- On KineticsSounds, the learning-based variant reaches 72.53% ACC and 78.38% MAP, versus 70.85% ACC for the best baseline ReconBoost.
- On CREMA-D, the learning-based variant reaches 83.62% ACC and 90.06% MAP, versus 79.43% ACC for MLA.
- On VGGSound, the learning-based variant reaches 52.74% ACC and 55.98% MAP, above MLA (51.65% ACC).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use learning-based dynamic integration instead of the strongest baseline | multimodal classification · KineticsSounds | ACC: 70.85% -> 72.53% | Helped |
| Use learning-based dynamic integration instead of the strongest baseline | multimodal classification · CREMA-D | MAP: 85.72% -> 90.06% | Helped |
| Use learning-based dynamic integration instead of the strongest baseline | multimodal classification · Sarcasm | ACC: 84.37% -> 84.97% | Helped |
| Use learning-based dynamic integration instead of the strongest baseline | multimodal classification · Twitter2015 | F1: 69.11% -> 70.57% | Helped |
| Use learning-based dynamic integration instead of the strongest baseline | multimodal classification · NVGesture | F1: 86.32% -> 84.68% | Hurt |
| Use learning-based dynamic integration instead of the strongest baseline | multimodal classification · VGGSound | MAP: 54.74% -> 55.98% | Helped |
| Add contrastive learning (ablation) | multimodal classification · KineticsSounds | MAP: 69.32% -> 71.76% | Helped |
| Add dynamic integration to contrastive learning (ablation) | multimodal classification · KineticsSounds | MAP: 71.76% -> 78.97% | Helped |
| Add contrastive learning (ablation) | multimodal classification · CREMA-D | MAP: 76.07% -> 86.32% | Helped |
| Add dynamic integration to contrastive learning (ablation) | multimodal classification · CREMA-D | MAP: 86.32% -> 90.06% | Helped |
| Add contrastive learning (ablation) | multimodal classification · Sarcasm | F1: 82.43% -> 83.10% | Helped |
| Add dynamic integration to contrastive learning (ablation) | multimodal classification · Sarcasm | F1: 83.10% -> 84.57% | Helped |
Surprising findings
- Fitting one-hot labels produces a larger audio-video performance gap, and a uniform-label loss narrows it while slightly improving accuracy.
- The heuristic decay schedule can be replaced by a third-order polynomial fit to the learned alpha curve, which further improves KineticsSounds accuracy (71.22%).
Limitations
- Root cause of imbalance from label fitting is not analyzed in depth; whether specific categories suit certain modalities is left open.
- The heuristic schedule does not track the learned alpha curve closely, so it is only an approximation.
- Experiments are limited to small-to-medium classification datasets with ResNet/BERT backbones.
Experimental setup
Datasets: KineticsSounds CREMA-D Sarcasm Twitter2015 NVGesture VGGSound
Benchmarks: KineticsSounds CREMA-D Sarcasm Twitter2015 NVGesture VGGSound
Metrics: accuracy mAP macro F1
Baselines: Concat Affine Channel ML-LSTM Sum Weight ETMC MSES G-Blend OGM Greedy DOMFN
Builds on: contrastive learning (CLIP) gradient blending on-the-fly gradient modulation prototypical modality rebalance bi-level optimization approximation (Franceschi et al.)
Closest papers by meaning
- Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportion (NeurIPS 2025 · similarity 0.92)
- Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.89)
- Semi-Supervised Multimodal Classification Through Learning from Modal and Strategic Complementarities (AAAI 2025 · similarity 0.89)
- Continual Multimodal Contrastive Learning (NeurIPS 2025 · similarity 0.89)
- Beyond Modality Fusion: Deep Ensembles for Multimodal Classification (NeurIPS 2026 · similarity 0.88)
- Multimodal Negative Learning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.