Research map / Vision-language and multimodal / Cross-modal learning
Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes sustained boosting with adaptive classifier assignment to rebalance classification ability of strong and weak modalities in multimodal learning, with a convergence analysis of the cross-modal loss gap.
Problem
Modality imbalance in multimodal learning, where a strong modality converges faster and the weak modality underperforms, often making multimodal models worse than unimodal ones. Prior work rebalances the learning process rather than explicitly improving the weak modality's classifier.
Approach
Apply gradient-boosting-style residual learning to add multiple shared-encoder classifiers for the weak modality, optimizing classification and residual losses jointly (sustained boosting), and decide when to add a classifier per modality using a confidence-score gap with a dead zone (adaptive classifier assignment).
Key results
- On CREMAD, the method reaches 0.8515 accuracy and 0.9103 MAP, versus 0.8362 / 0.9006 for the next best (LFM).
- On KSounds, accuracy is 0.7263 versus 0.7253 for LFM; on VGGSound, 0.5301 versus 0.5274 for LFM.
- On actor-independent CREMAD, the ablation shows gains from each objective term; the method is reported to be best against SOTA baselines there.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adaptive rather than fixed weak-classifier assignment (ablation) | Ours · CREMAD | accuracy: 0.8091/0.8118 -> 0.8515 | Helped |
| Include all three objectives (classification, all-classifier, and previous-classifier losses) (ablation) | Sustained boosting · CREMAD and KSounds | multimodal accuracy: CREMAD: 0.8333, 0.8320, 0.8360 -> 0.8515; KSounds: 0.6942, 0.7190, 0.7198 -> 0.7263 | Helped |
| Use adaptive classifier assignment instead of fixed assignment (ablation) | Ours · CREMAD | video accuracy: 0.6156/0.6277 -> 0.6828 | Helped |
| Increase the weak video backbone from ResNet18 to ResNet34 in naive MML (ablation) | Naive MML · CREMAD | accuracy: 0.6507 -> 0.6277 | Hurt |
| Use sustained boosting with about 1M additional parameters instead of naive MML with the same ResNet18 architecture (ablation) | Naive MML · CREMAD | accuracy: 0.6507 -> 0.8515 | Helped |
| Use Ours rather than the strongest listed baseline | Multimodal learning · CREMAD, KSounds, VGGSound, Twitter, Sarcasm, and NVGesture | accuracy: CREMAD .8362 -> .8515; KSounds .7253 -> .7263; VGGSound .5274 -> .5301; Twitter .7501 -> .7512; Sarcasm .8497 -> .8510; | Helped |
| Use Ours rather than the strongest listed baseline | Multimodal learning · Twitter | F1: .7057 -> .6962 | Hurt |
| Use Ours rather than the strongest listed baseline | Multimodal learning · CREMAD, VGGSound, KSounds, and Sarcasm | MAP or F1: CREMAD MAP .9006 -> .9103; VGGSound MAP .5694 -> .5826; KSounds MAP .7897 -> .7901; Sarcasm F1 .8457 -> .8458 | Helped |
| Use Ours rather than MLA under test-time missing modalities | Multimodal learning · CREMAD, 20% and 50% missing | accuracy: 0.6935 -> 0.7540; 0.5753 -> 0.6008 | Helped |
Surprising findings
- Using a ResNet34 video backbone in naive MML performs worse than ResNet18 (0.6277 vs 0.6507), which the authors attribute to harder convergence.
Limitations
- Theoretical analysis only covers the convergence of the cross-modal gap function under the boosting step, not the full iterative framework.
- Method is described mainly for classifiers per modality; extension to early-fusion settings is only sketched.
Experimental setup
Datasets: CREMAD KSounds NVGesture VGGSound Twitter Sarcasm
Benchmarks: CREMAD KSounds VGGSound Twitter Sarcasm NVGesture
Metrics: accuracy mAP MacroF1
Baselines: Concat Affine ML-LSTM Sum Weight MSES G-blend MSLR OGM PMR AGM MMParato
Builds on: gradient boosting ReconBoost OGM MLA G-Blend
Closest papers by meaning
- Facilitating Multimodal Classification via Dynamically Learning Modality Gap (NeurIPS 2024 · similarity 0.92)
- Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.90)
- Multimodal Negative Learning (NeurIPS 2025 · similarity 0.90)
- Classifier-guided Gradient Modulation for Enhanced Multimodal Learning (NeurIPS 2024 · similarity 0.89)
- MIDAS: Misalignment-based Data Augmentation Strategy for Imbalanced Multimodal Learning (NeurIPS 2025 · similarity 0.89)
- Beyond Modality Fusion: Deep Ensembles for Multimodal Classification (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.