Research map / Vision-language and multimodal / Cross-modal learning
Asymmetric Reinforcing Against Multi-Modal Representation Bias
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ARM, an asymmetric reinforcement method that uses mutual information and conditional mutual information contribution estimates to narrow modality contribution gaps while preserving dominant modalities in imbalanced multimodal learning.
Problem
Imbalanced multimodal learning, where dominant modalities change across samples and environments, causing weak modalities to be under-learned and dominant modalities to degrade (modality forgetting) when balancing methods only boost weak modalities.
Approach
A mutual-information-based valuation metric (MIV) estimates each modality's marginal contribution (via CMI with interaction information) and joint contribution (lower bound via min over modalities); these drive dynamic feature-level fusion weights, a balanced min-max loss, and sample-level resampling so that all modalities are reinforced without forgetting the dominant one.
Key results
- ARM reaches 66.52% accuracy on Kinetics Sounds, versus 65.33% for the runner-up Sample-valuation.
- ARM reaches 93.36% on UPMC Food-101, versus 91.56% for PMF, and 75.60% on UCF-51, behind MLA (76.01%).
- Adding ARM to Concatenation raises accuracy by 6.91 points on KS, 7.37 on UCF-51, and 10.98 on Food-101.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ARM instead of Sample-val. | Multimodal classification · Kinetics Sounds | accuracy: 65.33 -> 66.52 | Helped |
| Use ARM instead of MLA | Multimodal classification · UCF-51 | accuracy: 76.01 -> 75.60 | Hurt |
| Use ARM instead of PMF | Multimodal classification · Food-101 | accuracy: 91.56 -> 93.36 | Helped |
| Add the joint-contribution loss L_phiMI (ablation) | ARM loss · Kinetics Sounds, UCF-51, Food-101 | accuracy: KS: 63.88 -> 64.20; UCF-51: 70.03 -> 73.56; Food-101: 88.52 -> 91.25 | Helped |
| Add the marginal-contribution loss L_phiCMI (ablation) | ARM loss · Kinetics Sounds, UCF-51, Food-101 | accuracy: KS: 63.88 -> 65.19; UCF-51: 70.03 -> 72.10; Food-101: 88.52 -> 89.78 | Helped |
| Add both L_phiMI and L_phiCMI (ablation) | ARM loss · Kinetics Sounds, UCF-51, Food-101 | accuracy: KS: 63.88 -> 66.52; UCF-51: 70.03 -> 75.60; Food-101: 88.52 -> 93.36 | Helped |
| Add dynamic feature-level fusion (DFF) (ablation) | ARM · Kinetics Sounds, UCF-51, Food-101 | accuracy: KS: 59.61 -> 64.34; UCF-51: 68.23 -> 72.84; Food-101: 82.38 -> 91.29 | Helped |
| Add balanced min-max loss (BMML) (ablation) | ARM · Kinetics Sounds, UCF-51, Food-101 | accuracy: KS: 59.61 -> 63.65; UCF-51: 68.23 -> 72.68; Food-101: 82.38 -> 90.83 | Helped |
| Add dynamic sample-level re-sampling (DSR) (ablation) | ARM · Kinetics Sounds, UCF-51, Food-101 | accuracy: KS: 59.61 -> 63.78; UCF-51: 68.23 -> 71.81; Food-101: 82.38 -> 90.57 | Helped |
| Add DFF, BMML, and DSR together (ablation) | ARM · Kinetics Sounds, UCF-51, Food-101 | accuracy: KS: 59.61 -> 66.52; UCF-51: 68.23 -> 75.60; Food-101: 82.38 -> 93.36 | Helped |
| Use DSR with slope k=-3.0 instead of random re-sampling (ablation) | Dynamic sample-level re-sampling · Kinetics Sounds, UCF-51, Food-101 | accuracy: KS: 60.78 -> 67.41; UCF-51: 68.59 -> 76.83; Food-101: 84.21 -> 93.69 | Helped |
| Vary DSR slope from k=-2.0 to k=-3.0 (ablation) | Dynamic sample-level re-sampling · Kinetics Sounds, UCF-51, Food-101 | accuracy: KS: 66.52 -> 67.41; UCF-51: 75.60 -> 76.83; Food-101: 93.36 -> 93.69 | Mixed |
Surprising findings
- MLA beats ARM on UCF-51 (76.01% vs 75.60%), so ARM is not best on every dataset.
- Resampling frequency sensitivity: the best DSR slope (k=-3.0) gives 67.41% on KS, but gains plateau at the higher sampling rates.
Limitations
- Mutual information estimation for continuous data needs discretization or kernel density estimation, which is complex and may yield different results.
- Does not resolve natural conflicts between modalities, such as RGB-infrared predictions that disagree in foggy conditions.
- Resampling more often increases computational cost, which may limit scalability.
Experimental setup
Datasets: Kinetics Sounds UCF-51 UPMC Food-101
Benchmarks: Kinetics Sounds UCF-51 UPMC Food-101
Metrics: accuracy
Baselines: Concatenation Summation OGM-GE Greedy QMF PMR Sample-valuation Modality-valuation MLA Decision fusion FiLM BiGated
Builds on: OGM-GE Greedy modality learning MLA Sample-valuation Modality-valuation
Closest papers by meaning
- Multimodal Negative Learning (NeurIPS 2025 · similarity 0.91)
- Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportion (NeurIPS 2025 · similarity 0.90)
- Mitigating Pervasive Modality Absence Through Multimodal Generalization and Refinement (AAAI 2025 · similarity 0.90)
- Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learning (NeurIPS 2025 · similarity 0.90)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.89)
- Balancing Multimodal Training Through Game-Theoretic Regularization (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.