Research map / Vision-language and multimodal / Cross-modal learning
Enhancing Multimodal Cooperation via Sample-level Modality Valuation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a Shapley-based sample-level modality valuation metric and two re-sampling methods (sample-level and modality-level) that boost low-contributing modalities, plus a new balanced audio-visual dataset MM-Debiased.
Problem
Multimodal models often fail to jointly use all modalities (imbalanced multimodal learning); existing methods only address dataset-level modality discrepancy and cannot observe fine-grained, sample-level contribution differences.
Approach
Valuate each modality's contribution per sample with a Shapley-value-based metric, then re-sample inputs of the low-contributing modality (frequency inversely tied to its contribution) so it is trained more, improving cooperation at sample level; a cheaper modality-level variant estimates averages on a subset.
Key results
- Sample-level method reaches 66.92% accuracy on Kinetics Sounds vs 62.30% for Concatenation baseline.
- Sample-level method reaches 84.71% on MM-Debiased vs 83.22% for Concatenation, while OGM-GE, Greedy and PMR fall below baseline there.
- Modality-level method achieves comparable gains (66.65% KS, 83.46% UCF-101) at lower cost (about half FLOPs of G-Blending).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add sample-level modality valuation and targeted re-sampling | Concatenation fusion · Kinetics Sounds | Acc: 62.30 -> 66.92 | Helped |
| Add sample-level modality valuation and targeted re-sampling | Concatenation fusion · UCF-101 | Acc: 81.15 -> 83.52 | Helped |
| Add sample-level modality valuation and targeted re-sampling | Concatenation fusion · MM-Debiased | accuracy: 83.22 -> 84.71 | Helped |
| Add modality-level modality valuation and targeted re-sampling | Concatenation fusion · Kinetics Sounds | Acc: 62.30 -> 66.65 | Helped |
| Add modality-level modality valuation and targeted re-sampling | Concatenation fusion · UCF-101 | Acc: 81.15 -> 83.46 | Helped |
| Add modality-level modality valuation and targeted re-sampling | Concatenation fusion · MM-Debiased | accuracy: 83.22 -> 84.31 | Helped |
| Randomly re-sample all modalities at the same frequency instead of contribution-targeted re-sampling (ablation) | Concatenation fusion · Kinetics Sounds | Acc: 66.92 -> 64.69 | Hurt |
| Re-sample the higher-contribution modality instead of the lower-contribution modality (ablation) | Concatenation fusion · Kinetics Sounds | Acc: 66.92 -> 59.03 | Hurt |
Surprising findings
- Existing imbalanced-learning methods (OGM-GE, Greedy, PMR) underperform the plain Concatenation baseline on the balanced MM-Debiased dataset.
- Reversed re-sampling (boosting the high-contribution modality) hurts performance, supporting that valuation-guided targeting matters.
Limitations
- Sample-wise Shapley valuation adds substantial computational cost on large datasets.
- Assumes non-negative marginal contribution of modalities for the theoretical analysis.
- Does not account for natural inherent differences between modalities; uni-modal contributions cannot be made equal.
Experimental setup
Datasets: Kinetics Sounds UCF-101 MM-Debiased
Benchmarks: Kinetics Sounds UCF-101 MM-Debiased
Metrics: accuracy mAP
Baselines: Concatenation Summation Decision fusion FiLM Gated Bayesian DNN Deep MKL OGM-GE G-Blending Greedy PMR AGM
Builds on: Shapley value OGM-GE Greedy G-Blending
Closest papers by meaning
- Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.89)
- Jointly Modeling Inter- & Intra-Modality Dependencies for Multi-modal Learning (NeurIPS 2024 · similarity 0.88)
- Improving Multimodal Learning Balance and Sufficiency through Data Remixing (ICML 2025 · similarity 0.88)
- Modality-Aware SAM: Sharpness-Aware-Minimization Driven Gradient Modulation for Harmonized Multimodal Learning (NeurIPS 2025 · similarity 0.88)
- MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data (NeurIPS 2026 · similarity 0.88)
- Fusion or Confusion? Multimodal Complexity Is Not All You Need (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.