Research map / Vision-language and multimodal / Cross-modal learning
Balancing Multimodal Training Through Game-Theoretic Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the Multimodal Competition Regularizer (MCR), a mutual-information-decomposition-based loss with game-theoretic modality balancing and latent-space permutation perturbations to counter modality competition during joint multimodal training.
Problem
Joint multimodal training often underutilizes some modalities because they compete for training resources, so the multimodal model underperforms unimodal ensembles or the best single modality.
Approach
Decomposes joint MI into task-relevant unique (conditional MI, estimated by JSD of latent-permutation perturbed predictions), shared (supervised contrastive loss) and task-irrelevant shared (CEB-style reconstruction penalty) terms, and balances modalities with a min-max game controlled by a strategy parameter k, with greedy (k=-1) as default.
Key results
- MCR reaches 76.1% on CREMA-D with ResNet (V-A), versus 73.3% for the best baseline (Uni-Pre Finetuned).
- MCR achieves 65.0% on Something-Something V2 with Swin-TF, versus 64.0% for Uni-Pre Frozen.
- MCR is top on most settings, but on AVE with Conformer D&R (91.0%) beats MCR (88.8%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply MCR instead of the strongest listed baseline | multimodal training · CREMA-D V-A, ResNet | accuracy: 73.3{ $\pm$1.8} -> 76.1{ $\pm$1.6} | Helped |
| apply MCR instead of the strongest listed baseline | multimodal training · AVE V-A, Conformer | accuracy: 91.0{ $\pm$0.7} -> 88.8{ $\pm$1.0} | Hurt |
| use MCR instead of the strongest listed baseline | multimodal training · UCF, MOSI, MOSEI, CREMA-D, and Something-Something settings where MCR is higher | accuracy | Helped |
| use greedy rather than collaborative strategy (ablation) | MCR · CREMA-D, ResNet | accuracy: 73.4{ $\pm$3.0} -> 76.2{ $\pm$1.7} | Helped |
| use greedy rather than independent strategy (ablation) | MCR · CREMA-D, ResNet | accuracy: 76.0{ $\pm$2.0} -> 76.2{ $\pm$1.7} | No clear effect |
| use greedy rather than collaborative strategy (ablation) | MCR · AVE, ResNet | accuracy: 67.9{ $\pm$2.5} -> 73.4{ $\pm$0.0} | Helped |
| use greedy rather than independent strategy (ablation) | MCR · AVE, ResNet | accuracy: 72.5{ $\pm$1.0} -> 73.4{ $\pm$0.0} | Helped |
| combine MIPD and Con losses rather than using no regularizer components (ablation) | MCR · CREMA-D, ResNet | accuracy: 73.4{ $\pm$2.5} -> 76.2{ $\pm$1.7} | Helped |
| add MIPD to Con-only regularization (ablation) | MCR · CREMA-D, ResNet | accuracy: 73.4{ $\pm$2.1} -> 76.2{ $\pm$1.7} | Helped |
| use MIPD alone rather than no regularizer components (ablation) | MCR · CREMA-D, ResNet | accuracy: 73.4{ $\pm$2.5} -> 74.1{ $\pm$2.9} | No clear effect |
| add CEB to MIPD and Con losses (ablation) | MCR with Conformer · CREMA-D | accuracy: 84.5{ $\pm$0.3} -> 85.7{ $\pm$0.2} | Helped |
| add CEB to MIPD and Con losses (ablation) | MCR with ResNet · CREMA-D | accuracy: 76.2{ $\pm$1.7} -> 75.6{ $\pm$1.9} | Hurt |
Surprising findings
- Most prior multimodal balancing methods fail to beat simple Ensemble or unimodal-pretrained baselines.
- MCR routes information well to unimodal-correct cases but does not improve synergy when all unimodal predictions are wrong, where AGM and MLB do better.
Limitations
- Synergy capture is weaker than AGM and MLB when all unimodal predictions fail.
- Game strategy is fixed to greedy by default; more adaptive per-modality strategies are left to future work.
- Hyperparameter search and many dataset/model configurations make the method sensitive to settings; the AVE Conformer setting is not won.
Experimental setup
Datasets: CREMA-D AVE UCF101 CMU-MOSI CMU-MOSEI Something-Something V2 synthetic Gaussian data
Benchmarks: CREMA-D AVE UCF101 CMU-MOSI CMU-MOSEI Something-Something V2 synthetic dataset
Metrics: accuracy binary accuracy (regression converted)
Baselines: Ensemble Joint Training Multi-Loss Uni-Pre Frozen Uni-Pre Finetuned MSLR MMCosine OGM AGM MLB ReconBoost MMPareto
Builds on: Conditional Entropy Bottleneck OGM-GE MLB AGM MMPareto
Closest papers by meaning
- Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.89)
- SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning (NeurIPS 2026 · similarity 0.89)
- Fusion or Confusion? Multimodal Complexity Is Not All You Need (NeurIPS 2026 · similarity 0.88)
- MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data (NeurIPS 2026 · similarity 0.88)
- Learning Optimal Multimodal Information Bottleneck Representations (ICML 2025 · similarity 0.88)
- Improving Multimodal Learning Balance and Sufficiency through Data Remixing (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.