Research map / Vision-language and multimodal / Cross-modal learning
Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InfoReg, which identifies a "prime learning window" in early multimodal training and applies an adaptive unimodal regulation term to slow information-sufficient modalities so information-insufficient ones can acquire more information.
Problem
Multimodal networks often suffer from imbalanced information acquisition across modalities, where information-sufficient modalities dominate early training and suppress weaker modalities, and existing balancing methods ignore the early training period.
Approach
Measures per-modality information acquisition via the trace of the Fisher Information Matrix, detects the early "prime learning window" when this rate is high, and during that window regulates the dominant modality with a weight-change penalty whose strength adapts to the performance gap between modalities.
Key results
- On CREMA-D, InfoReg reaches 71.90% accuracy versus 66.61% for joint training, with video accuracy rising from 35.14% to 49.65%.
- With unimodal loss added, InfoReg* reaches 75.71% on CREMA-D, above MMPareto's 73.08%.
- On CMU-MOSI, InfoReg achieves 62.31% accuracy and 62.03 macro F1, versus 61.09 and 60.74 for joint training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply InfoReg instead of Joint training | multimodal model · CREMA-D | Accuracy: 66.61 -> 71.90 | Helped |
| apply InfoReg instead of Joint training | multimodal model · Kinetics Sounds | Accuracy: 65.67 -> 69.31 | Helped |
| apply InfoReg instead of Joint training | video modality on CREMA-D · CREMA-D | Acc video: 35.14 -> 49.65 | Helped |
| apply InfoReg instead of Joint training | video modality on Kinetics Sounds · Kinetics Sounds | Acc video: 36.01 -> 44.73 | Helped |
| add unimodal loss to InfoReg | multimodal model on CREMA-D · CREMA-D | Accuracy: 71.90 -> 75.71 | Helped |
| add unimodal loss to InfoReg | multimodal model on Kinetics Sounds · Kinetics Sounds | Accuracy: 69.31 -> 72.03 | Helped |
| extend Joint training from 50 epochs to 100 epochs (ablation) | video modality on CREMA-D · CREMA-D | Accuracy: 35.14 -> 35.36 | No clear effect |
| adjust during the prime learning window rather than other periods (ablation) | InfoReg on CREMA-D · CREMA-D | Accuracy: 67.22 -> 71.90 | Helped |
| replace Concat fusion with Gated fusion (ablation) | InfoReg on CREMA-D · CREMA-D | Accuracy: 71.90 -> 69.18 | Hurt |
| replace Concat fusion with SUM fusion (ablation) | InfoReg on CREMA-D · CREMA-D | Accuracy: 71.90 -> 70.12 | Hurt |
| replace Concat fusion with FiLM fusion (ablation) | InfoReg on CREMA-D · CREMA-D | Accuracy: 71.90 -> 70.23 | Hurt |
| set β to 0.9 (ablation) | InfoReg · CREMA-D and Kinetics Sounds | Accuracy | Helped |
Surprising findings
- Simply extending joint training to 100 epochs does not recover the video modality's performance (35.36% vs 35.14%), suggesting information missed in the prime learning window cannot be compensated later.
Limitations
- Evaluated only on small-scale benchmarks with ResNet-18 and a small transformer trained from scratch, without pretrained foundation models.
- Relies on hyperparameters beta and K selected by sensitivity analysis on the test benchmarks.
- The orthogonal-gradient approximation used to simplify the regulation term is empirical, supported only by cosine similarity observations.
Experimental setup
Datasets: CREMA-D Kinetics Sounds CMU-MOSI
Benchmarks: CREMA-D Kinetics Sounds CMU-MOSI
Metrics: accuracy macro F1 per-modality accuracy Tr(Fisher Information)
Baselines: Joint training OGM Greedy PMR AGM G-Blending MMPareto
Builds on: OGM-GE Fisher information based information acquisition analysis MMPareto G-Blending
Closest papers by meaning
- Improving Multimodal Learning Balance and Sufficiency through Data Remixing (ICML 2025 · similarity 0.86)
- Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.85)
- Balancing Multimodal Training Through Game-Theoretic Regularization (NeurIPS 2025 · similarity 0.85)
- MODfinity: Unsupervised Domain Adaptation with Multimodal Information Flow Intertwining (CVPR 2025 · similarity 0.85)
- Fusion or Confusion? Multimodal Complexity Is Not All You Need (NeurIPS 2026 · similarity 0.83)
- Neuro-Inspired Information-Theoretic Hierarchical Perception for Multimodal Learning (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.