Research map / Vision-language and multimodal / Cross-modal learning
Improving Multimodal Learning Balance and Sufficiency through Data Remixing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Data Remixing, which decouples multimodal samples into single-modality subsets by unimodal KL-divergence separability and reassembles batches so each batch contains one modality, improving audio-visual classification without dataset expansion.
Problem
Joint multimodal training suffers from modality laziness (weak modality under-trained) and modality clash (cross-modal gradient interference), yielding insufficient and imbalanced multimodal learning.
Approach
Sample-level decoupling assigns each sample only its weakest modality (chosen by KL divergence of unimodal predictions from uniform), and batch-level reassembly ensures each batch contains data from a single modality so gradient directions are not mixed, all without expanding the dataset.
Key results
- Concat + Remix reaches 72.72% on CREMA-D vs 64.52% concatenation baseline (+8.20 points).
- Concat + Remix reaches 55.63% on Kinetics-Sounds vs 50.23% baseline; MLA + Remix reaches 57.75%.
- Remix keeps dataset expansion factor at 1 while Resample expands to 1.85-2.01x and training takes roughly 3x longer than Remix for MLA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Data Remixing to Concatenation | Concatenation · CREMAD | accuracy: 64.52% -> 72.72% | Helped |
| Apply Data Remixing to Concatenation | Concatenation · Kinetic-Sounds | accuracy: 50.23% -> 55.63% | Helped |
| Apply Data Remixing to MLA | MLA · CREMAD | accuracy: 68.01 -> 74.19 | Helped |
| Apply Data Remixing after Resample | Resample · Kinetic-Sounds | accuracy: 55.17 -> 58.40 | Helped |
| Use decoupling without reassembling (ablation) | Data Remixing · CREMAD | accuracy: 64.52% -> 69.89% | Helped |
| Use decoupling without reassembling (ablation) | Data Remixing · Kinetic-Sounds | accuracy: 50.23% -> 52.31% | Helped |
| Use reassembling without decoupling or modality masking (ablation) | Data Remixing · CREMAD | accuracy: 64.52% -> 68.68% | Helped |
| Use reassembling without decoupling or modality masking (ablation) | Data Remixing · Kinetic-Sounds | accuracy: 50.23% -> 51.70% | Helped |
| Combine decoupling with batch-level reassembling (ablation) | Data Remixing · CREMAD | accuracy: 69.89% -> 72.72% | Helped |
| Combine decoupling with batch-level reassembling (ablation) | Data Remixing · Kinetic-Sounds | accuracy: 52.31% -> 55.63% | Helped |
| Use unimodal classification heads instead of dropout for unimodal prediction (ablation) | Data Remixing with Concatenation · CREMAD | accuracy: 70.03 -> 72.72 | Helped |
| Use unimodal classification heads instead of dropout for unimodal prediction (ablation) | Data Remixing with Concatenation · Kinetic-Sounds | accuracy: 53.89 -> 55.63 | Helped |
Limitations
- Unimodal evaluation and allocation may need refinement when one modality is primarily auxiliary with limited information.
- Training time is still higher than the plain baseline (about 1.65x on CREMA-D).
Experimental setup
Datasets: CREMA-D Kinetics-Sounds
Benchmarks: CREMA-D Kinetics-Sounds
Metrics: accuracy training time dataset expansion factor imbalance ratio
Baselines: Concatenation Summation Decision fusion FiLM Bi-Gated OGM-GE G-Blend Greedy PMR MLA Resample MMTM
Builds on: OGM-GE Gradient-Blending MLA Resample MMTM
Closest papers by meaning
- Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.88)
- Fusion or Confusion? Multimodal Complexity Is Not All You Need (NeurIPS 2026 · similarity 0.88)
- Multimodal Representation Learning by Alternating Unimodal Adaptation (CVPR 2024 · similarity 0.88)
- Enhancing Multimodal Cooperation via Sample-level Modality Valuation (CVPR 2024 · similarity 0.88)
- MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance (ICML 2024 · similarity 0.88)
- Balancing Multimodal Training Through Game-Theoretic Regularization (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.