Research map / Vision-language and multimodal / Cross-modal learning
RedCore: Relative Advantage Aware Cross-Modal Representation Learning for Missing Modalities with Imbalanced Missing Rates
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RedCore, which combines a VIB-based cross-modal representation learner (with label and available-modality supervision under imperfect data training) with a relative-advantage bi-level supervision regulation to handle missing modalities with imbalanced missing rates.
Problem
Training multimodal models when modalities are missing in training data, using incomplete samples effectively, and handling imbalance when different modalities have different missing rates.
Approach
Learns cross-modal representations for every modality (missing or not) via a variational information bottleneck objective supervised by labels and, for available modalities, MSE to the cross-modal representation; a relative advantage metric based on imputation losses drives a bi-level optimization that adaptively reweights per-modality imputation supervision.
Key results
- On IEMOCAP with same missing rate 0.7 for training, RedCore averages 0.6393 F1 versus 0.5076 for MMIN and 0.4277 for CRA.
- Under imbalanced missing rates (0.8,0.5,0.2), RedCore averages 0.6017 F1 versus 0.4393 for MMIN.
- At missing rate 0.9 on MOSEI, RedCore achieves 0.7463 F1 versus 0.7136 for MMIN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add relative advantage aware supervision regulation (ablation) | RedCore · IEMOCAP, imbalanced missing rates | F1: 0.5696 -> 0.5999; 0.5599 -> 0.6109 | Helped |
| add VIB-based individual-modality label prediction (ablation) | RedCore · IEMOCAP, imbalanced missing rates | F1: 0.5793 -> 0.5999; 0.5708 -> 0.6109 | Helped |
| use RedCore rather than MMIN | IEMOCAP, imbalanced missing rates · IEMOCAP, average over input combinations | F1: 0.4748 -> 0.618; 0.4514 -> 0.6109 | Helped |
| increase missing-rate imbalance from (0.8, 0.5, 0.5) to (0.5, 0.5, 0.8) | MMIN and RedCore · IEMOCAP, average over input combinations | F1: 0.6088 -> 0.4748; 0.6306 -> 0.618 | Mixed |
Surprising findings
- Competing baselines degrade sharply at high missing rates while RedCore holds performance, even though RedCore uses only the incomplete-data (IDT) regime with less data.
Limitations
- Evaluation limited to three emotion/sentiment datasets with three modalities (acoustic, visual, lexical).
- Convergence of the bi-level algorithm and Pareto optimality are not analyzed (left as future work).
- Hyperparameters such as gamma, xi1, xi2 are fixed by hand without a sensitivity study shown.
Experimental setup
Datasets: IEMOCAP CMU-MOSEI MSP-IMPROV
Benchmarks: IEMOCAP CMU-MOSEI MSP-IMPROV
Metrics: weighted F1
Baselines: CRA AMAE MMIN
Builds on: variational information bottleneck MMIN cascaded residual autoencoder
Closest papers by meaning
- Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.89)
- Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness (NeurIPS 2026 · similarity 0.89)
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.88)
- Aligning Multimodal Representations through an Information Bottleneck (ICML 2025 · similarity 0.87)
- Facilitating Multimodal Classification via Dynamically Learning Modality Gap (NeurIPS 2024 · similarity 0.87)
- MIDAS: Misalignment-based Data Augmentation Strategy for Imbalanced Multimodal Learning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.