Research map / Vision-language and multimodal / Cross-modal learning
A Closer Look at Multimodal Representation Collapse
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives a theory of modality collapse in multimodal fusion via polysemantic neurons and low-rank simplicity bias, shows cross-modal knowledge distillation frees rank bottlenecks, and proposes Explicit Basis Reallocation (EBR) to prevent collapse and handle missing modalities.
Problem
Multimodal fusion models often rely on only a subset of modalities (modality collapse), and the underlying learning-theoretic mechanism has not been understood, which hurts robustness when modalities are missing at test time.
Approach
Modality collapse arises when noisy features of one modality share polysemantic neurons in the fusion head with predictive features of another, and low-rank gradient updates (SGD rank bottleneck) cause this entanglement. Cross-modal KD frees rank bottlenecks in student encoders; EBR explicitly reproduces this via an encoder-decoder head and an adversarial modality discriminator that reallocates ba
Key results
- On MIMIC-IV missing-modality evaluation, EBR reaches 0.8533 AUC-ROC on mortality and 0.7030 on readmission, versus 0.8236 and 0.6781 for MUSE.
- Replacing MUSE's unimodal encoders with EBR variants raises mortality AUC-ROC from 0.9201 to 0.9380 and readmission AUC-ROC from 0.7351 to 0.7597.
- EBR keeps multimodal representation rank higher than the default model beyond a critical upweighting point (beta=4), where vanilla rank drops below the unimodal baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace MUSE unimodal encoders with EBR encoders for missing-modality handling | MUSE · MIMIC-IV Mortality | AUC-ROC: 0.8236 -> 0.8533 | Helped |
| replace MUSE unimodal encoders with EBR encoders for missing-modality handling | MUSE · MIMIC-IV Mortality | AUC-PRC: 0.39.87 -> 0.4277 | Helped |
| replace MUSE unimodal encoders with EBR encoders for missing-modality handling | MUSE · MIMIC-IV Readmission | AUC-ROC: 0.6781 -> 0.7030 | Helped |
| replace MUSE unimodal encoders with EBR encoders for missing-modality handling | MUSE · MIMIC-IV Readmission | AUC-PRC: 0.4185 -> 0.4290 | Helped |
| add knowledge distillation (ablation) | MUSE · MIMIC-IV Mortality | AUC-ROC: 0.9201 -> 0.9350 | Helped |
| add Explicit Basis Reallocation | MUSE · MIMIC-IV Mortality | AUC-ROC: 0.9201 -> 0.9380 | Helped |
| add knowledge distillation (ablation) | MUSE · MIMIC-IV Readmission | AUC-ROC: 0.7351 -> 0.7402 | Helped |
| add Explicit Basis Reallocation | MUSE · MIMIC-IV Readmission | AUC-ROC: 0.7351 -> 0.7597 | Helped |
Surprising findings
- Vanilla multimodal rank collapses below the unimodal baseline around a critical upweighting point, suggesting a phase-transition-like failure.
- Reported theory assumes equal label information per feature, a condition the authors acknowledge may not hold in general.
Limitations
- Theoretical results assume equal conditional cross-entropy reduction across features, left open for unequal cases.
- Empirical validation limited to MIMIC-IV and avMNIST.
- Conjectured saddle-to-convex landscape transformation under EBR is not proven.
Experimental setup
Datasets: MIMIC-IV avMNIST
Benchmarks: MIMIC-IV avMNIST
Metrics: AUC-ROC AUC-PRC
Baselines: CM-AE SMIL MT Grape M3Care ShaSpec MUSE
Builds on: MUSE AGOP / average gradient outer product low-rank bias polysemanticity (superposition) cross-modal knowledge distillation
Closest papers by meaning
- Aligning Multimodal Representations through an Information Bottleneck (ICML 2025 · similarity 0.89)
- An Information Criterion for Controlled Disentanglement of Multimodal Data (ICLR 2025 · similarity 0.88)
- Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness (NeurIPS 2026 · similarity 0.88)
- Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.88)
- When to Align, When to Predict: A Phase Diagram for Multimodal Learning (NeurIPS 2026 · similarity 0.88)
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.