Research map / Vision-language and multimodal / Cross-modal learning
Learning Optimal Multimodal Information Bottleneck Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OMIB, a multimodal information bottleneck framework that sets the redundancy weight beta within a theoretically derived bound and dynamically reweights per-modality regularization, implemented with variational approximation and cross-attention fusion.
Problem
Existing multimodal information bottleneck (MIB) methods set regularization weights ad hoc and ignore imbalanced task-relevant information across modalities, so they may not reach the optimal MIB that keeps all task-relevant and drops superfluous information.
Approach
Derives an upper bound on the redundancy regularization weight beta under which the MIB objective provably achieves the optimal MIB, and sets per-modality regularization weight r adaptively from the ratio of remaining task-relevant information in each modality, computed via a tanh of KL-divergence ratios and implemented as a variational upper bound.
Key results
- On CREMA-D, OMIB reaches 63.6% accuracy vs 61.4% for the best MIB baseline (E-MIB), a 3.6% relative improvement as reported.
- On CMU-MOSI, OMIB achieves Acc7 48.6, Acc2 86.9, F1 87.1, MAE 0.709, Corr 0.802, best on all metrics.
- On anomalous tissue detection (10x-hBC-A to D), OMIB averages 0.754 AUC and 0.737 F1, reported as about 11.4% AUC gain over the best benchmark.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add warm-up training (ablation) | OMIB · CREMA-D | accuracy: 60.3 -> 63.6 | Helped |
| Use CAN cross-attention instead of concatenation (ablation) | OMIB · CREMA-D | accuracy: 61.5 -> 63.6 | Helped |
| Use the OMF block instead of concatenation fusion (ablation) | OMIB · CREMA-D | accuracy: 59.5 -> 63.6 | Helped |
| Use dynamic modality weight r instead of equal regularization weights (ablation) | OMIB · CREMA-D | accuracy: 62.2 -> 63.6 | Helped |
| Use OMIB instead of E-MIB | Multimodal emotion recognition · CREMA-D | accuracy: 61.4 -> 63.6 | Helped |
| Use OMIB instead of E-MIB | Multimodal sentiment analysis · CMU-MOSI | Acc7: 48.6 -> 48.6 | No clear effect |
| Use OMIB instead of E-MIB | Multimodal sentiment analysis · CMU-MOSI | Acc2: 85.3 -> 86.9 | Helped |
| Use OMIB instead of E-MIB | Multimodal sentiment analysis · CMU-MOSI | F1: 85.3 -> 87.1 | Helped |
| Use OMIB instead of E-MIB | Multimodal sentiment analysis · CMU-MOSI | MAE: 0.711 -> 0.709 | Helped |
| Use OMIB instead of E-MIB | Multimodal sentiment analysis · CMU-MOSI | Corr: 0.798 -> 0.802 | Helped |
| Use OMIB instead of the best benchmark on each target dataset | Anomalous tissue detection · 10x-hBC-A | AUC: 0.626 -> 0.728 | Helped |
| Use OMIB instead of the best benchmark on each target dataset | Anomalous tissue detection · 10x-hBC-B | AUC: 0.866 -> 0.903 | Helped |
Surprising findings
- On the tissue detection task, DMIB and several MIB baselines score below chance AUC (e.g., DMIB 0.423 on 10x-hBC-A), indicating the baselines fail in the SVDD setting.
Limitations
- The beta bound relies on mutual information and entropy quantities estimated with MINE, which adds estimation error.
- The theory assumes a specific task-relevant/superfluous information decomposition (a0, a1, a2, b0, b1, b2) that real data may not satisfy.
Experimental setup
Datasets: SIM-I SIM-II SIM-III CREMA-D CMU-MOSI 10x-hNB-A to H 10x-hBC-A to D
Benchmarks: CREMA-D CMU-MOSI 10x-hBC-A to D
Metrics: accuracy Acc7 Acc2 F1 MAE Corr AUC
Baselines: Concat BiGated MISA deep IB MMIB-Cui MMIB-Zhang DMIB E-MIB L-MIB C-MIB
Builds on: Information Bottleneck Variational Information Bottleneck MINE SVDD Cross-attention network
Closest papers by meaning
- Balancing Multimodal Training Through Game-Theoretic Regularization (NeurIPS 2025 · similarity 0.88)
- SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning (NeurIPS 2026 · similarity 0.87)
- Aligning Multimodal Representations through an Information Bottleneck (ICML 2025 · similarity 0.86)
- Narrowing Information Bottleneck Theory for Multimodal Image-Text Representations Interpretability (ICLR 2025 · similarity 0.85)
- Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.85)
- Improving Multimodal Learning Balance and Sufficiency through Data Remixing (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.