Research map / Vision-language and multimodal / Cross-modal learning

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning

NeurIPS 2026 · Vision-language and multimodal › Cross-modal learning · approach: Multimodal fusion with missing modalities · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SynIB, a loss-level objective that penalizes confident predictions under learned or random feature masking of one modality, to push multimodal models toward synergistic cross-modal signals.

Problem

Multimodal models tend to rely on unimodal or redundant shortcuts during training and fail to learn synergistic cross-modal information that is predictive only when modalities are used jointly.

Approach

Derive a variational surrogate of the mutual information perturbation difference and implement it as a loss: cross-entropy on intact inputs plus a KL-to-uniform penalty on predictions under modality-wise feature masking (learned adversarial mask or random mask), so that confident predictions under masking are penalized.

Key results

• On the synergy subset, learned-mask SynIB improves over the strongest baseline by +3.0 on Hateful Memes, +7.8 on MUStARD, and +3.6 on UR-Funny.
• On CREMA-D-Irony at alpha=0.5, SynIB reaches 21.7 irony F1 versus 17.6 for MCR.
• On Spurious XOR, vanilla fusion collapses to chance once beta >= 0.6 while SynIB with oracle masking stays near 100%.

Surprising findings

• Modality-balancing methods yield inconsistent gains on synergy-subset examples, suggesting rebalancing alone does not induce synergy.
• Vanilla training's synergistic training loss drops to zero while validation rises, with synergy test accuracy at chance (50.1%) despite high total accuracy.

Limitations

• Performance depends on the latent representation: synergy not expressible in encoder features cannot be recovered by the loss.
• Learned mask training can become unstable or collapse in regimes with very small synergy subsets.
• Surrogate is not guaranteed to be a one-sided bound on MIPD; training adds roughly 1+n forward passes per step.

Experimental setup

Datasets: MultiBench UR-Funny MUStARD CMU-MOSI Hateful Memes CREMA-D XOR synthetic

Benchmarks: UR-Funny MUStARD CMU-MOSI Hateful Memes CREMA-D-Irony Spurious XOR PID-Controlled XOR

Metrics: accuracy F1 synergy-subset accuracy

Baselines: vanilla fusion late ensembling D&R MMPareto ReconBoost MCR

Builds on: partial information decomposition information bottleneck neural tangent kernel Gumbel-softmax MCR

Closest papers by meaning

• Balancing Multimodal Training Through Game-Theoretic Regularization (NeurIPS 2025 · similarity 0.89)
• Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning (ICLR 2026 · similarity 0.89)
• InfMasking: Unleashing Synergistic Information by Contrastive Multimodal Interactions (NeurIPS 2025 · similarity 0.88)
• Learning Optimal Multimodal Information Bottleneck Representations (ICML 2025 · similarity 0.87)
• Fusion or Confusion? Multimodal Complexity Is Not All You Need (NeurIPS 2026 · similarity 0.87)
• CyIN: Cyclic Informative Latent Space for Bridging Complete and Incomplete Multimodal Learning (NeurIPS 2025 · similarity 0.87)