Research map / Label-efficient and robust learning / Semi-supervised learning

Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications

ICLR 2024 · Label-efficient and robust learning › Semi-supervised learning · approach: Self-training theory · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives information-theoretic lower and upper bounds on multimodal synergy (PID) using only labeled unimodal data and unlabeled paired multimodal data, and uses them to estimate multimodal performance, guide data collection, and model selection.

Problem

Quantifying multimodal interactions (redundancy, uniqueness, synergy) in a semi-supervised setting where only labeled unimodal data and unlabeled paired multimodal data are available, so synergy cannot be computed exactly.

Approach

Connects partial information decomposition to the semi-supervised setting: redundancy and uniqueness are computable from unimodal marginals via convex programming; synergy is lower-bounded using shared information (redundancy bound) and unimodal classifier disagreement (uniqueness bound), and upper-bounded via min-entropy coupling approximations.

Key results

• On 100,000 synthetic bitwise distributions, the two lower bounds track true synergy from below (average gap 0.18 for lower bounds) and the upper bound from above (average gap 0.62, 0.24 when S>0.6).
• On MultiBench, estimated performance average for MOSEI is 0.80 vs. actual best multimodal 0.85-0.88 and best unimodal 0.82, estimated without training the fusion model.
• Estimated performance correlates with complex-vs-simple fusion gap at rho = 0.77.

Measured effects reported in the paper

Surprising findings

• The synergy lower bound from uniqueness can be negative (e.g., MIMIC -0.12), and the upper bound is loose on most datasets such as ENRICO.
• Computing the min-entropy upper bound is NP-hard in general, so only approximations are used.

Limitations

• Estimators only approximate true interactions due to clustering discretization and unimodal model errors.
• Upper bound is loose on several datasets, and ENRICO with all interactions is hard to disentangle.
• Ground-truth data generating process unknown for real datasets, so validation relies on proxies like downstream performance.

Experimental setup

Datasets: MOSI MOSEI MUStARD UR-FUNNY MIMIC ENRICO IRFL NYCaps VQA ScienceQA synthetic bitwise distributions

Benchmarks: MOSI MOSEI MUStARD UR-FUNNY MIMIC ENRICO NYCaps IRFL VQA ScienceQA

Metrics: synergy bound gap accuracy

Baselines: simple fusion (ensembling) bilinear pooling tensor fusion cross-modal self-attention multiplicative interactions

Builds on: partial information decomposition min-entropy coupling Fano's inequality

Closest papers by meaning

• Efficient Quantification of Multimodal Interaction at Sample Level (ICML 2025 · similarity 0.89)
• What to align in multimodal contrastive learning? (ICLR 2025 · similarity 0.88)
• $\texttt{I$^2$MoE}$: Interpretable Multimodal Interaction-aware Mixture-of-Experts (ICML 2025 · similarity 0.87)
• Semi-Supervised Multimodal Classification Through Learning from Modal and Strategic Complementarities (AAAI 2025 · similarity 0.87)
• Enhancing Multimodal Cooperation via Sample-level Modality Valuation (CVPR 2024 · similarity 0.86)
• CyIN: Cyclic Informative Latent Space for Bridging Complete and Incomplete Multimodal Learning (NeurIPS 2025 · similarity 0.86)