Research map / Vision-language and multimodal / Cross-modal learning
InfMasking: Unleashing Synergistic Information by Contrastive Multimodal Interactions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InfMasking, a contrastive multimodal method that aligns unmasked fused representations with infinitely many randomly masked fused views (approximated via a Gaussian lower bound) to better capture synergistic information.
Problem
Existing contrastive multimodal representation methods emphasize redundant and unique information and capture synergy poorly, which hurts tasks where cross-modal synergy is critical.
Approach
Randomly mask most features of each modality before Transformer fusion and maximize mutual information between masked and unmasked fused representations; take the infinite-mask limit and approximate the expectation with a Gaussian lower bound (InfMasking loss) so it is tractable, added on top of CoMM's objective.
Key results
- On Trifeature synergy, InfMasking reaches 77.0% vs CoMM 71.4%, and on MultiBench bimodal average it reaches 67.05 vs CoMM 64.45.
- On MM-IMDb with CLIP backbone, InfMasking gets 62.60 weighted F1 and 55.93 macro F1 versus CoMM's 61.29 and 53.79.
- With three modalities, InfMasking gets 65.6% on UR-FUNNY vs CoMM 64.8%, and matches CoMM at 94.1% on Vision&Touch contact prediction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add infinite masking to the CoMM objective | CoMM on Trifeature · Trifeature synergy | accuracy: 71.4_{\pm3.47} -> \textbf{77.0}_{\pm4.22} | Helped |
| Use InfMasking instead of CoMM | Multibench 2-modality models · MOSI | top-1 accuracy: 63.7_{\pm2.5} -> \textbf{69.0}_{\pm1.2} | Helped |
| Use InfMasking instead of CoMM | Multibench 2-modality models · MIMIC | top-1 accuracy: 66.4_{\pm0.41} -> \textbf{68.1}_{\pm0.42} | Helped |
| Use InfMasking instead of CoMM | Multibench 2-modality models · UR-FUNNY | top-1 accuracy: 63.3_{\pm0.51} -> \textbf{64.3}_{\pm0.9} | Helped |
| Use InfMasking instead of CoMM | Multibench 2-modality models · MUSTARD | top-1 accuracy: 64.4_{\pm1.1} -> \textbf{66.8}_{\pm2.5} | Helped |
| Add the third modality | InfMasking on Vision&Touch · Vision&Touch contact prediction | top-1 accuracy: 88.5_{\pm0.33} -> 94.1_{\pm0.09} | Helped |
| Add the third modality | InfMasking on UR-FUNNY · UR-FUNNY | top-1 accuracy: 64.3_{\pm0.9} -> \textbf{65.6}_{\pm1.15} | Helped |
| Add infinite masking to the CLIP-backbone model | CoMM on MM-IMDb · MM-IMDb | weighted F1: 61.29_{\pm0.73} -> \textbf{62.60}_{\pm0.26} | Helped |
| Add infinite masking to the CLIP-backbone model | CoMM on MM-IMDb · MM-IMDb | macro F1: 53.79_{\pm0.22} -> \textbf{55.93}_{\pm0.19} | Helped |
| Add the masking loss (λ1 = 1) (ablation) | InfMasking loss ablation · Trifeature synergy | accuracy: 71.4_{\pm3.47} -> \textbf{77.0}_{\pm4.22} | Helped |
| Add view diversity from masking instead of using only L loss (ablation) | InfMasking loss ablation · Trifeature synergy | accuracy: 58.7_{\pm7.11} -> \textbf{77.0}_{\pm4.22} | Helped |
| Add the L term (cross-view contrastive loss) (ablation) | InfMasking loss ablation · Trifeature synergy | accuracy: 69.2_{\pm6.20} -> \textbf{77.0}_{\pm4.22} | Helped |
Surprising findings
- CLIP with original public weights (58.9% weighted F1) outperforms CLIP fine-tuned on MM-IMDb (54.59%), suggesting redundant information learning may not benefit complex genre prediction.
Limitations
- Lacks a rigorous theoretical framework to formally characterize synergistic information.
- Infinite masking increases GPU memory usage with each additional masked view.
- Gaussian assumption for masked fused representations is justified only heuristically and via t-SNE visualization.
Experimental setup
Datasets: Trifeature MIMIC MOSI UR-FUNNY MUSTARD Vision&Touch MM-IMDb
Benchmarks: Trifeature MultiBench MIMIC MultiBench MOSI MultiBench UR-FUNNY MultiBench MUSTARD MultiBench Vision&Touch MM-IMDb
Metrics: linear probing accuracy MSE weighted F1 macro F1
Baselines: Cross Cross+Self FactorCL CoMM MAE CMC CLIP SLIP SimCLR
Builds on: CoMM FactorCL InfoNCE Masked Autoencoders SimCLR
Closest papers by meaning
- Aligning Multimodal Representations through an Information Bottleneck (ICML 2025 · similarity 0.88)
- SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning (NeurIPS 2026 · similarity 0.88)
- $\texttt{I$^2$MoE}$: Interpretable Multimodal Interaction-aware Mixture-of-Experts (ICML 2025 · similarity 0.88)
- What to align in multimodal contrastive learning? (ICLR 2025 · similarity 0.88)
- Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.87)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.