Research map / Vision-language and multimodal / Cross-modal learning
What to align in multimodal contrastive learning?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoMM, a contrastive multimodal method that fuses modalities into one representation and maximizes mutual information between augmented versions of that fused representation, capturing redundancy, uniqueness and synergy.
Problem
Cross-modal contrastive learning only captures redundant information shared between modalities, failing to model uniqueness and synergy needed for diverse multimodal tasks.
Approach
Align fused multimodal features across label-preserving multimodal augmentations (plus per-modality masked projections) via InfoNCE, grounded in Partial Information Decomposition, so that redundancy, uniqueness and synergy naturally emerge in a single multimodal space.
Key results
- On MultiBench self-supervised linear probing, CoMM averages 65.22 vs 58.7 for FactorCL (second best) over classification tasks, with gains of 16.3 points on MOSI.
- On MM-IMDb self-supervised setting, CoMM with BLIP-2 backbone reaches 64.75 weighted-F1 and 58.44 macro-F1, outperforming CLIP and SLIP baselines.
- On Trifeature, CoMM is the only model capturing redundancy, uniqueness and synergy (synergy 71.87% vs 50% for Cross and Cross+Self).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CoMM instead of FactorCL | self-supervised multimodal learning · MOSI | accuracy (%): 51.2_{\pm 1.6} -> 67.5_{\pm 1.30} | Helped |
| use CoMM instead of FactorCL | self-supervised multimodal learning · MIMIC | accuracy (%): 67.3_{\pm 0.0} -> 66.4_{\pm 0.32} | Hurt |
| use CoMM instead of FactorCL | self-supervised multimodal learning · Vision&Touch EE | MSE (× 10^-4): 10.82_{\pm 0.56} -> 4.55_{\pm 0.30} | Helped |
| use CoMM instead of FactorCL-SUP | fine-tuned multimodal learning · MIMIC | accuracy (%): 76.8_{\pm 0.0} -> 68.18_{\pm 0.23} | Hurt |
| add a third modality | CoMM · Vision&Touch contact prediction | accuracy (%): 88.1 -> 94.2 | Helped |
| add a third modality | CoMM · UR-FUNNY | accuracy (%): 63.1 -> 64.6 | Helped |
| replace attention-based fusion with concat + linear fusion (ablation) | CoMM · Trifeature synergy | accuracy (%): 71.87_{\pm 2.06} -> 50.0_{\pm 0.0} | Hurt |
| replace attention-based fusion with concat + linear fusion (ablation) | CoMM · Trifeature uniqueness | accuracy (%): 87.83_{\pm 1.55} -> 81.49_{\pm 2.88} | Hurt |
| use only Σᵢ Lᵢ instead of the full CoMM loss (ablation) | CoMM · MM-IMDb | weighted-F1: 61.48_{\pm 0.18} -> 60.71_{\pm 0.17} | Hurt |
| use only L instead of the full CoMM loss (ablation) | CoMM · MM-IMDb | weighted-F1: 61.48_{\pm 0.18} -> 54.94_{\pm 0.50} | Hurt |
| add augmentations to both modalities rather than only modality 1 (ablation) | CoMM · Trifeature synergy | accuracy (%): 50.0_{\pm 0.0} -> 71.87_{\pm 2.06} | Helped |
| use both full loss components L and Σᵢ Lᵢ instead of either alone (ablation) | CoMM · Trifeature synergy | accuracy | Helped |
Surprising findings
- Linear fusion fails to learn synergy (50% chance), indicating non-linear fusion is required.
- CLIP with publicly released weights outperforms CLIP further trained on MM-IMDb, suggesting genre prediction does not benefit from redundant-only learning.
Limitations
- Theoretical PID analysis limited to two modalities; empirical extension to n>2 only.
- Higher computational cost due to computing multimodal augmentations and modality-specific encoders.
- Interpretability: difficult to disentangle contributions of redundancy, uniqueness and synergy in the representation.
Experimental setup
Datasets: Trifeature MultiBench (MIMIC, MOSI, UR-FUNNY, MUsTARD, Vision&Touch) MM-IMDb
Benchmarks: Trifeature MultiBench MM-IMDb
Metrics: top-1 accuracy MSE weighted-f1 macro-f1 linear probing accuracy
Baselines: Cross (CLIP-style) Cross+Self (SLIP) FactorCL SupCon FactorCL-SUP CMC CLIP BLIP-2 SimCLR MFAS ReFNet LLaVA-NeXT
Builds on: SimCLR InfoNCE FactorCL SLIP Partial Information Decomposition
Closest papers by meaning
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.90)
- Continual Multimodal Contrastive Learning (NeurIPS 2025 · similarity 0.90)
- Aligning Multimodal Representations through an Information Bottleneck (ICML 2025 · similarity 0.89)
- MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data (NeurIPS 2026 · similarity 0.89)
- Understanding the Emergence of Multimodal Representation Alignment (ICML 2025 · similarity 0.89)
- Asymmetric Reinforcing Against Multi-Modal Representation Bias (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.