Research map / Vision-language and multimodal / Cross-modal learning
THE MORE, THE MERRIER: CONTRASTIVE FUSION FOR HIGHER-ORDER MULTIMODAL ALIGNMENT
Stefanos Koutoupis, Michaela Areti Zervou, Konstantinos Kontras, Maarten De Vos, Panagiotis Tsakalides, Grigorios Tsagkatakis
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add fused-modality (2->1) contrastive term to pairwise objective | ConFu · AV-MNIST zero-shot | accuracy: 70.9 -> 71.2 vs Symile | Improved |
| fuse audio and vision instead of using the strongest unimodal model | ConFu · AV-MNIST zero-shot | accuracy: 8% improvement over strongest unimodal baseline | Improved |
| fuse audio with weak-audio visual inputs on VB100 | ConFu · VB100 zero-shot multi-frame | accuracy: 20.69 -> 18.15 (vs best visual-only) | Hurt |
| add audio fusion when both modalities are informative | ConFu · SSW60 zero-shot multi-frame | accuracy: 70.1 -> 71.4 | Improved |
| add feature-level modality masking (ratio 0.3) (ablation) | ConFu · MOSI retrieval | Recall@10: +0.93 to +7.29 across directions | Improved |
| add feature-level modality masking (ablation) | ConFu · MUStARD | accuracy and retrieval | Hurt |
| add feature-level modality masking (ablation) | ConFu · UR-FUNNY | accuracy and retrieval | No clear effect |
| increase lambda from 0.5 to 0.8 (ablation) | ConFu · MOSI mean Recall@10 1->1 | mean Recall@10: 19.5 -> 13.5 | Hurt |
| lambda 0.5 instead of 0.2 or 0.8 (ablation) | ConFu · MUStARD | accuracy: 59.7 / 61.1 -> 64.1 | Improved |
| use ConFu instead of pairwise-only Tri-CLIP under vision noise | ConFu · Vision noise 20dB SNR | accuracy: 35.4 -> 45.4 | Improved |
| embedding dimension 64 needed for ConFu on XOR vs 8 for Symile (ablation) | ConFu · XOR task (p=1) | accuracy | Mixed |
| add third modality fusion to pairwise CLIP | ConFu · synthetic XOR | accuracy: Tri-CLIP ~3% (chance) vs ConFu above chance | Improved |
Closest papers by meaning
- Extending Multi-modal Contrastive Representations (NeurIPS 2024 · similarity 0.91)
- What to align in multimodal contrastive learning? (ICLR 2025 · similarity 0.91)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.90)
- Gramian Multimodal Representation Learning and Alignment (ICLR 2025 · similarity 0.90)
- Aligning Multimodal Representations through an Information Bottleneck (ICML 2025 · similarity 0.90)
- ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
