Research map / Vision-language and multimodal / Cross-modal learning
Extending Multi-modal Contrastive Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Ex-MCR extends leaf multimodal contrastive spaces (CLAP audio-text, ULIP 3D-image) into a frozen base CLIP space without any paired data, yielding a unified audio-image-text-3D representation.
Problem
Learning unified contrastive representations for more than three modalities requires large paired data and expensive training; prior C-MCR cannot chain multiple MCR spaces because it forgets original alignments.
Approach
Freeze a base MCR (CLIP) and map leaf MCRs into it using a decoupled projector (linear layer for intra-MCR gap closing, 2-layer MLP for inter-MCR alignment), trained on modality-centric pseudo-pairs with a dense InfoNCE objective over all modality pairs.
Key results
- Ex-MCR reaches 4.46 mAP on AVE and 11.19 mAP on AudioCaps audio-text retrieval, above C-MCR (4.21, 9.50) without any paired data.
- On ModelNet40 zero-shot 3D classification, Ex-MCR gets 66.53 Acc@1 versus ULIP v2's 73.06 but above ULIP's 60.40 and C-MCR's 64.90 (ULIP v2 is higher on Acc@1 so Ex-MCR is best only on Acc@3/Acc@5).
- Ablation: linear f_l and 2-layer MLP f_m with all three modality-centric data types and dense alignment gives the best average performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Extend CLAP to CLIP with Ex-MCR rather than C-MCR | Audio-image-text retrieval · AVE audio-image retrieval | mAP: 4.21 -> 4.46 | Helped |
| Extend CLAP to CLIP with Ex-MCR rather than C-MCR | Audio-image-text retrieval · VGGSS audio-image retrieval | mAP: 5.95 -> 6.39 | Hurt |
| Extend CLAP to CLIP with Ex-MCR rather than C-MCR | Audio-image-text retrieval · FlickrNet audio-image retrieval | mAP: 4.74 -> 4.94 | Helped |
| Extend CLAP to CLIP with Ex-MCR rather than C-MCR | Audio-text retrieval · AudioCaps audio-text retrieval | mAP: 9.50 -> 11.19 | Helped |
| Extend ULIP to CLIP with Ex-MCR rather than ULIP v2 | 3D-text classification · ModelNet40 | Acc@1: 73.06 -> 66.53 | Hurt |
| Extend ULIP to CLIP with Ex-MCR rather than ULIP v2 | 3D-image retrieval · Objaverse-LVIS | mAP: 11.41 -> 6.23 | Hurt |
| Use all audio-, image-, and text-centric training data instead of any subset (ablation) | Ex-MCR · AVE audio-image retrieval | mAP: 4.05, 4.11, 4.12, 4.17, 3.41, 4.10 -> 4.46 | Helped |
| Use all audio-, image-, and text-centric training data instead of any subset (ablation) | Ex-MCR · AudioCaps audio-text retrieval | mAP: 5.54, 8.39, 9.89, 10.88, 11.09, 11.11 -> 11.19 | Helped |
| Use dense alignment objectives rather than a single pair objective (ablation) | Ex-MCR · AVE audio-image retrieval | mAP: 3.97, 4.00, 4.15, 4.18 -> 4.46 | Helped |
| Use dense alignment objectives rather than a single pair objective (ablation) | Ex-MCR · AudioCaps audio-text retrieval | mAP: 7.49, 7.68, 10.82, 11.30 -> 11.19 | Mixed |
| Use a linear f_l rather than a 1- or 2-layer MLP (ablation) | Ex-MCR · AVE audio-image retrieval | mAP: 4.16, 4.04 -> 4.46 | Helped |
| Use a linear f_l rather than a 1- or 2-layer MLP (ablation) | Ex-MCR · AudioCaps audio-text retrieval | mAP: 10.25, 9.93 -> 11.19 | Helped |
Surprising findings
- Activation layers in the intra-MCR projector hurt modality-gap closing; a plain linear layer works best.
- Aligning text-text (overlapping modality) pairs matters more than directly aligning pseudo audio-visual pairs.
Limitations
- Retrieval on audio-visual benchmarks remains low in absolute mAP (around 4-6).
- 3D-image retrieval is lower than ULIP v2, the leaf space it extends, and Acc@1 on ModelNet40 trails ULIP v2.
Experimental setup
Datasets: CC3M COCO MSRVTT MAD AudioCaps Clotho ImageNet1K AudioSet Objaverse Flickr-SoundNet VGGSS AVE
Benchmarks: Flickr-SoundNet VGGSS AVE AudioCaps COCO ModelNet40 Objaverse-LVIS
Metrics: mAP R@5 Acc@1 Acc@3 Acc@5 R@1
Baselines: C-MCR AudioCLIP WAV2CLIP ULIP ULIP v2 CLIP CLAP
Builds on: C-MCR CLIP CLAP ULIP
Closest papers by meaning
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.91)
- Gramian Multimodal Representation Learning and Alignment (ICLR 2025 · similarity 0.90)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.90)
- Aligning Multimodal Representations through an Information Bottleneck (ICML 2025 · similarity 0.89)
- ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretraining (NeurIPS 2026 · similarity 0.89)
- The "Law'' of the Unconscious Contrastive Learner: Probabilistic Alignment of Unpaired Modalities (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.