Research map / Vision-language and multimodal / Cross-modal learning
Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes C^3 (Connect, Collapse, Corrupt), a theory-motivated recipe that removes the modality gap in frozen multi-modal contrastive spaces so decoders trained on uni-modal data can caption or generate images across modalities zero-shot.
Problem
Cross-modal tasks lack paired multi-modal data; using a pre-trained contrastive space (CLIP/ImageBind) to train on uni-modal data fails because image and text embeddings are not interchangeable due to a modality gap and alignment noise.
Approach
Theoretically explains the contrastive embedding geometry (constant modality gap orthogonal to embedding span from dimensional collapse at init plus alignment noise from the loss's stable region), then removes the gap by subtracting per-modality embedding means and adds Gaussian noise during uni-modal training.
Key results
- Image-free zero-shot MS-COCO captioning: C^3 reaches CIDEr 93.3 and BLEU-4 27.7, beating CapDec (CIDEr 91.8, BLEU-4 26.4).
- Language-free text-to-image on MS-COCO: C^3 achieves FID 19.6 and IS 26.0, versus LAFITE's FID 20.9 and IS 24.9.
- With ImageBind embeddings, C^3 gets BLEU-1 74.0 on image captioning, 29.5 on audio captioning (Clotho), and 31.4 on video captioning (MSR-VTT).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add collapse and corruption to the connected model (ablation) | C³ image captioning · MS-COCO image-free zero-shot | BLEU-4: 2.4 -> 27.7 | Helped |
| add collapse to the connected and corrupted model (ablation) | C³ image captioning · MS-COCO image-free zero-shot | CIDEr: 87.6 -> 93.3 | Helped |
| add corruption to the connected and collapsed model (ablation) | C³ image captioning · MS-COCO image-free zero-shot | CIDEr: 25.2 -> 93.3 | Helped |
| add collapse and corruption to the connected model (ablation) | C³ text-to-image generation · MS-COCO language-free zero-shot | FID: 29.8 -> 19.6 | Helped |
| add collapse to the connected and corrupted model (ablation) | C³ text-to-image generation · MS-COCO language-free zero-shot | IS: 25.5 -> 26.0 | Helped |
| add corruption to the connected and collapsed model (ablation) | C³ text-to-image generation · MS-COCO language-free zero-shot | IS: 24.4 -> 26.0 | Helped |
| add collapse and corruption to the connected model (ablation) | C³ captioning with ImageBind · MS-COCO image captioning | BLEU-1: 33.5 -> 74.0 | Helped |
| add collapse and corruption to the connected model (ablation) | C³ captioning with ImageBind · Clotho audio captioning | BLEU-1: 21.8 -> 29.5 | Helped |
| add collapse and corruption to the connected model (ablation) | C³ captioning with ImageBind · MSR-VTT video captioning | BLEU-1: 16.7 -> 31.4 | Helped |
| use C³ instead of CapDec | zero-shot image captioning · MS-COCO image-free zero-shot | CIDEr: 91.8 -> 93.3 | Helped |
| use C³ instead of LAFITE_G | language-free text-to-image generation · MS-COCO language-free zero-shot | FID: 20.9 -> 19.6 | Helped |
Surprising findings
- Adding Gaussian noise alone (CapDec-style) was argued to close the modality gap; the authors state this intuition is inaccurate, since the gap is closed by mean subtraction instead.
- In the randomly initialized CLIP analysis, the effective dimension of text features (230) was much larger than that of image features (25).
Limitations
- Relies on the assumption that the alignment noise can be approximated by isotropic Gaussian noise and the modality gap is constant.
- Evaluation focuses on captioning and text-to-image using specific frozen encoders (CLIP, ImageBind); generalization to other embedding spaces is only partially explored.
Experimental setup
Datasets: MS-COCO Clotho MSR-VTT
Benchmarks: MS-COCO Clotho MSR-VTT
Metrics: BLEU-1 BLEU-4 METEOR ROUGE-L CIDEr SPICE FID IS
Baselines: ZeroCap MAGIC ESPER CLIPRe DeCap WS-ClipCap CapDec DALL-E CogView LAFITE
Builds on: CLIP ImageBind ClipCap CapDec LAFITE
Closest papers by meaning
- Extending Multi-modal Contrastive Representations (NeurIPS 2024 · similarity 0.91)
- Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion (ICLR 2025 · similarity 0.91)
- On the Value of Cross-Modal Misalignment in Multimodal Representation Learning (NeurIPS 2025 · similarity 0.89)
- Accept the Modality Gap: An Exploration in the Hyperbolic Space (CVPR 2024 · similarity 0.89)
- Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness (NeurIPS 2026 · similarity 0.88)
- Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.