Research map / Vision-language and multimodal / Cross-modal learning
The Convergent Representation of Contrastive Vision-Language Models: Geometry, Modality Gap and Shared Space Alignment
Lingjie Yi, Raphael Douady, Chao Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the SSA shared-space alignment loss | CLIP ViT-B/16 (CC12M) · CIFAR-10 zero-shot | Top-1 accuracy: 61.6 -> 71.3 | Improved |
| add the SSA shared-space alignment loss | CLIP ViT-B/16 (CC12M) · CIFAR-100 zero-shot | Top-1 accuracy: 28.1 -> 37.8 | Improved |
| add the SSA shared-space alignment loss | CLIP ViT-B/16 (CC12M) · ImageNet-1K zero-shot | Top-1 accuracy: 31.4 -> 33.3 | Improved |
| add the SSA loss instead of modality-gap reduction by AlignCLIP | CLIP ViT-B/16 (CC12M) · CIFAR-10 zero-shot | Top-1 accuracy: 69.4 -> 71.3 | Improved |
| add the SSA shared-space alignment loss | CLIP ViT-B/16 (CC12M) · MSCOCO image-to-text retrieval | R@1: 31.4 -> 34.5 | Improved |
| add the SSA shared-space alignment loss | CLIP ViT-B/16 (CC12M) · MSCOCO text-to-image retrieval | R@1: 20.5 -> 22.8 | Improved |
| share encoder parameters to reduce modality gap (SharedCLIP) | CLIP ViT-B/16 (CC12M) · CIFAR-10 zero-shot | Top-1 accuracy: 61.6 -> 56.9 | Hurt |
| post-hoc project embeddings onto the shared space to shrink the modality gap | CLIP ViT-B/32 · MSCOCO image-to-text retrieval | R@1: 48.40 -> 39.26 | Hurt |
| post-hoc project embeddings onto the shared space to shrink the modality gap | CLIP ViT-B/32 · MSCOCO text-to-image retrieval | R@1: 29.89 -> 27.26 | Hurt |
| post-hoc normalize the shared-space projections | CLIP ViT-B/32 · MSCOCO image-to-text retrieval | R@1: 48.40 -> 36.70 | Hurt |
| larger shared space alignment across 78 VLMs | 78 pretrained VLMs · MSCOCO | T2I R@1 correlation: Kendall tau 0.733 | Improved |
| larger cone concentration kappa_avg across 78 VLMs | 78 pretrained VLMs · MSCOCO | modality gap correlation: Kendall tau 0.047 | No clear effect |
Closest papers by meaning
- Extending Multi-modal Contrastive Representations (NeurIPS 2024 · similarity 0.92)
- Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment (NeurIPS 2024 · similarity 0.91)
- Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models (ICLR 2025 · similarity 0.91)
- Gramian Multimodal Representation Learning and Alignment (ICLR 2025 · similarity 0.91)
- Post-pre-training for Modality Alignment in Vision-Language Foundation Models (CVPR 2025 · similarity 0.90)
- Connect, Collapse, Corrupt: Learning Cross-Modal Tasks with Uni-Modal Data (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
