Research map / Vision-language and multimodal / CLIP and zero-shot
Do Vision and Language Encoders Represent the World Similarly?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that unaligned vision and language encoders have CKA-similar latent spaces and proposes training-free seeded QAP matching and local-CKA retrieval to align them for caption matching, classification, and cross-lingual retrieval.
Problem
Whether uni-modal vision and language encoders, which represent the same physical world, share a latent-space alignment that can be exploited without training a mapping between them.
Approach
Treats maximizing CKA between image and caption kernels as a seeded quadratic assignment (graph matching) problem, and proposes a local CKA score for retrieval between unaligned encoders, requiring no training of stitching or mapping layers.
Key results
- On NoCaps cross-domain caption matching, DINOv2 with local CKA reaches 58.7% matching accuracy and 61.8% top-5 retrieval, versus relative representations at 46.0% and 46.4%.
- QAP matching with CLIP-V on cross-lingual XTD-10 averages 38.8% matching accuracy versus CLIP cosine 26.4%, with non-Latin languages such as ko and zh improving from near zero to 50%+ retrieval@5 under local CKA.
- On ImageNet-100, ConvNeXt with local CKA reaches 83.3% top-1, compared with 86.1% for CLIP cosine similarity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use QAP instead of CLIP cosine similarity for matching | CLIP-V with an unaligned language encoder · XTD-10 average | matching accuracy: 26.4 -> 38.8 | Helped |
| use Local CKA instead of CLIP cosine similarity for retrieval | CLIP-V with a multilingual sentence transformer · XTD-10 average | Retrieval @ 5: 39.6 -> 57.9 | Helped |
| add CKA rather than use normalized correlation matrices (ablation) | QAP matching · COCO | matching accuracy: 10.1 -> 48.8 | Helped |
| add stretching (ablation) | QAP matching with CKA and no clustering · COCO | matching accuracy: 48.8 -> 57.3 | Helped |
| add clustering (ablation) | QAP matching with stretching and CKA · COCO | matching accuracy: 57.3 -> 65.5 | Helped |
| add clustering (ablation) | Local CKA matching with stretching and CKA · COCO | matching accuracy: 56.7 -> 63.3 | Helped |
| add clustering (ablation) | Local CKA retrieval with stretching and CKA · COCO | Retrieval @ 5: 73.0 -> 77.2 | Helped |
| add stretching (ablation) | Local CKA matching with CKA and no clustering · COCO | matching accuracy: 48.5 -> 56.7 | Helped |
| add stretching (ablation) | Local CKA retrieval with CKA and no clustering · COCO | Retrieval @ 5: 60.2 -> 73.0 | Helped |
| add clustering and stretching (ablation) | QAP matching with CKA · COCO | matching accuracy: 48.8 -> 65.5 | Helped |
| add clustering and stretching (ablation) | Local CKA retrieval · COCO | Retrieval @ 5: 60.2 -> 77.2 | Helped |
Surprising findings
- Self-supervised DINOv2, without any language supervision, reaches CKA and matching accuracy comparable to CLIP-trained encoders.
- ConvNeXt trained on ImageNet classification gives over 14% higher top-1 on ImageNet-100 classification than CLIP and DINOv2 under the proposed method, despite weaker caption matching.
Limitations
- Decoder-only language models were not studied because they do not give a global sentence embedding.
- Local CKA has O(n^4) complexity in the naive version, making it slow for large query sets.
- Matching accuracy remains below CLIP cosine similarity for in-domain and cross-domain caption matching.
Experimental setup
Datasets: COCO NoCaps XTD-10 ImageNet-100
Benchmarks: COCO NoCaps XTD-10 ImageNet-100
Metrics: matching accuracy top-5 retrieval top-1 accuracy top-5 accuracy CKA
Baselines: CLIP cosine similarity linear regression relative representations
Builds on: centered kernel alignment relative representations fast QAP algorithm LiMBeR
Closest papers by meaning
- Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment (CVPR 2025 · similarity 0.91)
- Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models (ICML 2025 · similarity 0.91)
- SmartCLIP: Modular Vision-language Alignment with Identification Guarantees (CVPR 2025 · similarity 0.90)
- Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understanding (NeurIPS 2026 · similarity 0.89)
- Assessing and Learning Alignment of Unimodal Vision and Language Models (CVPR 2025 · similarity 0.89)
- Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.