Research map / Vision-language and multimodal / Human and embodied modeling
It's a (Blind) Match! Towards Vision-Language Correspondence without Parallel Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies whether vision and language foundation model embeddings can be matched without parallel data by formulating blind matching as a quadratic assignment problem, proposing a factorized Hahn-Grant solver, and demonstrating an unsupervised classifier.
Problem
Matching vision and language embedding spaces without any paired image-text data, i.e. unsupervised (blind) vision-language correspondence, motivated by the platonic representation hypothesis.
Approach
Formulates unsupervised vision-language matching as a quadratic assignment problem over pairwise within-modality similarities (Gromov-Wasserstein distortion), introduces a factorized Hahn-Grant solver with primal heuristics and lower memory use, and a subset-selection method to find matchable class sets.
Key results
- On CIFAR-10 with all-mpnet-base-v2, DINOv2 reaches up to 72% matching accuracy and 100% on CINIC-10 (vs 10% random).
- The factorized Hahn-Grant solver reaches the global optimum 100% of the time on CINIC-10 small problems (60% accuracy), while MPOpt reaches it 50% and FAQ/OT fail (1-2% accuracy).
- Unsupervised K-Means plus blind matching classifier on CIFAR-10 achieves 51.5% accuracy with DINOv2 and All-Roberta-large-v1 (random 10%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the factorized Hahn-Grant solver instead of FAQ | CLIP + all-mpnet-base-v2 matching · CINIC-10 | matching accuracy (%): 1±3 -> 60±0 | Helped |
| Use the factorized Hahn-Grant solver instead of MPOpt | CLIP + all-mpnet-base-v2 matching · CINIC-10 | global optimum frequency (%): 50±51 -> 100±0 | Helped |
| Use the factorized Hahn-Grant solver instead of LocalCKA | CLIP + all-mpnet-base-v2 matching · CINIC-10 | matching accuracy (%): 10±9 -> 60±0 | Helped |
| Match DINOv2 embeddings with language embeddings | DINOv2 vision and language models · CIFAR-10 | matching accuracy: 72% | Helped |
| Match DINOv2 embeddings with language embeddings | DINOv2 vision and language models · CINIC-10 | matching accuracy: 100% | Helped |
| Use DINOv2 with All-Roberta-large-v1 for unsupervised classification | Unsupervised classifier · CIFAR-10 | classification accuracy (%): 10% -> 51.5±9.2 | Helped |
Surprising findings
- Local solvers (OT, FAQ) reduce the matching cost but yield accuracy no better than random, showing local optima are insufficient.
- Pre-training strategy has a larger impact on matchability than model size.
Limitations
- Matching accuracy varies strongly with class choice; some abstract concepts cannot be matched.
- Solver complexity O(N^5) limits problem size to about N=40 for guaranteed optimality; bounds loose at N=100.
- Unsupervised classifier accuracy (max 51.5%) is far below supervised models.
Experimental setup
Datasets: CIFAR-10 CINIC-10 ImageNet-100 CIFAR-100 CocoCaptions
Benchmarks: CIFAR-10 CINIC-10 ImageNet-100 CIFAR-100
Metrics: matching accuracy Gromov-Wasserstein cost lower bound global optimum frequency unsupervised classification accuracy mutual k-NN
Baselines: LocalCKA MPOpt Gurobi FAQ entropic Gromov-Wasserstein OT random matching
Builds on: Hahn-Grant solver platonic representation hypothesis Gromov-Wasserstein distance FAQ relative representations
Closest papers by meaning
- DH-Set: Improving Vision-Language Alignment with Diverse and Hybrid Set-Embeddings Learning (CVPR 2025 · similarity 0.89)
- Bridging Vision and Language Spaces with Assignment Prediction (ICLR 2024 · similarity 0.88)
- Do Vision and Language Encoders Represent the World Similarly? (CVPR 2024 · similarity 0.87)
- DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment (CVPR 2025 · similarity 0.87)
- Revisiting the Role of Language Priors in Vision-Language Models (ICML 2024 · similarity 0.87)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.