Research map / Vision-language and multimodal / CLIP and zero-shot
Assessing and Learning Alignment of Unimodal Vision and Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes alignment probing (a frozen-backbone linear alignment layer) to assess unimodal vision-language alignment, and SAIL, an efficient framework that aligns frozen DINOv2 with pretrained text encoders using a GLU alignment layer, sigmoid loss, and high-quality captions.
Problem
Existing assessments of whether unimodal vision and language models are aligned (e.g., mutual nearest-neighbor metrics) are proxies that do not directly match how VLMs compute cross-modal distances, and training CLIP-like models from scratch is data- and compute-heavy.
Approach
Freeze pretrained vision and language backbones and train only a lightweight alignment layer with contrastive loss on pre-encoded image-text embeddings; use this probe to show k-NN clustering quality predicts alignment and that strong text encoders matter, then build SAIL with a GLU alignment layer, sigmoid loss, and multi-positive long-caption training.
Key results
- SAIL-L-NV2 reaches 73.4% zero-shot ImageNet-1k accuracy vs. CLIP-L's 72.7% while using about 6% of CLIP's paired data (23M vs 400M).
- k-NN performance correlates with alignment score at Pearson 0.991 vs 0.847 for linear probing, supporting clustering quality as the key SSL property.
- SAIL-L-NV2 outperforms CLIP-L/14 on COCO and Flickr retrieval and on Winoground, and ranks above CLIP in 5 of 7 LLaVA-1.5 downstream tasks when using its vision encoder.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DINOv2-L with NV-Embed-2 in SAIL instead of CLIP-L | zero-shot image-text model · ImageNet-1k | top-1 accuracy: 72.7 -> 73.4 | Helped |
| Use DINOv2-L with NV-Embed-2 in SAIL instead of CLIP-L | zero-shot image-text model · MSCOCO | T2I Recall@1: 43.0 -> 48.6 | Helped |
| Use DINOv2-L with NV-Embed-2 in SAIL instead of CLIP-L | zero-shot image-text model · Winoground | Group score: 8.75 -> 15.0 | Helped |
| Use DINOv2-L with NV-Embed-2 in SAIL instead of CLIP-L | zero-shot image-text model · MMVP | average score: 20.0 -> 28.9 | Helped |
| Use DINOv2-L with NV-Embed-2 in SAIL instead of CLIP-based segmentation | open-vocabulary segmentation · ADE20K | mIOU: 7.1 -> 14.2 | Helped |
| Replace the linear alignment layer with MLP ↓4 (ablation) | SAIL alignment probing · COCO | T2I R@1: 11.1 -> 8.0 | Hurt |
| Use a GLU ↓8 alignment layer instead of the preceding GLU ↓4 (ablation) | SAIL alignment probing · ImageNet-1k | top-1 accuracy: 39.6 -> 45.4 | Helped |
| Replace InfoNCE with Sigmoid loss (ablation) | SAIL alignment probing · COCO | T2I R@1: 16.1 -> 25.4 | Helped |
| Average loss across all pairs instead of only positive pairs (ablation) | SAIL alignment probing · ImageNet-1k | top-1 accuracy: 50.7 -> 51.8 | Helped |
| Train with long high-quality synthetic captions instead of short web captions (ablation) | SAIL alignment probing · ImageNet-1k | top-1 accuracy: 51.8 -> 48.4 | Hurt |
| Train with long high-quality synthetic captions instead of short web captions (ablation) | SAIL alignment probing · COCO | T2I R@1: 26.2 -> 31.4 | Helped |
| Combine short and long high-quality captions as multiple positives (ablation) | SAIL alignment probing · ImageNet-1k | top-1 accuracy: 48.4 -> 54.0 | Helped |
Surprising findings
- MAE-series models show markedly weaker alignment than other SSL models despite large size, and AIM-L (1B) underperforms DINOv2-B (86M).
- CLIP scaled from 400M to 2B data and 427M to 1366M parameters still scores low on Winoground Image and Group metrics.
Limitations
- Alignment probing uses a linear layer and retrieval on COCO, a proxy for general alignment across tasks.
- SAIL performs poorly on OCR-heavy benchmarks such as TextVQA and MMBench, attributed to DINOv2's limited OCR ability.
- Some data (image URLs) expired, so training used subsets of CC3M, CC12M and YFCC15M.
Experimental setup
Datasets: CC3M CC12M YFCC15M ImageNet-1k COCO Flickr30k Winoground MMVP ADE20K COCO-Stuff VOC20
Benchmarks: ImageNet-1k COCO Flickr30k Winoground MMVP ADE20K COCO-Stuff VOC20 MMBench GQA VQAv2 TextVQA
Metrics: zero-shot top-1 accuracy recall@1 recall@10 Winoground text/image/group score MMVP average mIoU Pearson correlation
Baselines: CLIP DreamLIP LiT ShareLock MaskCLIP SCLIP DINOv2 with LLaVA-1.5
Builds on: LiT ShareLock CLIP DreamLIP MaskCLIP
Closest papers by meaning
- Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment (CVPR 2025 · similarity 0.91)
- Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.89)
- LVLM-Aided Alignment of Task-Specific Vision Models (CVPR 2026 · similarity 0.89)
- SineProject: Machine Unlearning for Stable Vision-Language Alignment (CVPR 2026 · similarity 0.89)
- Do Vision and Language Encoders Represent the World Similarly? (CVPR 2024 · similarity 0.89)
- Locality Alignment Improves Vision-Language Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.