Research map / Vision-language and multimodal / CLIP and zero-shot
DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Aligns a frozen DINOv2 vision encoder with a text encoder trained from scratch via LiT-style contrastive learning (dino.txt), using concatenated CLS+avg-patch pooling and text+image data curation, to get zero-shot classification and training-free open-vocabulary segmentation.
Problem
Self-supervised visual features like DINOv2 are not aligned with language, which limits open-vocabulary recognition and dense tasks; naive LiT with DINOv2 gives poor retrieval and segmentation performance.
Approach
Concatenate the [CLS] token with the average-pooled patch tokens as the image descriptor for contrastive LiT training, add two learnable vision blocks on the frozen encoder, and curate training data by balancing both text and image distributions; the patch outputs then directly give open-vocabulary segmentation without MaskCLIP-style adaptation.
Key results
- dino.txt reaches 81.4% zero-shot ImageNet-1K accuracy at 224 resolution, matching DFN and surpassing CLIP-L/14 at 76.6%, with state-of-the-art on IN-v2, IN-A and iNat21 among compared models.
- On open-vocabulary segmentation, dino.txt achieves 20.6 mIoU on ADE20K and 32.1 on Cityscapes, above the best generalist CLIP-like baseline (EVA-02-CLIP 10.9 / 14.1) and specialized TCL/CoDe on ADE20K's comparison; high-res inference reaches 25.1 ADE20K.
- dino.txt reaches 81.4% on IN1K in 19 hours on 128 A100 GPUs versus 110 hours for CLIP to reach 79.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Concatenate [CLS] with average-pooled patch features rather than using [CLS] alone (ablation) | dino.txt image representation · ImageNet-1K | accuracy: 78.8 -> 79.2 | Helped |
| Concatenate [CLS] with average-pooled patch features rather than using [CLS] alone (ablation) | dino.txt image representation · ADE20K | mIoU: 8.3 -> 18.2 | Helped |
| Use average pooling alone instead of the combined [CLS]-plus-average representation (ablation) | dino.txt · ImageNet-1K | accuracy: 79.2 -> 74.7 | Hurt |
| Use max pooling alone instead of the combined [CLS]-plus-average representation (ablation) | dino.txt · ADE20K | mIoU: 18.0 -> 18.2 | No clear effect |
| Add one trainable vision block on top of the frozen vision encoder (ablation) | dino.txt · COCO2017 | Recall@1: 34.7 -> 40.8 | Helped |
| Add two trainable vision blocks on top of the frozen vision encoder (ablation) | dino.txt · COCO2017 | Recall@1: 34.7 -> 42.1 | Helped |
| Increase batch size to 65K (ablation) | dino.txt · ImageNet-1K | accuracy: 79.2 -> 79.8 | Helped |
| Increase text embedding size from 768 to 1280 (ablation) | dino.txt · ImageNet-1K | accuracy: 79.7 -> 80.8 | Helped |
| Add image-based curation to text-based data curation (ablation) | dino.txt training data · COCO2017 | Recall@1: 43.9 -> 45.4 | Helped |
| Use image-based curation (ablation) | dino.txt training data · ImageNet-1K | accuracy: 80.3 -> 80.9 | Helped |
| Use text-based curation (ablation) | dino.txt training data · ImageNet-1K | accuracy: 80.3 -> 80.8 | Helped |
| Apply high-resolution inference with multi-scale crops | dino.txt · ADE20K | mIoU: 20.6 -> 25.1 | Helped |
Surprising findings
- Average-pooling patch tokens alone degrades classification (IN1K 74.7 vs 78.8), but concatenating CLS with average-pooled patches improves both classification and segmentation.
- Text-image retrieval lags SigLIP, attributed to the frozen vision encoder degrading the text encoder.
Limitations
- Text-image retrieval performance lags SigLIP, likely because the frozen vision encoder limits text encoder quality.
- Text encoder underperforms CLIP's on MTEB text tasks by about 4.2% on average.
- Standard segmentation benchmarks have overlapping classes and poorly chosen class names, limiting evaluation validity.
Experimental setup
Datasets: CommonCrawl LVTD-2.3B ImageNet-1K COCO Flickr30K ADE20K Cityscapes COCO-Stuff PASCAL Context PASCAL VOC20
Benchmarks: ImageNet-1K ImageNet-v2 ImageNet-A ObjectNet iNaturalist 2021 Places205 COCO retrieval Flickr30K ADE20K Cityscapes COCO-Stuff PASCAL Context
Metrics: top-1 accuracy Recall@1 mIoU
Baselines: CLIP SigLIP OpenCLIP EVA-02-CLIP DFN MetaCLIP TCL CoDe CLIPpy
Builds on: LiT DINOv2 CLIP MetaCLIP MaskCLIP
Closest papers by meaning
- Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models (ICML 2025 · similarity 0.91)
- Text as Partial Constraint: Core–Residual Alignment for Robust Vision–Language Learning (NeurIPS 2026 · similarity 0.90)
- Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment (CVPR 2025 · similarity 0.89)
- Building Vision-Language Models on Solid Foundations with Masked Distillation (CVPR 2024 · similarity 0.89)
- DH-Set: Improving Vision-Language Alignment with Diverse and Hybrid Set-Embeddings Learning (CVPR 2025 · similarity 0.89)
- Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.