Research map / Vision-language and multimodal / CLIP and zero-shot
Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentation
Luca Barsellotti, Lorenzo Bianchi, Nicola Messina, Fabio Carrara, Marcella Cornia, Lorenzo Baraldi, Fabrizio Falchi, Rita Cucchiara
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace DINOv2 with DINO as visual backbone (ablation) | Talk2DINO · Pascal VOC20 (ViT-B) | mIoU: 87.1 -> 27.3 | Hurt |
| use DINOv2 with registers (ablation) | Talk2DINO ViT-B · Pascal VOC20 | mIoU: 74.2 -> 87.1 | Improved |
| use DINOv2 with registers (ablation) | Talk2DINO ViT-S · Pascal Context 59 | mIoU: 38.3 -> 35.3 | Hurt |
| use nonlinear tanh mapping instead of linear projection on text (ablation) | Talk2DINO · Pascal Context 59 (ViT-B) | mIoU: 37.9 -> 39.8 | Improved |
| also map the vision side with the learned projection (ablation) | Talk2DINO · Pascal Context 59 (ViT-B) | mIoU: 39.8 -> 27.3 | Hurt |
| select max-similarity attention head instead of standard average (ablation) | Talk2DINO · Pascal Context 59 (ViT-B) | mIoU: 36.9 -> 39.8 | Improved |
| select max-similarity attention head instead of standard average (ablation) | Talk2DINO · Pascal VOC20 (ViT-B) | mIoU: 89.6 -> 87.1 | Hurt |
| add attention-based background cleaning (ablation) | Talk2DINO · COCO Object (ViT-B, no refinement) | mIoU: 37.1 -> 41.0 | Improved |
| add attention-based background cleaning (ablation) | Talk2DINO · Pascal VOC21 (ViT-B, no refinement) | mIoU: 59.9 -> 61.5 | Improved |
| keep CLIP text encoder frozen rather than unfreezing last layer (ablation) | Talk2DINO · Pascal Context 59 (ViT-B) | mIoU: 31.5 -> 39.8 | Improved |
| align CLS text token instead of average of text tokens (ablation) | Talk2DINO · Pascal Context 59 (ViT-B) | mIoU: 37.9 -> 39.8 | Improved |
| use ViT-L instead of ViT-B without mask refinement (ablation) | Talk2DINO · 8-benchmark average | mIoU (avg): 43.8 -> 42.8 | No clear effect |
Closest papers by meaning
- CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentation (ECCV 2024 · similarity 0.93)
- DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignment (CVPR 2025 · similarity 0.92)
- DINOde: Continuous Vision-Text Alignment for Open-Vocabulary Semantic Segmentation (ECCV 2026 · similarity 0.91)
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.91)
- OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts (NeurIPS 2025 · similarity 0.91)
- Grounding Everything: Emerging Localization Properties in Vision-Language Transformers (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
