Research map / Vision-language and multimodal / CLIP and zero-shot

A Simple Framework for Open-Vocabulary Zero-Shot Segmentation

ICLR 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: Attribute-based zero-shot learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SimZSS, which freezes a DINOv2 vision encoder and trains only a text encoder with a global image-caption loss plus a noun-phrase-to-patch concept consistency loss, reaching state-of-the-art zero-shot segmentation in about 15 minutes on 8 GPUs.

Problem

Vision-language contrastive models trained on image-caption pairs classify well but perform poorly on dense zero-shot open-vocabulary segmentation, due to missing localization cues in captions and entangled learning of image/text representations and cross-modal alignment.

Approach

Freeze a spatially aware pure vision model (DINOv2) and align only a text encoder to it; identify concepts (noun phrases filtered by a concept bank) in captions and retrieve the matching visual patches via similarity-based pooling, then apply a semantic-level concept consistency loss on top of the global CLIP-style loss.

Key results

• Trained on COCO Captions with a DINOv2 ViT-B/14 vision tower, SimZSS reaches 90.3 mIoU on Pascal VOC and 43.1 on Pascal Context (without background), beating LiT reproduction (86.1 and 35.5).
• SimZSS with ViT-B/14 on COCO Captions achieves state-of-the-art on 7 of 8 segmentation benchmarks, per the authors, with the same training taking under 15 minutes on 8 GPUs.
• Zero-shot ImageNet-1k accuracy with LAION-400M alignment reaches 64.1% (1.6B samples gives 69.3%), comparable to LiT's 63.6%.

Measured effects reported in the paper

Surprising findings

• Training without the concept bank still outperforms LiT by a large margin, suggesting the gain comes from the noun-phrase-to-patch consistency objective rather than the bank itself.
• ViT-L/14 DINOv2 does not improve over ViT-B/14, and its [CLS] token was poorly aligned with patch tokens, forcing average pooling.

Limitations

• Background class is hard to represent with text concepts, so background results are less consistent and rely on a crude confidence threshold.
• Relies on a predefined concept bank of 574 class names and a POS tagger to identify concepts in captions.
• Requires training, and LAION-400M gains on segmentation favor curated data over scale.

Experimental setup

Datasets: COCO Captions LAION-400M Pascal VOC Pascal Context COCO-Stuff COCO-Object Cityscapes ADE20K ImageNet-1k

Benchmarks: Pascal VOC Pascal Context COCO-Stuff Cityscapes ADE20K COCO-Object ImageNet-1k 38-dataset zero-shot classification suite

Metrics: mIoU top-1 accuracy

Baselines: LiT CLIP ReCo GroupViT TCL MaskCLIP OVDiff CLIP-DINOiser ProxyCLIP SCLIP

Builds on: LiT CLIP DINOv2 TCL MaskCLIP

Closest papers by meaning

• Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training (AAAI 2024 · similarity 0.89)
• Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation (CVPR 2024 · similarity 0.89)
• Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.89)
• OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts (NeurIPS 2025 · similarity 0.88)
• Exploring Simple Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.88)
• Text Augmented Correlation Transformer For Few-shot Classification & Segmentation (CVPR 2025 · similarity 0.87)