Research map / Vision-language and multimodal / CLIP and zero-shot
Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PnP-OVSS, a training-free method that extracts open-vocabulary semantic segmentation masks from off-the-shelf VLMs (BLIP, BridgeTower) using text-to-image cross-attention, GradCAM with the ITM loss, Salience DropOut, and Dense CRF, with hyperparameters tuned by a CLIP-based reward instead of pixel labels.
Problem
Open-vocabulary semantic segmentation from pretrained vision-language models without extra training or dense annotations, since VLMs implicitly associate image regions with words but this association is hard to exploit.
Approach
Use cross-attention maps from an off-the-shelf VLM's image-text matching architecture, sharpened by ITM-loss GradCAM, expanded by iteratively dropping the most salient patches (Salience DropOut), and tuned via a weakly supervised CLIP contrastive reward with random search, requiring no training or segmentation labels.
Key results
- PnP-OVSS + BLIP outperforms training-free baselines by +29.4% mIoU on Pascal VOC, +13.2% on Pascal Context, +14.0% on COCO Object at equal resolution.
- PnP-OVSS + BLIPFlickr reaches 53.6 mIoU on Pascal VOC and 39.8 on COCO Stuff on unseen-class comparison, beating most methods trained with dense labels.
- Ablation shows Salience DropOut and Gaussian blur each give large gains; Gaussian blur alone adds +9.6 on Pascal Context.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add GradCAM sharpening (ablation) | PnP-OVSS + BLIPFlickr · Pascal Context and COCO Stuff | mIoU | Helped |
| add the first Salience DropOut iteration (ablation) | PnP-OVSS + BLIPFlickr · Pascal Context and COCO Stuff | mIoU: +3.5/2.3 | Helped |
| add the second Salience DropOut iteration (ablation) | PnP-OVSS + BLIPFlickr · Pascal Context and COCO Stuff | mIoU: +1.4/0.8 | Helped |
| add the third and fourth Salience DropOut iterations (ablation) | PnP-OVSS + BLIPFlickr · Pascal Context and COCO Stuff | mIoU | Helped |
| add Gaussian blur after Salience DropOut (ablation) | PnP-OVSS + BLIPFlickr · Pascal Context and COCO Stuff | mIoU: +9.6/7.7 | Helped |
| add Dense CRF without Gaussian blur (ablation) | PnP-OVSS + BLIPFlickr · Pascal Context | mIoU: 1.5 mIoU | Hurt |
| combine Dense CRF with Gaussian blur (ablation) | PnP-OVSS + BLIPFlickr · Pascal Context and COCO Stuff | mIoU | Helped |
| use different cross-attention layers or attention heads (ablation) | PnP-OVSS + BLIPFlickr · COCO Stuff | mIoU: 20; 23.8 | Mixed |
Surprising findings
- Gaussian blur alone yields large gains, while Dense CRF alone is worse than blur by 1.5 mIoU on Pascal Context.
- Best attention layer/head varies drastically (up to 20 mIoU across layers), and the CLIP-based reward selects the best layer-head combination without segmentation labels.
Limitations
- Depends on VLMs having text-to-image cross-attention and an ITM loss, limiting applicability.
- Hyperparameters (layer, head, threshold) have large influence and require a search with a CLIP-based reward.
- Gap to fully supervised or finetuned methods remains on some datasets (e.g., Pascal VOC vs. PACL and supervised methods).
Experimental setup
Datasets: Pascal VOC 2012 Pascal Context COCO Object COCO Stuff ADE20K
Benchmarks: Pascal VOC 2012 Pascal Context COCO Object COCO Stuff ADE20K
Metrics: mIoU
Baselines: MaskClip Reco MaskClip w/ ST ZeroSeg PACL TCL GroupViT CLIPpy
Builds on: GradCAM MaskCLIP Dense CRF CutLER
Closest papers by meaning
- Relationship Prompt Learning is Enough for Open-Vocabulary Semantic Segmentation (NeurIPS 2024 · similarity 0.92)
- Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation (NeurIPS 2025 · similarity 0.92)
- CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor (CVPR 2024 · similarity 0.92)
- Transferable and Principled Efficiency for Open-Vocabulary Segmentation (CVPR 2024 · similarity 0.91)
- Unveiling the Knowledge of CLIP for Training-Free Open-Vocabulary Semantic Segmentation (AAAI 2025 · similarity 0.91)
- OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.