Research map / Detection and segmentation / Open-set and few-shot detection
Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FreeDA, a training-free open-vocabulary semantic segmentation method that builds offline diffusion-generated visual prototypes and textual keys, then combines DINOv2 superpixel local similarities with CLIP global similarities at inference.
Problem
Open-vocabulary semantic segmentation typically requires expensive contrastive training on large image-caption datasets to align text with pixels, and captions lack direct localization of individual concepts.
Approach
Use cross-attention localization in Stable Diffusion on COCO captions to generate weak masks, pool DINOv2 features into visual prototypes indexed by textual keys offline; at test time retrieve prototypes per class and match superpixel regions by combining local DINOv2 similarity with global CLIP similarity, with no training.
Key results
- FreeDA (ViT-L) reaches 87.9 mIoU on Pascal VOC, 43.5 on COCO Stuff and 23.2 on ADE20K without the unknown class, state of the art among unsupervised methods.
- Averaged over datasets, FreeDA improves over TCL without PAMR by 10.0 (ViT-B) and 10.9 (ViT-L) mIoU points.
- With the unknown category, FreeDA (ViT-L) obtains 55.4 on Pascal VOC, 38.3 on Pascal Context and 37.4 on COCO Object, beating the best PAMR-refined competitor.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add global CLIP similarity to local DINOv2 ViT-B/14 similarity (ablation) | FreeDA · Pascal VOC | mIoU: 75.6 -> 86.2 | Helped |
| Add global CLIP similarity to local DINOv2 ViT-B/14 similarity (ablation) | FreeDA · Cityscapes | mIoU: 34.4 -> 35.0 | Helped |
| Add global CLIP similarity to local DINOv2 ViT-B/14 similarity (ablation) | FreeDA · ADE20K | mIoU: 20.7 -> 21.9 | Helped |
| Add superpixel regions to DINOv2 ViT-B/14 with global similarity (ablation) | FreeDA · Pascal VOC | mIoU: 86.2 -> 87.0 | Helped |
| Add superpixel regions to DINOv2 ViT-B/14 with global similarity (ablation) | FreeDA · Cityscapes | mIoU: 35.0 -> 36.6 | Helped |
| Add superpixel regions to DINOv2 ViT-B/14 with global similarity (ablation) | FreeDA · ADE20K | mIoU: 21.9 -> 23.2 | Helped |
| Use DINOv2 instead of DINO as local visual backbone (ablation) | FreeDA without global similarity or superpixels · Pascal VOC | mIoU: 34.2 -> 75.6 | Helped |
| Use larger ViT-L/14 instead of ViT-B/14 visual and textual backbones (ablation) | FreeDA · Pascal VOC | mIoU: 85.6 -> 87.9 | Helped |
| Use larger ViT-L/14 instead of ViT-B/14 visual and textual backbones (ablation) | FreeDA · Cityscapes | mIoU: 36.7 -> 36.7 | No clear effect |
| Use Felzenszwalb superpixels instead of Watershed (ablation) | FreeDA with mean prototype embedding · Cityscapes | mIoU: 32.7 -> 36.7 | Helped |
| Aggregate retrieved prototypes by mean embedding instead of mean similarity (ablation) | FreeDA · Pascal VOC | mIoU: 79.5 -> 87.9 | Helped |
| Aggregate retrieved prototypes by mean embedding instead of max similarity (ablation) | FreeDA · Cityscapes | mIoU: 26.2 -> 36.7 | Helped |
Surprising findings
- Retrieving more prototypes (K above 350) does not improve performance, and approximate HNSW search only partially degrades results while cutting time.
Limitations
- Relies on superpixel partitioning and a fixed set of hyperparameters (beta, K, gamma) tuned per dataset.
- Offline prototype generation requires generating a large set of images with Stable Diffusion, which is computationally costly upfront.
Experimental setup
Datasets: COCO Captions
Benchmarks: Pascal VOC 2012 Pascal Context COCO Stuff Cityscapes ADE20K COCO Object
Metrics: mIoU
Baselines: ReCo GroupViT MaskCLIP TCL OVDiff ViewCo SegCLIP OVSegmentor
Builds on: DAAM DINOv2 CLIP Felzenszwalb superpixels OVDiff
Closest papers by meaning
- Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.92)
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.91)
- Image-to-Image Matching via Foundation Models: A New Perspective for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.90)
- Open-Vocabulary Segmentation with Semantic-Assisted Calibration (CVPR 2024 · similarity 0.90)
- Dual Semantic Guidance for Open Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.89)
- A Simple Framework for Open-Vocabulary Zero-Shot Segmentation (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.