Research map / Detection and segmentation / Open-set and few-shot detection
InvSeg: Test-Time Prompt Inversion for Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes InvSeg, a test-time prompt inversion method that optimizes class-name text embeddings of Stable Diffusion per image using contrastive soft clustering over self-attention structure, achieving unsupervised open-vocabulary semantic segmentation.
Problem
Open-vocabulary semantic segmentation with diffusion models is hurt by the mismatch between isolated class names used for segmentation and the context-rich sentences diffusion models are trained on, yielding incomplete or confused cross-attention masks.
Approach
Treat the text token embeddings as the only learnable parameters at test time and optimize them per image, using a contrastive soft clustering loss that aligns refined cross-attention masks with a KL-based distance structure derived from self-attention, plus an entropy-minimization term over augmented views.
Key results
- InvSeg reaches 63.4 mIoU on PASCAL VOC and 27.8 on PASCAL Context, the best among compared methods, versus 60.1 and 27.5 for DiffSegmenter.
- On COCO Object InvSeg scores 36.0 mIoU, behind DiffSegmenter's 37.9.
- Gains over the diffusion baseline are up to 3.8% on VOC and 2.8% on Context.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| invert image-specific prompts with InvSeg instead of the original-prompt Diffusion Baseline | Stable Diffusion v2.1 · PASCAL VOC 2012 | mIoU: 59.6 -> 63.4 | Helped |
| invert image-specific prompts with InvSeg instead of the original-prompt Diffusion Baseline | Stable Diffusion v2.1 · PASCAL Context | mIoU: 25.0 -> 27.8 | Helped |
| use InvSeg instead of DiffSegmenter | open-vocabulary segmentation · COCO Object | mIoU: 37.9 -> 36.0 | Hurt |
| add entropy minimization with alpha=1 rather than alpha=0 (ablation) | InvSeg · PASCAL VOC 2012 | mIoU: 61.9 -> 63.4 | Helped |
| add entropy minimization with alpha=1 rather than alpha=0 (ablation) | InvSeg · COCO Object | mIoU: 35.0 -> 36.0 | Helped |
| remove Contrastive Soft Clustering (ablation) | InvSeg · PASCAL VOC 2012 | mIoU: 61.9 -> 63.4 | Helped |
| remove Contrastive Soft Clustering (ablation) | InvSeg · COCO Object | mIoU: 35.7 -> 36.0 | Helped |
| set anchor selection scale to 4 rather than 8 or 16 (ablation) | InvSeg · PASCAL VOC 2012 | mIoU: 63.0; 62.7 -> 63.4 | Helped |
| initialize text using ground-truth class names instead of DiffSegmenter strategy (ablation) | InvSeg · PASCAL VOC 2012 | mIoU: 63.4 -> 69.4 | Helped |
| initialize text using ground-truth class names instead of DiffSegmenter strategy (ablation) | InvSeg · COCO Object | mIoU: 36.0 -> 40.3 | Helped |
| use only Res16 for prompt optimization instead of other tested resolutions (ablation) | InvSeg · PASCAL VOC 2012 | mIoU: 18.4; 46.5; 22.9 -> 63.4 | Helped |
| use Res16 and Res32 rather than Res16 only (ablation) | InvSeg · PASCAL VOC 2012 | mIoU: 63.4 -> 60.7 | Hurt |
Surprising findings
- Using ground-truth class names raises performance substantially (69.4 VOC mIoU), suggesting candidate class name extraction is a major bottleneck.
- Optimizing only at a specific resolution (16x16 self-attention layers) works best, and combining multiple resolutions does not help.
Limitations
- Trails DiffSegmenter on COCO Object (36.0 vs 37.9 mIoU), attributed to simultaneous rather than per-class segmentation.
- Per-image test-time optimization adds inference cost (about 7.9 s per image).
- Performance depends on the class-name extraction step, which uses BLIP/CLIP and is a bottleneck relative to ground-truth names.
Experimental setup
Datasets: PASCAL VOC 2012 PASCAL Context COCO Object
Benchmarks: PASCAL VOC 2012 PASCAL Context COCO Object
Metrics: mIoU mAcc
Baselines: DiffSegmenter Diffusion Baseline TPT GroupViT TCL SimSeg OVSegmentor CLIPpy SegCLIP MaskCLIP
Builds on: DiffSegmenter Textual Inversion Test-time Prompt Tuning (TPT) DiffSeg
Closest papers by meaning
- Seg4Diff: Unveiling Open-Vocabulary Semantic Segmentation in Text-to-Image Diffusion Transformers (NeurIPS 2025 · similarity 0.89)
- Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.89)
- VidSeg: Training-free Video Semantic Segmentation based on Diffusion Models (CVPR 2025 · similarity 0.88)
- Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation (ICML 2025 · similarity 0.88)
- DPSeg: Dual-Prompt Cost Volume Learning for Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.88)
- Text-Image Alignment for Diffusion-Based Perception (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.