Research map / Vision-language and multimodal / CLIP and zero-shot
DPSeg: Dual-Prompt Cost Volume Learning for Open-Vocabulary Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DPSeg, a dual-prompt (text + Stable Diffusion-generated visual prompt) cost-volume framework with a cost-volume-guided decoder and two-pass semantic-guided prompt refinement for open-vocabulary semantic segmentation.
Problem
Open-vocabulary semantic segmentation with CLIP text embeddings suffers from a modality gap between image and text embeddings, and cost volumes upsampled from deep features lose fine detail for small objects.
Approach
Averages text embeddings with visual prompt embeddings (images generated by Stable Diffusion from the text templates) to build a dual-prompt cost volume, adds multi-scale visual cost volumes from shallow features inside the decoder instead of upsampling the cost volume, and refines visual prompts with first-pass masks crops in a second inference pass.
Key results
- With ConvNeXt-L CLIP, DPSeg (Inference II) reaches 15.7 A-847, 24.1 PC-459, 37.1 A-150, 62.3 PC-59 and 98.5 PAS-20 mIoU, beating SED by +1.8, +1.5, +1.9, +1.7, +2.4 mIoU.
- Inference II adds about +0.9 mIoU over Inference I (e.g., 0.92 for the large-VLM setting).
- Averaging text and visual prompt embeddings beats text-only (10.4 A-847) and visual-only (11.1 A-847) prompts, reaching 12.0 A-847 in the base setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply semantic-guided prompt refinement in a second inference pass | DPSeg · A-847 | mIoU: 12.0 -> 12.5 | Helped |
| apply semantic-guided prompt refinement in a second inference pass | DPSeg · PC-459 | mIoU: 19.5 -> 20.1 | Helped |
| apply semantic-guided prompt refinement in a second inference pass | DPSeg · A-150 | mIoU: 32.9 -> 33.3 | Helped |
| apply semantic-guided prompt refinement in a second inference pass | DPSeg · PC-59 | mIoU: 58.1 -> 58.4 | Helped |
| apply semantic-guided prompt refinement in a second inference pass | DPSeg · PAS-20 | mIoU: 96.0 -> 96.9 | Helped |
| replace averaged text and visual prompts with text-only prompts (ablation) | DPSeg · A-847 | mIoU: 12.0 -> 10.4 | Hurt |
| replace averaged text and visual prompts with visual-only prompts (ablation) | DPSeg · A-847 | mIoU: 12.0 -> 11.1 | Hurt |
| average text and visual embeddings before calculating cosine similarity instead of concatenating separate cost volumes (ablation) | DPSeg · A-847 | mIoU: 11.7 -> 12.0 | Helped |
| average text and visual embeddings before calculating cosine similarity instead of averaging the resulting cost volumes (ablation) | DPSeg · A-847 | mIoU: 11.9 -> 12.0 | Helped |
| add multi-scale visual cost volume guidance at scales 2, 3, and 4 (ablation) | DPSeg decoder · A-847 | mIoU: 10.6 -> 12.0 | Helped |
| replace multi-scale visual cost volumes with upsampled Fc (ablation) | DPSeg decoder · A-847 | mIoU: 12.0 -> 11.4 | Hurt |
| increase prompt templates from 1 to 80 (ablation) | DPSeg · A-847 | mIoU: 3.6 -> 12.0 | Helped |
Surprising findings
- Visual-only prompts outperform text-only prompts in the ablation (11.1 vs 10.4 A-847 mIoU), supporting the claim that visual prompts align better with image features.
Limitations
- Requires a Stable Diffusion pass to generate visual prompts for every template and category, adding cost.
- Two-pass inference doubles the inference computation.
- Visual prompt quality depends on text-to-image generation, with only limited ablation on generators.
Experimental setup
Datasets: COCO-Stuff
Benchmarks: ADE20K-847 PC-459 ADE20K-150 PC-59 PASCAL VOC-20
Metrics: mIoU
Baselines: SED CAT-Seg SAN EBSeg SCAN OVSeg FC-CLIP ODISE
Builds on: CAT-Seg SED CLIP Stable Diffusion
Closest papers by meaning
- CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.89)
- Dual Semantic Guidance for Open Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.89)
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.89)
- Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation (ICML 2025 · similarity 0.89)
- Relationship Prompt Learning is Enough for Open-Vocabulary Semantic Segmentation (NeurIPS 2024 · similarity 0.88)
- Unknown Prompt the only Lacuna: Unveiling CLIP's Potential for Open Domain Generalization (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.