Research map / Vision-language and multimodal / CLIP and zero-shot
Dual Semantic Guidance for Open Vocabulary Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds dual (visual and textual) semantic guidance and pixel-level pseudo-masks from CC3M image-text pairs using SAM instances and noun extraction, then fine-tunes a MaskCLIP-based CLIP segmenter for open-vocabulary semantic segmentation.
Problem
Open-vocabulary semantic segmentation with CLIP-based models suffers from coarse text supervision (inherent biases from nouns) and lack of pixel-level supervision when fine-tuning on image-text data.
Approach
Generate visual semantic guidance from SAM instance regions (refined by a training-free Region Awareness Attention that focuses CLIP features on foreground, then clustered) and textual guidance from nouns matched to those visual guides, and train with BCE/contrastive losses plus a background-screening module.
Key results
- Achieves average mIoU of 48.3 on three datasets with background and 42.8 on five datasets without background, the best reported in the table.
- On datasets with background it improves VOC21 by +5.6, Ctx60 by +8.4 and Object by +1.5 mIoU over prior SOTA.
- Ablation: removing Region Awareness Attention drops performance from 48.3 to 45.6 (w/ BG) and 42.8 to 39.9 (w/o BG).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Dual Semantic Guidance instead of the strongest reported baseline | Open-vocabulary semantic segmentation · PASCAL VOC21 | mIoU: 62.1 -> 67.7 | Helped |
| use Dual Semantic Guidance instead of the strongest reported baseline | Open-vocabulary semantic segmentation · PASCAL Context60 | mIoU: 32.4 -> 40.8 | Helped |
| use Dual Semantic Guidance instead of the strongest reported baseline | Open-vocabulary semantic segmentation · COCO Object | mIoU: 34.8 -> 36.3 | Helped |
| use Dual Semantic Guidance instead of the strongest reported baseline | Open-vocabulary semantic segmentation · PASCAL VOC20 | mIoU: 85.9 -> 85.0 | Hurt |
| use Dual Semantic Guidance instead of the strongest reported baseline | Open-vocabulary semantic segmentation · PASCAL Context59 | mIoU: 35.9 -> 46.5 | Helped |
| use Dual Semantic Guidance instead of the strongest reported baseline | Open-vocabulary semantic segmentation · COCO Stuff | mIoU: 24.6 -> 25.7 | Helped |
| use Dual Semantic Guidance instead of the strongest reported baseline | Open-vocabulary semantic segmentation · ADE20K | mIoU: 20.0 -> 20.7 | Helped |
| use Dual Semantic Guidance instead of the strongest reported baseline | Open-vocabulary semantic segmentation · CityScapes | mIoU: 34.8 -> 36.0 | Helped |
| remove initial filtering from semantic guidance generation (ablation) | Dual Semantic Guidance · Datasets with and without background | mIoU: 48.3 -> 11.8; 42.8 -> 15.4 | Hurt |
| add Region Awareness Attention (ablation) | Dual Semantic Guidance · Datasets with and without background | mIoU: 45.6 -> 48.3; 39.9 -> 42.8 | Helped |
| remove clustering from semantic guidance generation (ablation) | Dual Semantic Guidance · Datasets with and without background | mIoU: 48.3 -> 46.7; 42.8 -> 40.8 | Hurt |
| apply Class and Patch attention at different stages rather than all stages (ablation) | Region Awareness Attention · Datasets with and without background | mIoU: 47.1 -> 48.3; 42.3 -> 42.8 | Helped |
Surprising findings
- Textual-only guidance outperforms visual-only guidance on datasets with background, while visual-only is better on datasets without background.
- Guidance Enhance applied to textual guidance drops performance on w/ BG to 12.3 mIoU, showing modality discrepancy issues.
Limitations
- Relies on SAM to generate many masks per image, which is time-consuming.
- Segmentation boundaries remain rough, as in existing OVSS methods.
- K in K-means is fixed (K=5) rather than adaptively selected.
Experimental setup
Datasets: CC3M COCO Stuff ADE20K PASCAL Context59 PASCAL VOC20 Cityscapes PASCAL Context60 PASCAL VOC21 COCO Object
Benchmarks: PASCAL VOC21 PASCAL Context60 COCO Object PASCAL VOC20 PASCAL Context59 COCO Stuff ADE20K Cityscapes
Metrics: mIoU
Baselines: CLIP ViL-Seg ViewCo SimSeg OVSegmentor TCL SegCLIP CoCu PGSeg CoDe CLIP-DINOiser PixelCLIP
Builds on: CLIP MaskCLIP SAM TCL CoDe
Closest papers by meaning
- Open-Vocabulary Segmentation with Semantic-Assisted Calibration (CVPR 2024 · similarity 0.91)
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.90)
- CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.90)
- Towards Open-Vocabulary Semantic Segmentation Without Semantic Labels (NeurIPS 2024 · similarity 0.89)
- Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.89)
- Effective SAM Combination for Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.