Research map / Detection and segmentation / Open-set and few-shot detection
Open-Vocabulary Segmentation with Semantic-Assisted Calibration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SCAN, a two-stage open-vocabulary segmentation network that injects CLIP global semantic priors into mask proposal embeddings and applies a contextual shift to replace background tokens in CLIP, plus a new SG-IoU metric.
Problem
Open-vocabulary segmentation in two-stage pipelines suffers because mask proposal embeddings overfit to training categories and CLIP classification of cropped/masked sub-images is domain-biased, limiting alignment with unbounded text.
Approach
A Semantic Integration Module transfers frozen CLIP [CLS] and low-frequency-enhanced spatial token features into proposal embeddings, and a Contextual Shift strategy replaces background tokens in CLIP layers with the whole-image [CLS] embedding to reduce domain bias; additionally, SG-IoU credits predictions of synonymous or parent categories.
Key results
- With CLIP ViT-B/16, SCAN reaches 30.8 mIoU on ADE-150 and 58.4 on PC-59, outperforming prior methods such as SAN (27.5 / 53.8).
- With CLIP ViT-L/14, SCAN achieves 33.5 on ADE-150, 14.0 on ADE-847, 59.3 on PC-59, 16.7 on PC-459 and 97.2 on VOC, state of the art among compared methods.
- Ablation: SIM alone raises ADE-150 from 31.5 to 32.9 and CS alone to 32.8; both together reach 33.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Semantic Integration Module (ablation) | SCAN baseline · ADE20K-150 | mIoU: 31.5 -> 32.9 | Helped |
| add Contextual Shift (ablation) | SCAN baseline · ADE20K-150 | mIoU: 31.5 -> 32.8 | Helped |
| add both SIM and Contextual Shift (ablation) | SCAN baseline · ADE20K-150, ADE20K-847, Pascal Context-59 | mIoU: 31.5 -> 33.5; 11.4 -> 14.0; 57.1 -> 59.3 | Helped |
| add the final CLIP [CLS] embedding to proposal embeddings (ablation) | SIM · ADE20K-150, ADE20K-847, Pascal Context-59 | mIoU: 31.5 -> 32.5; 11.4 -> 12.7; 57.1 -> 57.5 | Helped |
| add CLIP spatial token embeddings (ablation) | SIM · ADE20K-150, ADE20K-847, Pascal Context-59 | mIoU: 31.5 -> 32.0; 11.4 -> 11.9; 57.1 -> 57.5 | Helped |
| add low-frequency enhancement to spatial token embeddings (ablation) | SIM with [CLS] and spatial tokens · ADE20K-150, ADE20K-847, Pascal Context-59 | mIoU: 32.6 -> 32.9; 13.0 -> 13.3; 58.1 -> 58.6 | Helped |
| use SIM layers 12, 18, 24 and kernels 9, 7, 3 rather than layers 12, 18, 24 and kernels 11, 9, 7 (ablation) | SIM · ADE20K-150, ADE20K-847, Pascal Context-59 | mIoU: 33.0 -> 33.5; 13.4 -> 14.0; 58.4 -> 59.3 | Mixed |
| replace background tokens with original-image [CLS] embeddings rather than random noise (ablation) | Contextual Shift · ADE20K-150, ADE20K-847, Pascal Context-59 | mIoU: 32.3 -> 33.5; 13.0 -> 14.0; 57.8 -> 59.3 | Helped |
| replace background tokens with original-image [CLS] embeddings rather than original background (ablation) | Contextual Shift · ADE20K-150, ADE20K-847, Pascal Context-59 | mIoU: 32.3 -> 33.5; 13.1 -> 14.0; 58.0 -> 59.3 | Helped |
| replace background tokens with original-image [CLS] embeddings rather than learnable prompts (ablation) | Contextual Shift · ADE20K-150, ADE20K-847, Pascal Context-59 | mIoU: 33.1 -> 33.5; 13.2 -> 14.0; 58.8 -> 59.3 | Mixed |
| apply Contextual Shift at layers 1, 3, 5, 7, 9 with 30% replacement rather than at layers 11, 13, 15, 17, 19 (ablation) | Contextual Shift · ADE20K-150, ADE20K-847, Pascal Context-59 | mIoU: 32.2 -> 33.5; 12.0 -> 14.0; 59.1 -> 59.3 | Mixed |
| increase Contextual Shift replacement from 10% to 40% (ablation) | Contextual Shift at layers 1, 3, 5, 7, 9 · ADE20K-150, ADE20K-847, Pascal Context-59 | mIoU: 33.5 -> 32.9; 13.6 -> 13.6; 59.4 -> 59.3 | Mixed |
Surprising findings
- Replacing background with a learnable prompt improves ADE-150 and PC-59 but degrades ADE-847, suggesting overfitting to seen categories.
- Applying contextual shift at layers after the 11th degrades performance, implying the biased background is already encoded in shallow layers.
Limitations
- SG-IoU relies on manually determined category hierarchies and a balance factor, which may be subjective.
- Gains over SAN on ViT-B/16 and ViT-L/14 are moderate on some benchmarks, e.g., ADE-847 at ViT-B/16 (10.8 vs 10.1).
Experimental setup
Datasets: COCO-Stuff COCO Captions masked-image dataset ADE20K-150 ADE20K-847 Pascal VOC Pascal Context-59 Pascal Context-459
Benchmarks: ADE20K-150 ADE20K-847 Pascal VOC Pascal Context-59 Pascal Context-459
Metrics: mIoU SG-IoU
Baselines: OVSeg MAFT SAN SimSeg ODISE OpenSeg LSeg+ MaskCLIP Group-VIT
Builds on: Mask2Former OVSeg CLIP SAN
Closest papers by meaning
- Dual Semantic Guidance for Open Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.91)
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.90)
- CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation (CVPR 2024 · similarity 0.90)
- Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation (CVPR 2024 · similarity 0.90)
- Training-free Open-Vocabulary Semantic Segmentation via Diverse Prototype Construction and Sub-region Matching (AAAI 2025 · similarity 0.89)
- Understanding Fine-tuning CLIP for Open-vocabulary Semantic Segmentation in Hyperbolic Space (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.