Research map / Detection and segmentation / Open-set and few-shot detection
Effective SAM Combination for Open-Vocabulary Semantic Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ESC-Net, a one-stage open-vocabulary semantic segmentation model that feeds CLIP image-text correlation-derived pseudo prompts into pretrained SAM decoder blocks plus a Vision-Language Fusion module.
Problem
Two-stage open-vocabulary segmentation pipelines that pair SAM/MaskFormer mask proposals with CLIP are computationally expensive and memory-inefficient, while one-stage correlation-based methods produce coarse masks due to CLIP's weak spatial localization.
Approach
Generate pseudo points and masks from CLIP image-text correlation maps, embed them via SAM's prompt encoder, and pass CLIP features through pretrained SAM transformer decoder blocks (without the SAM image encoder) for class-agnostic spatial aggregation, then refine correlation maps with a VLF module.
Key results
- With CLIP ViT-L/14, ESC-Net reports A-847 18.1 mIoU and PC-459 27.0 mIoU, versus 16.0 and 23.8 for CAT-Seg (+2.1 and +3.2 over the second-best).
- With CLIP ViT-B/16, ESC-Net reports A-150 31.8+3.8 gain over CAT-Seg (35.6 vs 31.8) and PAS-20 97.3 vs 94.6 for CAT-Seg.
- Ablation: using pretrained SAM blocks raises A-847 from 4.8 (without SAM) to 18.1 and from 5.9 (random init SAM) to 18.1; efficiency table shows 451.3M params and 0.76 s inference versus CAT-Seg's 433.7M and 0.78 s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add pretrained SAM blocks and pseudo prompts (ablation) | ESC-Net · ADE20K A-847 | mIoU: 4.8 -> 18.1 | Helped |
| use pretrained SAM blocks instead of randomly initialized SAM blocks (ablation) | ESC-Net · ADE20K A-847 | mIoU: 5.9 -> 18.1 | Helped |
| use points and masks together as pseudo prompts (ablation) | ESC-Net · PASCAL-Context PC-459 | mIoU: 15.2 -> 18.1 | Helped |
| use points and masks instead of bounding boxes only (ablation) | ESC-Net · PASCAL-Context PC-459 | mIoU: 14.3 -> 18.1 | Helped |
| add bounding boxes to points and masks (ablation) | ESC-Net · PASCAL-Context PC-459 | mIoU: 18.1 -> 17.6 | Hurt |
Surprising findings
- Random-initialized SAM blocks yield much lower gains than pretrained SAM blocks (A-847 5.9 vs 18.1), indicating the benefit depends on SAM pretraining rather than the architecture alone.
Limitations
- Operates on low-resolution feature levels, limiting mask resolution and boundary precision.
- Memory requirements grow with number of classes due to image-text correlation maps.
Experimental setup
Datasets: COCO-Stuff
Benchmarks: ADE20K (A-847, A-150) PASCAL-Context (PC-459, PC-59) PASCAL-VOC (PAS-20, PAS-20b)
Metrics: mIoU
Baselines: CAT-Seg MAFT+ EBSeg SCAN SED OVSeg ZegFormer ZSseg SAN ODISE
Builds on: CAT-Seg Segment Anything Model CLIP Swin Transformer
Closest papers by meaning
- Exploring Simple Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.92)
- USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation (CVPR 2024 · similarity 0.91)
- Open-Vocabulary Semantic Segmentation with Image Embedding Balancing (CVPR 2024 · similarity 0.91)
- Emergent Open-Vocabulary Semantic Segmentation from Off-the-shelf Vision-Language Models (CVPR 2024 · similarity 0.91)
- Mask-Adapter: The Devil is in the Masks for Open-Vocabulary Segmentation (CVPR 2025 · similarity 0.91)
- CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.