Research map / Detection and segmentation / Open-set and few-shot detection
SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAM-CP, which equips frozen SAM patches with two composable prompts (semantic labeling and instance merging) unified in an affinity-based query decoder for semantic, instance, and panoptic segmentation in closed and open vocabulary settings.
Problem
Applying SAM to semantic-aware segmentation (semantic, instance, panoptic) is hard because SAM patches lack semantic labels and often over-segment instances into sub-patches.
Approach
Two composable prompts over SAM patches: Type-I judges whether a patch matches a text class, Type-II judges whether two same-class patches belong to the same instance. A unified query-based affinity decoder (DETR-style queries over patch keys, with dynamic cross-attention and affinity refinement) implements both efficiently, avoiding quadratic pairwise enumeration.
Key results
- In open-vocabulary COCO→ADE20K, SAM-CP reaches 27.2 PQ, 17.0 AP and 31.8 mIoU, above FCCLIP (26.8 PQ) and FrozenSeg (25.9 PQ).
- Closed-domain COCO-Panoptic with Swin-L reaches 52.7 PQ, 45.2 AP and 61.8 mIoU.
- On COCO→Cityscapes open-vocabulary, SAM-CP reports 41.0 PQ and 29.3 AP, the best instance result in the table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Mask DINO proposals to SAM proposals (ablation) | SAM-CP · COCO, 36 epochs | PQ: 48.6 -> 51.4 | Helped |
| remove mask focal loss from the loss and label assignment (ablation) | SAM-CP · COCO→ADE20K, 12 epochs | PQ: 27.2 -> 0.6 | Hurt |
| remove Dice loss from the loss and label assignment (ablation) | SAM-CP · COCO→ADE20K, 12 epochs | PQ: 27.2 -> 23.8 | Hurt |
| remove mask focal loss from label assignment (ablation) | SAM-CP · COCO→ADE20K, 12 epochs | PQ: 27.2 -> 26.5 | Hurt |
| remove Dice loss from label assignment (ablation) | SAM-CP · COCO→ADE20K, 12 epochs | PQ: 27.2 -> 26.6 | Hurt |
| remove box and gIoU costs from label assignment (ablation) | SAM-CP · COCO→ADE20K, 12 epochs | PQ: 27.2 -> 25.9 | Hurt |
| remove dynamic cross-attention (ablation) | SAM-CP · COCO, 12 epochs | PQ: 47.0 -> 43.5 | Hurt |
| remove affinity refinement (ablation) | SAM-CP · COCO, 12 epochs | PQ: 47.0 -> 44.1 | Hurt |
| remove MaskRoI (ablation) | SAM-CP · COCO, 12 epochs | PQ: 47.0 -> 44.1 | Hurt |
| remove query enhancement (ablation) | SAM-CP · COCO, 12 epochs | PQ: 47.0 -> 44.8 | Hurt |
| remove background self-affinity (ablation) | SAM-CP · COCO, 12 epochs | PQ: 47.0 -> 45.2 | Hurt |
| fuse learnable and CLIP classifiers instead of using either alone (ablation) | SAM-CP · COCO→ADE20K | PQ: 17.5 -> 27.2; 16.9 -> 27.2 | Helped |
Surprising findings
- Closed-domain results trail Mask DINO and other specialized models; the authors attribute this to SAM's high missing rate (16.7% of instances lack a proposal with IoU above 0.5), which the merging cannot recover.
- A SAM2 variant reuses SAM-CP weights without retraining and reaches 27.9 PQ while running faster.
Limitations
- Performance is bounded by SAM patch quality; missed small, occluded, low-contrast, or cluttered objects cannot be recovered.
- Closed-domain accuracy lags strong specialized segmenters such as Mask DINO.
Experimental setup
Datasets: COCO-Panoptic ADE20K Cityscapes Pascal Part
Benchmarks: COCO-Panoptic ADE20K Cityscapes Pascal Part
Metrics: PQ AP mIoU SQ RQ
Baselines: FCCLIP FrozenSeg OpenSeeD X-Decoder ODISE Mask2Former MaskFormer Mask DINO SEEM Semantic-SAM
Builds on: Segment Anything DETR Mask DINO FC-CLIP GLIP
Closest papers by meaning
- Semantic-aware SAM for Point-Prompted Instance Segmentation (CVPR 2024 · similarity 0.90)
- From SAM to CAMs: Exploring Segment Anything Model for Weakly Supervised Semantic Segmentation (CVPR 2024 · similarity 0.90)
- Effective SAM Combination for Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.89)
- Bridge the Points: Graph-based Few-shot Segment Anything Semantically (NeurIPS 2024 · similarity 0.89)
- OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts (NeurIPS 2025 · similarity 0.89)
- AoP-SAM: Automation of Prompts for Efficient Segmentation (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.