Research map / Detection and segmentation / Open-set and few-shot detection
Semantic-aware SAM for Point-Prompted Instance Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SAPNet, an end-to-end point-prompted instance segmentation framework that uses SAM-generated mask proposals selected by multiple instance learning with point distance guidance and box mining to supervise a SOLOv2 segmenter.
Problem
Point-prompted instance segmentation with SAM is class-agnostic and semantically ambiguous, so SAM proposals do not reliably correspond to the annotated category's full object, and MIL-based selection suffers group and local issues.
Approach
Uses MIL-based proposal selection over SAM point-prompted proposals, augmented with a point-distance penalty between same-class annotated points, positive/negative proposal bags with a refinement module, and a box mining strategy, so that selected masks supervise an end-to-end SOLOv2 branch.
Key results
- SAPNet with ResNet-50 at 1x schedule reaches 31.2 AP on COCO val, versus 24.6 AP for SAM-top1 supervision and 26.8 AP for single-MIL baseline.
- Under a 3x schedule with ResNet-101 and multi-scale training, SAPNet reaches 34.6 AP on COCO, and it reports a 13.5 AP gain over BESTIE at 1x.
- On VOC2012 test, SAPNet with ResNet-101 reports 64.8 AP50, a 7.7 AP improvement over AttnShift as stated in the text.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace SAM-top1 mask supervision with end-to-end SAPNet | SOLOv2 with point prompts · COCO | AP: 24.6 -> 31.2 | Helped |
| train end-to-end instead of using two-stage training (ablation) | SAPNet · COCO | AP: 30.2 -> 31.2 | Helped |
| add Point Distance Guidance to MIL1 (ablation) | SAPNet proposal selection · COCO | AP: 26.8 -> 27.5 | Helped |
| add a second MIL selection module (ablation) | SAPNet proposal selection · COCO | AP: 27.5 -> 27.7 | Helped |
| add PNPG positive and negative sample sets (ablation) | SAPNet refinement · COCO | AP: 27.7 -> 29.7 | Helped |
| add Box Mining Strategy (ablation) | SAPNet refinement · COCO | AP: 29.7 -> 30.8 | Helped |
| add multi-mask proposal supervision (ablation) | SAPNet segmentation branch · COCO | AP: 30.8 -> 31.2 | Helped |
| use BMS thresholds Tmin1=0.6 and Tmin2=0.3 (ablation) | SAPNet · COCO | AP: 30.9; 30.7; 31.2; 30.8; 31.0; 30.7 | Mixed |
| add PPG positive proposals (ablation) | SAPNet PNPG · COCO | AP: 29.3 -> 29.8 | Helped |
| add NPG negative proposals (ablation) | SAPNet PNPG · COCO | AP: 29.3 -> 30.7 | Helped |
| add both PPG and NPG rather than neither (ablation) | SAPNet PNPG · COCO | AP: 29.3 -> 31.2 | Helped |
| replace Single-MIL with SAPNet proposal selection | proposal boxes · COCO training set | mIoUbox: 63.8 -> 69.1 | Helped |
Surprising findings
- The text states that SAM's top-scoring mask often corresponds to a part (e.g., clothes) rather than the whole object, so relying on SAM's score is insufficient.
Limitations
- Relies on SAM proposal quality and SAM's class-agnostic outputs, so semantic ambiguity is only mitigated by MIL selection.
- Group and local issues are only partly resolved; the paper notes residual part-level errors (Gap 0.131 vs 0.199 for single-MIL).
Experimental setup
Datasets: MS COCO 2017 VOC2012 SBD
Benchmarks: MS COCO 2017 val VOC2012 test
Metrics: mAP AP50 AP75 APs APm APl AP25 mIoU box
Baselines: SAM-SOLOv2 MIL-SOLOv2 AttnShift BESTIE WISE-Net BoxInst DiscoBox Mask R-CNN
Builds on: SAM WSDDN P2BNet PCL SOLOv2
Closest papers by meaning
- SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation (ICLR 2025 · similarity 0.90)
- Bridge the Points: Graph-based Few-shot Segment Anything Semantically (NeurIPS 2024 · similarity 0.90)
- P2Seg: Pointly-supervised Segmentation via Mutual Distillation (ICLR 2024 · similarity 0.90)
- From SAM to CAMs: Exploring Segment Anything Model for Weakly Supervised Semantic Segmentation (CVPR 2024 · similarity 0.90)
- Curriculum Point Prompting for Weakly-Supervised Referring Image Segmentation (CVPR 2024 · similarity 0.89)
- Effective SAM Combination for Open-Vocabulary Semantic Segmentation (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.