Research map / Detection and segmentation / CNN features
SAMRefiner: Taming Segment Anything Model for Universal Mask Refinement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts the Segment Anything Model to refine coarse pre-existing masks via noise-tolerant multi-prompt excavation (distance-guided points, context-aware elastic boxes, Gaussian-style masks), split-then-merge for semantic masks, and an optional self-boosted IoU adaption step (SAMRefiner++).
Problem
Coarse or noisy pseudo masks and human annotations hinder their use as training data; SAM is prompt-driven, and it is unclear how to derive reliable prompts for SAM from coarse masks alone.
Approach
Mine multiple complementary prompts (distance-guided points, CEBox, Gaussian-style mask prompts) from a coarse mask so they collaborate to be robust to defects; handle multi-object semantic masks via split-then-merge; and adapt SAM's IoU head with a LoRA-style adaptor trained with a ranking loss supervised by coarse-mask IoU, without extra annotations.
Key results
- On DAVIS-585, SAMRefiner++ with all prompts reaches 87.1 IoU vs 81.4 for the coarse mask and 82.9 for CRM.
- On COCO train5K, SAMRefiner improves PointWSSIS (1% labels) pseudo-mask AP from 15.1 to 25.4 (+10.3).
- Refinement on COCO val with LVIS annotations improves MaskRCNN (RN50) AP mask from 39.8 to 45.3, and is reported to take about 0.6 h vs 1.0-3.4 h for baselines on COCO train5K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| refine PointWSSIS pseudo masks | PointWSSIS on COCO train5K with 1% annotations · COCO train5K | AP mask: 15.1 -> 25.4 | Helped |
| refine MaskCLIP pseudo masks | MaskCLIP on PASCAL VOC 2012 · PASCAL VOC 2012 train | mIoU: 47.8 -> 57.3 | Helped |
| use SAMRefiner for mask refinement | DAVIS-585 coarse masks · DAVIS-585 | IoU: 81.4 -> 87.1 | Helped |
| add IoU-head adaptation (ablation) | SAMRefiner on DAVIS-585 using point prompts · DAVIS-585 | Top-1 Acc: 51.5 / 62.1 | Helped |
| use point, box, and mask prompts together instead of point alone (ablation) | SAMRefiner on DAVIS-585 · DAVIS-585 | IoU: 53.7 -> 86.9 | Helped |
| use mask prompt alone (ablation) | SAM on DAVIS-585 · DAVIS-585 | IoU: 81.4 -> 37.3 | Hurt |
| sample the positive point by maximum distance from background instead of randomly (ablation) | SAMRefiner on DAVIS-585 · DAVIS-585 | IoU: 37.8 -> 53.7 | Helped |
| sample the point at the box center instead of randomly (ablation) | SAMRefiner on DAVIS-585 · DAVIS-585 | IoU: 37.8 -> 22.2 | Hurt |
| use context-aware elastic boxes instead of tight boxes (ablation) | SAMRefiner on COCO with PointWSSIS coarse masks · COCO | AP box: 36.7 -> 38.2 | Helped |
| add split-then-merge (ablation) | SAMRefiner with MaskCLIP masks · PASCAL VOC | mIoU: 51.1 -> 57.3 | Helped |
| add split-then-merge (ablation) | SAMRefiner with CLIP-ES masks · PASCAL VOC | mIoU: 79.1 -> 79.3 | Helped |
| use the Gaussian-style mask prompt rather than SAM-predicted mask logits in later cascade steps (ablation) | SAMRefiner cascaded refinement · DAVIS-585 | mask quality | Helped |
Surprising findings
- Directly using the coarse mask as the SAM mask prompt performs very poorly (37.3 IoU on DAVIS-585), unlike point and box prompts.
- Selecting the best of SAM's three multi-mask outputs outperforms the single-mask token, and the IoU head is poorly calibrated for target datasets.
Limitations
- Fails when the coarse mask is extremely inaccurate, making refinement ill-posed.
- Underperforms on objects with intricate structures, inherited from SAM's limitations.
- Relies on the SAM IoU head, and the IoU adaption step is optional and only applied on DAVIS-585.
Experimental setup
Datasets: DAVIS-585 MS COCO 2017 PASCAL VOC 2012 LVIS BIG relabeled VOC
Benchmarks: DAVIS-585 COCO val2017 COCO train5K PASCAL VOC 2012 BIG relabeled VOC
Metrics: IoU boundary IoU mask AP boundary AP mIoU top-1 accuracy mBA inference time
Baselines: dense CRF CascadePSP CRM SegRefiner SegFix BPR
Builds on: Segment Anything Model HQ-SAM LoRA
Closest papers by meaning
- Bridge the Points: Graph-based Few-shot Segment Anything Semantically (NeurIPS 2024 · similarity 0.88)
- SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation (ICLR 2025 · similarity 0.87)
- SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost (CVPR 2025 · similarity 0.87)
- SAM-REF: Introducing Image-Prompt Synergy during Interaction for Detail Enhancement in the Segment Anything Model (CVPR 2025 · similarity 0.87)
- Stable Segment Anything Model (ICLR 2025 · similarity 0.87)
- SAM-PARSER: Fine-Tuning SAM Efficiently by Parameter Space Reconstruction (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.