Research map / Detection and segmentation / Open-set and few-shot detection
Stable Segment Anything Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Stable-SAM makes SAM robust to imprecise box and sparse point prompts by adding a deformable sampling plugin (offsets and amplitudes on image features) plus a dynamic routing plugin, training only ~0.08M parameters with the SAM weights frozen.
Problem
SAM's mask decoder produces unstable and inaccurate masks when given low-quality prompts such as noisy boxes or one or three points, with attention drifting to background or object parts.
Approach
Keep SAM frozen and calibrate only its token-to-image cross-attention by learning deformable sampling offsets and feature amplitudes for image features, with a prompt-quality-conditioned router that toggles between deformable and regular sampling, trained with a robust training strategy over mixed-quality prompts and synthetic occlusions.
Key results
- With noisy box prompts on four HQ datasets, Stable-SAM (1 epoch) reaches 82.3 mIoU and 82.3 mSF versus 48.8 mIoU and 39.5 mSF for SAM ViT-L.
- With single-point prompts on four HQ datasets, Stable-SAM reaches 76.9 mIoU versus 43.3 for SAM and 81.8 for HQ-SAM with 1 point not reached; HQ-SAM gets 43.2 at 1 point.
- Stable-SAM adds only 0.08M learnable parameters and improves COCO noisy-box mAP from 27.3 (SAM) to 44.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DSP, DRP, and RTS | SAM ViT-Large · Four HQ datasets, noisy box | mIoU: 48.8 -> 82.3 | Helped |
| Add DSP, DRP, and RTS | SAM ViT-Large · Four HQ datasets, 1 point | mIoU: 43.3 -> 76.9 | Helped |
| Add DSP, DRP, and RTS | SAM ViT-Large · Four HQ datasets, 3 points | mIoU: 78.7 -> 84.0 | Helped |
| Add DSP, DRP, and RTS | SAM ViT-Large · MS COCO, N-Box (0.5-0.6) | mAP: 31.9 -> 44.8 | Helped |
| Add DSP, DRP, and RTS | SAM ViT-Large · SGinW, N-Box (0.5-0.6) | mAP: 33.6 -> 43.3 | Helped |
| Add DSP without RTS or DRP (ablation) | SAM ViT-Large · Four HQ datasets, noisy box | mIoU: 48.8 -> 69.9 | Helped |
| Add DSP without RTS or DRP (ablation) | SAM ViT-Large · Four HQ datasets, 1 point | mIoU: 43.3 -> 46.8 | Helped |
| Add RTS to DSP without DRP (ablation) | SAM ViT-Large · Four HQ datasets, noisy box | mIoU: 69.9 -> 81.7 | Helped |
| Add RTS to DSP without DRP (ablation) | SAM ViT-Large · Four HQ datasets, 1 point | mIoU: 46.8 -> 75.9 | Helped |
| Add DRP to DSP and RTS (ablation) | SAM ViT-Large · Four HQ datasets, 1 point | mIoU: 75.9 -> 76.9 | Helped |
| Replace DSP with conventional deformable attention and finetune the mask decoder (ablation) | SAM ViT-Large · MS COCO, N-Box (0.5-0.6) | mAP: 44.8 -> 25.7 | Hurt |
| Replace DSP with spatial attention (ablation) | SAM ViT-Large · Four HQ datasets, noisy box | mIoU: 82.3 -> 69.3 | Hurt |
Surprising findings
- Conventional deformable attention fine-tuned into SAM's decoder generalizes worse on MS COCO than the original SAM.
- HQ-SAM with 1 point prompt drops to about 43 mIoU, roughly matching SAM, despite its box-prompt gains.
Limitations
- Gains on one-point prompts remain limited in absolute terms and rely on robust training with synthetic occlusion data.
- Stability is measured with a metric based on agreement among prompts rather than ground truth, and evaluation focuses on box and point prompts.
Experimental setup
Datasets: HQSeg-44K SA-1B COCO SGinW MESS
Benchmarks: DIS ThinObject-5K COIFT HR-SOD MS COCO SGinW MESS
Metrics: mIoU mBIoU mSF mAP mAP50
Baselines: SAM HQ-SAM DT-SAM PT-SAM FocalNet-L-DINO
Builds on: Segment Anything Model Deformable DETR HQ-SAM
Closest papers by meaning
- AoP-SAM: Automation of Prompts for Efficient Segmentation (AAAI 2025 · similarity 0.91)
- SAM-REF: Introducing Image-Prompt Synergy during Interaction for Detail Enhancement in the Segment Anything Model (CVPR 2025 · similarity 0.90)
- SAM 2: Segment Anything in Images and Videos (ICLR 2025 · similarity 0.89)
- BA-SAM: Scalable Bias-Mode Attention Mask for Segment Anything Model (CVPR 2024 · similarity 0.89)
- SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation (ICLR 2025 · similarity 0.89)
- SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.