Research map / Detection and segmentation / Open-set and few-shot detection
Personalize Segment Anything Model with One Shot
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Personalizes SAM to segment a user-specified concept from one reference image+mask, via a training-free PerSAM (target-guided attention, target-semantic prompting) and PerSAM-F (2 learnable scale weights), plus a PerSAM-assisted DreamBooth variant.
Problem
Segment Anything Model cannot automatically segment user-specified personal visual concepts (e.g., a specific pet dog across images) without manual prompting, and it suffers from mask-scale ambiguity for hierarchical objects.
Approach
Derive a location confidence map from SAM features of a one-shot reference mask to produce positive-negative point prompts, inject the confidence map as attention bias in SAM's decoder and the averaged target embedding into prompt tokens, and learn only two mask-scale weights (PerSAM-F) with SAM frozen.
Key results
- PerSAM-F reaches 95.3 mIoU and 77.9 bIoU on PerSeg, +2.4 mIoU over SegGPT (94.3).
- Training-free PerSAM reaches 89.3 mIoU on PerSeg, above Painter (56.4), VP (65.9) and SEEM (87.1).
- On DAVIS 2017, training-free PerSAM surpasses Painter by +32.3% J&F, and PerSAM-F is +0.5% over SegGPT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply scale-aware fine-tuning with two learnable mask weights (ablation) | PerSAM · PerSeg | mIoU: +2.7% | Helped |
| Apply scale-aware fine-tuning with two learnable mask weights (ablation) | PerSAM · PerSeg | bIoU: +5.9% | Helped |
| Add a negative point prompt to the positive location prior (ablation) | PerSAM · PerSeg | mIoU: +3.6% | Helped |
| Add cascaded post-refinement (ablation) | PerSAM · PerSeg | mIoU: +11.4% | Helped |
| Add target-guided attention (ablation) | PerSAM · PerSeg | mIoU: +1.9% | Helped |
| Add target-semantic prompting (ablation) | PerSAM · PerSeg | mIoU: +3.5% | Helped |
| Use scale-aware fine-tuning instead of the training-free variant (ablation) | PerSAM · PerSeg | mIoU: +6.0% | Helped |
| Use prompt tuning modules in the decoder instead of scale-aware fine-tuning (ablation) | PerSAM · PerSeg | accuracy | Hurt |
| Use Adapter modules in the decoder instead of scale-aware fine-tuning (ablation) | PerSAM · PerSeg | accuracy | Hurt |
| Use a bounding box rather than an accurate mask as the reference (ablation) | PerSAM · PerSeg | mIoU | Hurt |
| Use a bounding box rather than an accurate mask as the reference (ablation) | PerSAM-F · PerSeg | mIoU | Hurt |
| Use PerSAM instead of Painter | PerSAM · DAVIS 2017 validation | J&F: +32.3% | Helped |
Surprising findings
- Prompt tuning and Adapter over-fit the one-shot data and degrade accuracy, whereas tuning only two scale weights helps.
Limitations
- Mask-scale ambiguity for hierarchical objects (e.g., hat on teddy bear) causes failures in the training-free version.
- Requires an accurate or rough reference mask as one-shot input; box references reduce performance somewhat.
Experimental setup
Datasets: PerSeg DAVIS 2017 FSS-1000 LVIS-92i PASCAL-Part PACO-Part
Benchmarks: PerSeg DAVIS 2017 FSS-1000 LVIS-92i PASCAL-Part PACO-Part
Metrics: mIoU bIoU J&F
Baselines: Painter Visual Prompting (VP) SEEM SegGPT Matcher DreamBooth
Builds on: Segment Anything Model DreamBooth Visual Prompting
Closest papers by meaning
- Bridge the Points: Graph-based Few-shot Segment Anything Semantically (NeurIPS 2024 · similarity 0.89)
- AoP-SAM: Automation of Prompts for Efficient Segmentation (AAAI 2025 · similarity 0.89)
- SAM 2: Segment Anything in Images and Videos (ICLR 2025 · similarity 0.89)
- SAM-PARSER: Fine-Tuning SAM Efficiently by Parameter Space Reconstruction (AAAI 2024 · similarity 0.88)
- SAM-REF: Introducing Image-Prompt Synergy during Interaction for Detail Enhancement in the Segment Anything Model (CVPR 2025 · similarity 0.88)
- Segment Anything without Supervision (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.