Research map / Detection and segmentation / Open-set and few-shot detection
SANSA: Unleashing the Hidden Semantics in SAM2 for Few-Shot Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts a frozen SAM2 (Hiera-L) with lightweight AdaptFormer bottleneck adapters in its last two encoder stages, trained with a propagate-across-targets clip objective, to turn its memory-attention tracking into semantic few-shot segmentation and promptable FSS.
Problem
Few-shot segmentation requires matching semantically related objects across images and producing accurate masks; SAM2's features are tuned for instance tracking and perform poorly on high semantic-shift benchmarks.
Approach
Reinterpret SAM2's prompt-and-propagate memory mechanism as semantic tracking across a pseudo-video of reference and target images, and disentangle latent semantics in SAM2 features via small adapters trained with multi-target propagation where predicted targets become pseudo-references.
Key results
- Strict 1-shot: SANSA reaches 48.8 mIoU on LVIS-92i (vs 35.2 GF-SAM), 60.2 on COCO-20i (vs 58.7 GF-SAM), and 91.4 on FSS-1000.
- Generalist setting with LVIS training: 53.2 1-shot on LVIS-92i, surpassing SegIC (47.8) with a 250M-parameter model.
- Annotation study: SANSA annotates 10,000 images in 16 minutes versus 38-77 minutes for competitors, with highest COCO and LVIS mIoU on fold 0.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| adapt SAM2 with SANSA adapters | SAM2 Hiera-Large · LVIS-92ᶦ 1-shot | mIoU: +13.6% | Helped |
| adapt SAM2 with SANSA adapters | SAM2 Hiera-Large · COCO-20ᶦ 1-shot | mIoU: +1.5% | Helped |
| adapt SAM2 with SANSA adapters | SAM2 Hiera-Large · FSS-1000 1-shot | mIoU: +3.4% | Helped |
| adapt SAM2 with SANSA adapters | SAM2 Hiera-Large · COCO-20ᶦ 5-shot | mIoU: -2.5% | Hurt |
| replace SAM2 adapters with full fine-tuning (ablation) | SAM2 on COCO-20ᶦ · COCO-20ᶦ | mIoU: 60.2 -> 51.6 | Hurt |
| replace SAM2 adapters with decoder fine-tuning (ablation) | SAM2 on COCO-20ᶦ · COCO-20ᶦ | mIoU: 60.2 -> 52.1 | Hurt |
| replace AdaptFormer with MONA (ablation) | SAM2 on COCO-20ᶦ · COCO-20ᶦ | mIoU: 60.2 -> 56.9 | Hurt |
| increase AdaptFormer bottleneck to 0.8× channel dimension (ablation) | SAM2 on COCO-20ᶦ · COCO-20ᶦ | mIoU: 60.2 -> 58.2 | Hurt |
| place adapters in stages 0–1 instead of the last two stages (ablation) | SAM2 on COCO-20ᶦ · COCO-20ᶦ | mIoU: 60.2 -> 38.7 | Hurt |
| add sequential propagation during training, using J=3 rather than J=1 (ablation) | SANSA · COCO-20ᶦ 1-shot | mIoU: 57.0 -> 60.2 | Helped |
| increase training sequence length from J=3 to J=4 (ablation) | SANSA · COCO-20ᶦ 1-shot | mIoU: 60.2 -> 59.8 | Hurt |
| increase training sequence length from J=3 to J=4 (ablation) | SANSA · LVIS-92ᶦ 1-shot | mIoU: 49.6 -> 49.6 | No clear effect |
Surprising findings
- Frozen SAM2 gets 65.8% mIoU with a linear probe on COCO fold 0 but only 28.9% in few-shot segmentation, suggesting semantics are present but entangled.
- Replacing SAM with SAM2 in existing SAM-based pipelines (Matcher, GF-SAM, VRP-SAM) yields negligible gains.
Limitations
- Each object is segmented independently, so multi-class few-shot segmentation and cross-instance context are not exploited.
- Frozen SAM2 performs poorly on FSS without adaptation, so the method depends on base-class training for semantic structure.
Experimental setup
Datasets: COCO-20i FSS-1000 LVIS-92i PASCAL-Part PACO-Part ADE20K PACO
Benchmarks: LVIS-92i COCO-20i FSS-1000 PASCAL-Part PACO-Part
Metrics: mIoU
Baselines: SAM 2 PerSAM Matcher GF-SAM AMFormer HMNet VRP-SAM SegIC DiffewS SegGPT SINE Painter
Builds on: SAM2 AdaptFormer Matcher GF-SAM
Closest papers by meaning
- Bridge the Points: Graph-based Few-shot Segment Anything Semantically (NeurIPS 2024 · similarity 0.90)
- Semantic-aware SAM for Point-Prompted Instance Segmentation (CVPR 2024 · similarity 0.88)
- SAM 2: Segment Anything in Images and Videos (ICLR 2025 · similarity 0.88)
- Segment Anything without Supervision (NeurIPS 2024 · similarity 0.88)
- Boosting Segment Anything Model Towards Open-Vocabulary Learning (AAAI 2025 · similarity 0.88)
- SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.