Research map / Detection and segmentation / Open-set and few-shot detection
Unlocking the Power of SAM 2 for Few-Shot Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts SAM 2 to few-shot segmentation by generating pseudo query memories from DINOv2-based prior masks, refining them iteratively, and suppressing background in cross attention.
Problem
Few-shot segmentation must generalize to novel classes without overfitting base classes; SAM 2's same-object video matching is incompatible with the different-object support-query matching required by FSS.
Approach
Encode DINOv2-derived prior masks on query features as pseudo query memories so SAM 2's same-object matching applies, then add Iterative Memory Refinement to fuse more query foreground into the discriminative memory and Support-Calibrated Memory Attention to suppress query background in memory attention.
Key results
- 1-shot mIoU of 81.0% on PASCAL-5i, 4.2 points above the best baseline FounFSS (76.8%).
- 1-shot mIoU of 62.3% on COCO-20i, compared with 60.4% for VRP-SAM.
- Component ablation: SAM 2 alone reaches 69.4% mIoU; adding PPG gives 78.2%, and adding IMR and SCMA gives 81.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add PPG to enable pseudo-query memory matching (ablation) | SAM 2-S for FSS · PASCAL-5ᶦ 1-shot | mIoU (Mean): 69.4% to 78.2% | Helped |
| Add IMR to complement query foreground features (ablation) | FSSAM with PPG · PASCAL-5ᶦ 1-shot | mIoU (Mean): 78.2 to 79.6 | Helped |
| Add SCMA to suppress unexpected background memory features (ablation) | FSSAM with PPG · PASCAL-5ᶦ 1-shot | mIoU (Mean): 79.0 | Helped |
| Use both IMR and SCMA (ablation) | FSSAM with PPG · PASCAL-5ᶦ 1-shot | mIoU (Mean): 81.0 | Helped |
| Increase IMR iterations from 1 to 3 (ablation) | FSSAM · PASCAL-5ᶦ 1-shot | mIoU (Mean): 79.3% to 81.0% | Helped |
| Increase IMR iterations from 3 to 4 (ablation) | FSSAM · PASCAL-5ᶦ 1-shot | mIoU (Mean): 81.0 to 80.8 | Hurt |
| Replace DINOv2-B with DINOv2-L while using SAM 2-S (ablation) | FSSAM · PASCAL-5ᶦ 1-shot | mIoU (Mean): 81.0 to 80.6 | Hurt |
| Replace SAM 2-S with SAM 2-L while using DINOv2-B (ablation) | FSSAM · PASCAL-5ᶦ 1-shot | mIoU (Mean): 81.0 to 81.5 | Helped |
| Fine-tune the model rather than use it without fine-tuning (ablation) | FSSAM · PASCAL-5ᶦ 1-shot | mIoU (Mean): 58.5 to 81.0 | Helped |
| Fine-tune SAM 2 rather than use it without fine-tuning (ablation) | SAM 2 baseline · PASCAL-5ᶦ 1-shot | mIoU (Mean): 44.7 to 69.4 | Helped |
| Use FSSAM instead of the strongest reported baseline | FSSAM · PASCAL-5ᶦ 1-shot | mIoU (Mean): 76.8% to 81.0% | Helped |
| Use FSSAM instead of the strongest reported baseline | FSSAM · COCO-20ᶦ 1-shot | mIoU (Mean): 60.4 to 62.3 | Helped |
Surprising findings
- Upgrading DINOv2 from base to large did not improve PASCAL-5i mIoU (80.6% vs 81.0%), so larger foundation models are not always better.
- Without fine-tuning, SAM 2 scores only 44.7% mIoU and FSSAM scores 58.5%, so fine-tuning is required despite the foundation model.
Limitations
- Relies on pseudo mask prompts; when FG and BG are very similar, the prompt can miss the real foreground entirely.
- Error bars show large variance on COCO-20i (std 0.9 mIoU across seeds), and results depend on episode sampling.
- Inference is slow relative to lighter methods, with about 2.8 FPS and 132M total parameters.
Experimental setup
Datasets: PASCAL-5i COCO-20i
Benchmarks: PASCAL-5i COCO-20i LVIS-92i PASCAL-Part
Metrics: mIoU FB-IoU
Baselines: HMNet AMNet AENet HDMNet PAM MIANet SCCAN ABCNet Matcher VRP-SAM GF-SAM FounFSS
Builds on: SAM 2 DINOv2 AENet SCCAN
Closest papers by meaning
- Foreground-Covering Prototype Generation and Matching for SAM-Aided Few-Shot Segmentation (AAAI 2025 · similarity 0.90)
- Adaptive FSS: A Novel Few-Shot Segmentation Framework via Prototype Enhancement (AAAI 2024 · similarity 0.88)
- SAM-Aware Graph Prompt Reasoning Network for Cross-Domain Few-Shot Segmentation (AAAI 2025 · similarity 0.87)
- SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost (CVPR 2025 · similarity 0.87)
- SAM 2: Segment Anything in Images and Videos (ICLR 2025 · similarity 0.87)
- Hybrid Mamba for Few-Shot Segmentation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.