Research map / Detection and segmentation / Open-set and few-shot detection
VRP-SAM: SAM with Visual Reference Prompt
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VRP-SAM, which adds a meta-learning-style Visual Reference Prompt encoder to SAM so annotated reference images (point, scribble, box, or mask) guide segmentation of matching objects in target images.
Problem
SAM relies on per-image geometric prompts (points, boxes, coarse masks), which is laborious across many images, and sampling-based pseudo-mask prompting for visual reference segmentation is sensitive to pseudo-mask quality and generalizes poorly.
Approach
A VRP encoder (feature augmenter with prototype-enhanced features plus a learnable-query prompt generator) turns annotated reference images directly into prompt embeddings fed to SAM's mask decoder, replacing geometric prompts, while the image encoder is frozen.
Key results
- VRP-SAM reaches 53.9 mIoU on COCO-20i and 71.9 mIoU on PASCAL-5i novel folds with 1.6M learnable parameters.
- On COCO-20i one-shot segmentation, VRP-SAM with mask prompts (53.9) outperforms SAM-based Matcher (52.7) and is comparable to SegGPT (56.1).
- In domain shift COCO-20i to PASCAL-5i, VRP-SAM with scribble annotations beats FP-Trans by 2.8 points and with mask annotations beats DGPNet by 5.8 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add BCE loss to Dice loss (ablation) | VRP-SAM · PASCAL-5i | mean IoU: 70.3 -> 71.9 | Helped |
| add Dice loss to BCE loss (ablation) | VRP-SAM · PASCAL-5i | mean IoU: 70.1 -> 71.9 | Helped |
| use foreground prototype query initialization instead of random initialization (ablation) | VRP-SAM · PASCAL-5i | mean IoU: 71.9 -> 68.2 | Hurt |
| use background prototype query initialization instead of random initialization (ablation) | VRP-SAM · PASCAL-5i | mean IoU: 71.9 -> 62.6 | Hurt |
| initialize half the queries with foreground prototypes and half with background prototypes instead of random initialization (ablation) | VRP-SAM · PASCAL-5i | mean IoU: 71.9 -> 67.4 | Hurt |
| increase the number of visual reference prompts from 1 to 5 (ablation) | VRP-SAM · PASCAL-5i (point, scribble, box, and mask annotations) | mean IoU: +1.2, +1.6, +1.1, +1.0 | Helped |
| vary the number of queries (ablation) | VRP-SAM · PASCAL-5i | mIoU | Helped |
Surprising findings
- Random initialization of the learnable queries outperformed foreground, background, and hybrid prototype initializations.
- Performance gains saturate beyond 50 queries.
Limitations
- Lower performance with point annotations (38.4 mIoU on COCO-20i) than with mask annotations.
- Relies on a pseudo-mask from a training-free approach for target-image enhancement.
Experimental setup
Datasets: PASCAL-5i COCO-20i ImageNet
Benchmarks: PASCAL-5i COCO-20i
Metrics: mIoU
Baselines: Painter SegGPT PerSAM Matcher PFENet BAM HDMNet HSNet CyCTR DCAMA FP-Trans DGPNet
Builds on: Segment Anything Model PFENet Matcher SEEM
Closest papers by meaning
- AoP-SAM: Automation of Prompts for Efficient Segmentation (AAAI 2025 · similarity 0.88)
- SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation (ICLR 2025 · similarity 0.87)
- SAM-REF: Introducing Image-Prompt Synergy during Interaction for Detail Enhancement in the Segment Anything Model (CVPR 2025 · similarity 0.86)
- SAM-I2V: Upgrading SAM to Support Promptable Video Segmentation with Less than 0.2% Training Cost (CVPR 2025 · similarity 0.86)
- Stable Segment Anything Model (ICLR 2025 · similarity 0.86)
- SAM 2: Segment Anything in Images and Videos (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.