Research map / Detection and segmentation / Open-set and few-shot detection

SAM-REF: Introducing Image-Prompt Synergy during Interaction for Detail Enhancement in the Segment Anything Model

CVPR 2025 · Detection and segmentation › Open-set and few-shot detection · approach: Interactive and referring segmentation · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SAM-REF, a lightweight two-stage refiner on top of frozen SAM that re-fuses image and interaction prompts during interaction to improve detail in interactive segmentation while keeping late-fusion efficiency.

Problem

Late-fusion interactive segmentation models such as SAM are efficient because image embeddings are computed once, but they lose fine detail of the prompted target because the image and prompt are not fused at full resolution; early-fusion models are accurate but slow.

Approach

Keep SAM's late-fusion pipeline but add a lightweight refiner after SAM's coarse mask: a Global Fusion Refiner fuses the full-resolution image, dense prompt maps, coarse mask and SAM embeddings with convolutions, a Local Fusion Refiner re-refines a cropped prompt region via RoIAlign, and a Dynamic Selector chooses which mask to paste the local result onto.

Key results

• On DAVIS with ViT-H trained on COCO+LVIS, SAM-REF reaches NoC90 9.10 and NoC95 10.41 (overall), versus SimpleClick's 10.76 NoC90 and FocSAM's 11.67 on NoC90 (table-level comparison).
• On HQSeg-44K validation, SAM-REF (ViT-B, HQSeg-44K training) achieves 90.75 5-mIoU versus 89.45 for HQ-SAM and 88.0 for SAM ViT-H.
• Adds only 0.7MB parameters over SAM ViT-H (636.3 vs 635.6 MB) with 1.67 vs 1.70 FPS.

Measured effects reported in the paper

Surprising findings

• The refiner adds only 0.7MB parameters and negligible latency over SAM, yet gives state-of-the-art NoC90 on most benchmarks.
• Pasting the local refinement onto the previous mask alone performs worse than always pasting onto the global mask, showing the dynamic selector is needed.

Limitations

• Gains on NoC95 are not uniform: SimpleClick is slightly better on DAVIS and GrabCut at NoC95.
• Relies on frozen SAM encoders and a two-stage training schedule, so improvements are bounded by SAM embeddings in the frozen setting.

Experimental setup

Datasets: COCO LVIS HQSeg-44K GrabCut Berkeley DAVIS SBD

Benchmarks: GrabCut Berkeley DAVIS SBD HQSeg-44K

Metrics: NoC90 NoC95 NoF95 SPC mIoU (5-click) latency parameters FPS GPU memory

Baselines: F-BRS CDNet EdgeFlow RITM PseudoClick FocalClick AdaptiveClick SimpleClick InterFormer SAM HQ-SAM FocSAM

Builds on: Segment Anything Model RITM FocalClick FocSAM InterFormer

Closest papers by meaning

• Rethinking Interactive Image Segmentation with Low Latency High Quality and Diverse Prompts (CVPR 2024 · similarity 0.91)
• Stable Segment Anything Model (ICLR 2025 · similarity 0.90)
• FocSAM: Delving Deeply into Focused Objects in Segmenting Anything (CVPR 2024 · similarity 0.90)
• AoP-SAM: Automation of Prompts for Efficient Segmentation (AAAI 2025 · similarity 0.89)
• SAM 2: Segment Anything in Images and Videos (ICLR 2025 · similarity 0.88)
• SAM-CP: Marrying SAM with Composable Prompts for Versatile Segmentation (ICLR 2025 · similarity 0.88)