Research map / Detection and segmentation / Open-set and few-shot detection
Rethinking Interactive Image Segmentation with Low Latency High Quality and Diverse Prompts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SegNext, an interactive segmentation model that encodes five visual prompt types (clicks, boxes, polygons, scribbles, masks) as a dense three-channel map fused with image embeddings, giving low latency and high quality.
Problem
Interactive image segmentation needs low latency, high quality, and diverse prompts at once; specialist models recompute the image per prompt, and generalist models like SAM represent visual prompts sparsely and lag in high-quality segmentation.
Approach
Reintroduce dense representation and fusion of visual prompts (common in specialist models) into a generalist architecture: encode all visual prompts as a three-channel dense map added element-wise to the image embedding, followed by self-attention fusion, while keeping text prompts sparse via CLIP and cross-attention. Only clicks are simulated in training; other prompts generalize at test time.
Key results
- Ours (SA×2) trained on COCO+LVIS+HQ reaches 91.75 5-mIoU and NoC90 5.32 on HQSeg-44K, versus HQ-SAM 89.85 and SAM 86.16, with 17.6s SAT latency vs HQ-SAM 8.3s.
- On DAVIS with clicks, Ours (SA×2) achieves 5-mIoU 89.85 vs SAM 90.95 and SimpleClick 90.73 (COCO+LVIS-only variant); the COCO+LVIS+HQ variant reaches 91.87.
- Removing dense fusion drops 5-mIoU on HQSeg-44K from 85.71 to 65.34, indicating dense fusion is the key design choice.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune SegNext on HQSeg-44K | SegNext (SA×2) · HQSeg-44K | 5-mIoU: 85.71 -> 91.75 | Helped |
| Use two dense-fusion self-attention blocks instead of one (ablation) | SegNext · HQSeg-44K | 5-mIoU: 85.41 -> 85.71 | Helped |
| Add dense-fusion self-attention blocks (ablation) | SegNext · HQSeg-44K | 5-mIoU: 65.34 -> 85.71 | Helped |
| Represent each click as a disk rather than a point (ablation) | SegNext · HQSeg-44K | 5-mIoU: 83.72 -> 85.71 | Helped |
| Use a dense-map visual prompt representation | SegNext · HQSeg-44K | 5-mIoU | Helped |
| Use SegNext with scribble prompts instead of SimpleClick | SegNext (SA×2) · DAVIS, scribble | 1-mIoU: 76.63 -> 74.26 | Hurt |
| Use SegNext with polygon prompts instead of SimpleClick | SegNext (SA×2) · DAVIS, polygon | 1-mIoU: 68.16 -> 64.04 | Hurt |
| Use SegNext with box prompts instead of SimpleClick | SegNext (SA×2) · DAVIS, box | 1-mIoU: 69.21 -> 69.08 | No clear effect |
Surprising findings
- SAM fails badly on scribble and box prompts on DAVIS (30.42 and 44.87 1-mIoU) despite strong click performance, supporting the sparse-representation hypothesis.
- The model trained only on clicks generalizes to unseen box, scribble, and polygon prompts without training on them.
Limitations
- Dense visual prompt representation is more costly than sparse representations used by specialist models.
- Text prompt support is tentative and not completely stable; only explored in appendix.
- Only ViT-Base backbone was tested; larger backbones not explored.
Experimental setup
Datasets: COCO LVIS HQSeg-44K DAVIS ssTEM BraTS
Benchmarks: HQSeg-44K DAVIS ssTEM BraTS
Metrics: 5-mIoU NoC90 NoC95 NoF95 1-mIoU 10-mIoU SAT latency
Baselines: RITM FocalClick SimpleClick InterFormer SAM MobileSAM HQ-SAM
Builds on: SimpleClick Segment Anything Model RITM MAE CLIP
Closest papers by meaning
- SAM-REF: Introducing Image-Prompt Synergy during Interaction for Detail Enhancement in the Segment Anything Model (CVPR 2025 · similarity 0.91)
- AoP-SAM: Automation of Prompts for Efficient Segmentation (AAAI 2025 · similarity 0.90)
- Prompting Diffusion Models for Zero-Shot Instance Segmentation (NeurIPS 2026 · similarity 0.89)
- Visual Prompting for Generalized Few-shot Segmentation: A Multi-scale Approach (CVPR 2024 · similarity 0.89)
- Stable Segment Anything Model (ICLR 2025 · similarity 0.88)
- FocSAM: Delving Deeply into Focused Objects in Segmenting Anything (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.