Research map / Detection and segmentation / Open-set and few-shot detection
Interactive and referring segmentation
Segment objects from clicks or language expressions with refinement pipelines.
161 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Open-set and few-shot detection, keyword cluster “images · anomaly · interactive”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 2 of 6 tracked benchmarks.
63 papers in 2023–24 and 55 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 10.7% to 9.9%.
Open questions the data points to
- Berkeley (NoC@90): best went from 1.47 (up to 2024) to 1.43 in 2025 — SAM-REF: Introducing Image-Prompt Synergy during Interaction for Detail Enhancement in the Segment Anything Model (CVPR 2025 · 5 citations)
- DAVIS (NoC@90): best went from 4.77 (up to 2024) to 4.54 in 2025 — SAM-REF: Introducing Image-Prompt Synergy during Interaction for Detail Enhancement in the Segment Anything Model (CVPR 2025 · 5 citations)
What looks settled
- GrabCut (NoC@90): best up to 2024 1.2, best in 2025–26 1.36 from 2 papers, at most 0.2% better, if at all.
- SBD (NoC@90): best up to 2024 3.96, best in 2025–26 4.44 from 2 papers, at most 0.2% better, if at all.
No longer reported by papers here: COCO-Stuff (mIoU, last in 2024).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes (CVPR 2016 · 2,398 citations)
- 2017 · Indoor Scene Parsing With Instance Segmentation, Semantic Labeling and Support Relationship Inference (CVPR 2017 · 38 citations)
- 2018 · Interactive Image Segmentation With Latent Diversity (CVPR 2018 · 222 citations)
- 2019 · Interactive Image Segmentation via Backpropagating Refinement Scheme (CVPR 2019 · 225 citations)
- 2020 · F-BRS: Rethinking Backpropagating Refinement for Interactive Segmentation (CVPR 2020 · 226 citations)
- 2021 · DoDNet: Learning To Segment Multi-Organ and Tumors From Multiple Partially Labeled Datasets (CVPR 2021 · 176 citations)
- 2022 · AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation (NeurIPS 2022 · 199 citations)
- 2023 · GRES: Generalized Referring Expression Segmentation (CVPR 2023 · 182 citations)
- 2024 · VRP-SAM: SAM with Visual Reference Prompt (CVPR 2024 · 73 citations)
- 2025 · Segmenting Maxillofacial Structures in CBCT Volumes (CVPR 2025 · 30 citations)
- 2026 · SAM 3: Segment Anything with Concepts (ICLR 2026 · 13 citations)
Earliest papers here
- The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes (CVPR 2016 · 2,398 citations)
- Interactive Segmentation on RGBD Images via Cue Selection (CVPR 2016 · 30 citations)
- Automated scalable segmentation of neurons from multispectral images (NeurIPS 2016 · 21 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
Berkeley · NoC@90 (lower is better)
10 papers, 2019–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 5.08 | Interactive Image Segmentation via Backpropagating Refinement Scheme (CVPR 2019 · 225 citations) |
| 2020 | 4.34 | F-BRS: Rethinking Backpropagating Refinement for Interactive Segmentation (CVPR 2020 · 226 citations) |
| 2022 | 1.92 | FocalClick: Towards Practical Interactive Image Segmentation (CVPR 2022 · 167 citations) |
| 2024 | 1.47 | FocSAM: Delving Deeply into Focused Objects in Segmenting Anything (CVPR 2024 · 7 citations) |
| 2025 | 1.43 | SAM-REF: Introducing Image-Prompt Synergy during Interaction for Detail Enhancement in the Segment Anything Model (CVPR 2025 · 5 citations) |
DAVIS · NoC@90 (lower is better)
10 papers, 2019–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 8.24 | Interactive Image Segmentation via Backpropagating Refinement Scheme (CVPR 2019 · 225 citations) |
| 2020 | 7.41 | F-BRS: Rethinking Backpropagating Refinement for Interactive Segmentation (CVPR 2020 · 226 citations) |
| 2022 | 6.22 | FocusCut: Diving Into a Focus View in Interactive Segmentation (CVPR 2022 · 67 citations) |
| 2023 | 5.16 | Focused and Collaborative Feedback Integration for Interactive Image Segmentation (CVPR 2023 · 24 citations) |
| 2024 | 4.77 | FocSAM: Delving Deeply into Focused Objects in Segmenting Anything (CVPR 2024 · 7 citations) |
| 2025 | 4.54 | SAM-REF: Introducing Image-Prompt Synergy during Interaction for Detail Enhancement in the Segment Anything Model (CVPR 2025 · 5 citations) |
GrabCut · NoC@90 (lower is better)
9 papers, 2019–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 3.6 | Interactive Image Segmentation via Backpropagating Refinement Scheme (CVPR 2019 · 225 citations) |
| 2020 | 2.72 | F-BRS: Rethinking Backpropagating Refinement for Interactive Segmentation (CVPR 2020 · 226 citations) |
| 2022 | 1.64 | FocusCut: Diving Into a Focus View in Interactive Segmentation (CVPR 2022 · 67 citations) |
| 2023 | 1.46 | Focused and Collaborative Feedback Integration for Interactive Image Segmentation (CVPR 2023 · 24 citations) |
| 2024 | 1.2 | GraCo: Granularity-Controllable Interactive Segmentation (CVPR 2024 · 13 citations) |
SBD · NoC@90 (lower is better)
9 papers, 2019–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 9.78 | Interactive Image Segmentation via Backpropagating Refinement Scheme (CVPR 2019 · 225 citations) |
| 2020 | 7.73 | F-BRS: Rethinking Backpropagating Refinement for Interactive Segmentation (CVPR 2020 · 226 citations) |
| 2022 | 6.22 | FocusCut: Diving Into a Focus View in Interactive Segmentation (CVPR 2022 · 67 citations) |
| 2023 | 5.83 | Focused and Collaborative Feedback Integration for Interactive Image Segmentation (CVPR 2023 · 24 citations) |
| 2024 | 3.96 | GraCo: Granularity-Controllable Interactive Segmentation (CVPR 2024 · 13 citations) |
COCO-Stuff · mIoU (higher is better)
5 papers, 2023–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 47.6 | FreeMask: Synthetic Images with Dense Annotations Make Stronger Segmentation Models (NeurIPS 2023 · 9 citations) |
DAVIS · NoC@85 (lower is better)
5 papers, 2022–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 3.61 | FocalClick: Towards Practical Interactive Image Segmentation (CVPR 2022 · 167 citations) |
| 2024 | 3.26 | Variance-Insensitive and Target-Preserving Mask Refinement for Interactive Image Segmentation (AAAI 2024 · 4 citations) |
What has been tried
124 papers here report 1,357 measured changes (a component added, removed or swapped, compared with a baseline): helped 75%, no clear effect 5%, hurt 17%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add BCE loss to Dice loss | PASCAL-5i · mean IoU | 70.3 -> 71.9 | Helped | VRP-SAM: SAM with Visual Reference Prompt (CVPR 2024 · 73 citations) |
| add BLIP2 keyword generation | COD10K · Fβ | 0.448 -> 0.556 | Helped | Relax Image-Specific Prompt Requirement in SAM: A Single Generic Prompt for Segmenting Camouflaged Objects (AAAI 2024 · 52 citations) |
| Replace sliding-window inference with zoom-out-zoom-in inference | AMOS22 15-organ test split · average Dice | 0.6529 -> 0.7298 | Helped | SegVol: Universal and Interactive Volumetric Medical Image Segmentation (NeurIPS 2024 · 48 citations) |
| add reinforcement learning | CUHK · mIoU | 59.75 -> 68.47 | Helped | AlignSAM: Aligning Segment Anything Model to Open Context via Reinforcement Learning (CVPR 2024 · 32 citations) |
| Add clustering-based slice-to-volume propagation | WORD test · DSC | 78.67 -> 86.19 | Helped | Clustering Propagation for Universal Medical Image Segmentation (CVPR 2024 · 32 citations) |
| Disable left/right mirroring | ToothFairy2 test · DSC | 74.16 -> 80.79 | Helped | Segmenting Maxillofacial Structures in CBCT Volumes (CVPR 2025 · 30 citations) |
| Set K to 10 rather than the other tested values | NPC-170 · Dicesoft | 78.81, 79.48, 80.43, 80.17, 80.08 | Helped | Diversified and Personalized Multi-rater Medical Image Segmentation (CVPR 2024 · 29 citations) |
| Add ICL during SimpleClick fine-tuning | GrabCut · NoC@95 | 2.16 -> 2.00 | Helped | CFR-ICL: Cascade-Forward Refinement with Iterative Click Loss for Interactive Image Segmentation (AAAI 2024 · 25 citations) |
| Use two dense-fusion self-attention blocks instead of one | HQSeg-44K · 5-mIoU | 85.41 -> 85.71 | Helped | Rethinking Interactive Image Segmentation with Low Latency High Quality and Diverse Prompts (CVPR 2024 · 24 citations) |
| Add weak-supervision pretraining on Objects365 before VG fine-tuning | Visual Genome test split · CIDEr-D | 127.9 -> 134.5 | Helped | Segment and Caption Anything (CVPR 2024 · 19 citations) |
| add Pre-Fusion | RefCOCO · oIoU | 48.68 -> 65.38 | Helped | Universal Segmentation at Arbitrary Granularity with Language Instruction (CVPR 2024 · 19 citations) |
| Add Entity-Aware Fusion | ScanRefer validation · mIOU (%) | 24.66 -> 26.78 | Helped | X-RefSeg3D: Enhancing Referring 3D Instance Segmentation via Structured Cross-Modal Graph Neural Networks (AAAI 2024 · 18 citations) |
What reviewers will expect
Datasets and baselines named in the 28 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- Cityscapes · 4 of 28 papers
- ADE20K · 3 of 28 papers
- HQSeg-44K · 3 of 28 papers
- KiTS · 3 of 28 papers
- LiTS · 3 of 28 papers
- ACDC · 3 of 28 papers
- SBD · 2 of 28 papers
- COCO · 2 of 28 papers
Baselines
- SAM · 6 of 28 papers
- MedSAM · 3 of 28 papers
- SegNext · 3 of 28 papers
- SimpleClick · 3 of 28 papers
- UniverSeg · 3 of 28 papers
- OpenSeeD · 2 of 28 papers
- SAM 2 · 2 of 28 papers
- SAM-Med2D · 2 of 28 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 118 papers; from OpenAlex and the papers' affiliations).
- The Chinese University of Hong Kong · 7 papers
- Tsinghua University · 5 papers
- Xiamen University · 5 papers
- University of Oxford · 4 papers
- Johns Hopkins University · 4 papers
- Amazon · 4 papers
- The University of Hong Kong · 3 papers
- Nanyang Technological University · 3 papers
- Chinese Academy of Sciences · 3 papers
- Fudan University · 3 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Object-level representation learning (219 papers)
- Open-set and unknown object detection (160 papers)
- Amodal and open-world instance segmentation (137 papers)
- Industrial anomaly detection (117 papers)
- Remote sensing segmentation and change (84 papers)
Elsewhere in the topic
- Unsupervised object and part discovery (114 papers)
- Domain-adaptive object detection (103 papers)
- Open-vocabulary object detection (96 papers)
- Open-vocabulary semantic segmentation (93 papers)
- Object-centric learning theory and video (86 papers)
- Few-shot object detection (67 papers)
- Human-object interaction detection (66 papers)
- Few-shot segmentation (64 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).