Research map / Detection and segmentation / Open-set and few-shot detection
Open-vocabulary 3D and world perception
Extend open-vocabulary understanding to 3D instances, open-world detection, and language-guided segmentation.
41 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Open-set and few-shot detection, keyword cluster “vocabulary · language · text”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
14 papers in 2023–24 and 21 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 2.4% to 3.8%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Representing Verbs as Argument Concepts (AAAI 2016 · 7 citations)
- 2017 · Relationship Proposal Networks (CVPR 2017 · 131 citations)
- 2022 · DenseCLIP: Language-Guided Dense Prediction With Context-Aware Prompting (CVPR 2022 · 615 citations)
- 2023 · Objaverse: A Universe of Annotated 3D Objects (CVPR 2023 · 698 citations)
- 2024 · Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · 64 citations)
- 2025 · DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception (CVPR 2025 · 14 citations)
- 2026 · Person-Centric Annotations of LAION-400M: Auditing Bias and Its Transfer to Models (ICLR 2026 · 0 citations)
Earliest papers here
- Representing Verbs as Argument Concepts (AAAI 2016 · 7 citations)
- Relationship Proposal Networks (CVPR 2017 · 131 citations)
- Growing Interpretable Part Graphs on ConvNets via Multi-Shot Learning (AAAI 2017 · 62 citations)
What has been tried
30 papers here report 353 measured changes (a component added, removed or swapped, compared with a baseline): helped 76%, no clear effect 6%, hurt 17%, mixed 1%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Replace pointwise features with mask-wise features | ScanNet200 val · AP | 23.7 -> 22.2 | Hurt | Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · 64 citations) |
| Add spatial feature matching from both CLIP and DINOv2 | ADE20K · linear probe mIOU | 41.18 -> 48.13 | Helped | AM-RADIO: Agglomerative Vision Foundation Model Reduce All Domains Into One (CVPR 2024 · 46 citations) |
| add 3D superpoint primitives instead of point-based region growing | ScanNetV2 class-agnostic ablation · AP | 19.5 -> 24.2 | Helped | SAI3D: Segment Any Instance in 3D Scenes (CVPR 2024 · 45 citations) |
| equalize thing and stuff learning as instance-level objectives | COCO-Stuff · mIoU | 33.9 -> 37.6 | Helped | Aligning and Prompting Everything All at Once for Universal Visual Perception (CVPR 2024 · 30 citations) |
| Increase caption batch size from 4 to 20 without CLIM | OV-COCO · AP50 novel | 24.4 -> 27.4 | Helped | CLIM: Contrastive Language-Image Mosaic for Region Representation (AAAI 2024 · 19 citations) |
| Decouple self-distillation and VFM distillation features | COCO (Thing) · mAcc | 65.6 -> 75.0 | Helped | DeCLIP: Decoupled Learning for Open-Vocabulary Dense Perception (CVPR 2025 · 14 citations) |
| add guidance features to the fusion module | REAL275 · AR | 24.8 -> 32.2 | Helped | Open-Vocabulary Object 6D Pose Estimation (CVPR 2024 · 14 citations) |
| Increase false-premise query proportion from 0% to 75% at test time | FP-RefCOCOg · Segment (cIoU) | 66.02 -> 49.79 | Hurt | See Say and Segment: Teaching LMMs to Overcome False Premises (CVPR 2024 · 11 citations) |
| Add overlap ratio and third-view support to feature-only merging | ScanNet200 · AP | 9.7 -> 18.6 | Helped | OnlineAnySeg: Online Zero-Shot 3D Segmentation by Visual Foundation Model Guided 2D Mask Merging (CVPR 2025 · 7 citations) |
| Add attention generation to naive attention | CODA · mAR | +7.9 mAR | Helped | Training-Free Open-Ended Object Detection and Segmentation via Attention as Prompts (NeurIPS 2024 · 7 citations) |
| Scale training from ScanNet only to ScanNet, ARKitScenes, ScanNet++, Matterport3D, and Structured3D | ScanNet200 validation · f-mIoU | 13.0 -> 15.7 | Helped | Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation (CVPR 2025 · 5 citations) |
| Add MPFEM to the basic model | OVAS-25 · mIoU | 60.51 -> 67.41 | Helped | MaskPrompt: Open-Vocabulary Affordance Segmentation with Object Shape Mask Prompts (AAAI 2025 · 4 citations) |
What reviewers will expect
Datasets and baselines named in the 12 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- ScanNet++ · 3 of 12 papers
- LVIS · 2 of 12 papers
- Replica · 2 of 12 papers
Baselines
- CATSeg · 2 of 12 papers
- CLIP-DINOiser · 2 of 12 papers
- SCLIP · 2 of 12 papers
- MaskCLIP · 2 of 12 papers
- OpenScene · 2 of 12 papers
- OpenMask3D · 2 of 12 papers
- ConceptGraphs · 2 of 12 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 35 papers; from OpenAlex and the papers' affiliations).
- Tencent · 3 papers
- Peking University · 3 papers
- The Chinese University of Hong Kong · 2 papers
Use it
Download BibTeX of 29 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Open-vocabulary object detection (96 papers)
- Open-vocabulary semantic segmentation (93 papers)
Elsewhere in the topic
- Object-level representation learning (219 papers)
- Interactive and referring segmentation (161 papers)
- Open-set and unknown object detection (160 papers)
- Amodal and open-world instance segmentation (137 papers)
- Industrial anomaly detection (117 papers)
- Unsupervised object and part discovery (114 papers)
- Domain-adaptive object detection (103 papers)
- Object-centric learning theory and video (86 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).