Research map / 3D vision / 3D generation
Open-vocabulary 3D scene understanding
Aligns 3D scenes with language for open-world, part-level and affordance understanding.
95 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in 3D generation, keyword cluster “layout · indoor · 3d scene”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
21 papers in 2023–24 and 49 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 4.6% to 5.6%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Understanding Real World Indoor Scenes With Synthetic Data (CVPR 2016 · 243 citations)
- 2017 · Shading Annotations in the Wild (CVPR 2017 · 76 citations)
- 2018 · Learning Interpretable Spatial Operations in a Rich 3D Blocks World (AAAI 2018 · 56 citations)
- 2019 · Putting Humans in a Scene: Learning Affordance in 3D Indoor Environments (CVPR 2019 · 107 citations)
- 2020 · Learning 3D Semantic Scene Graphs From 3D Indoor Reconstructions (CVPR 2020 · 245 citations)
- 2021 · SceneGraphFusion: Incremental 3D Scene Graph Prediction From RGB-D Sequences (CVPR 2021 · 155 citations)
- 2022 · ScanQA: 3D Question Answering for Spatial Scene Understanding (CVPR 2022 · 186 citations)
- 2023 · OpenScene: 3D Scene Understanding With Open Vocabularies (CVPR 2023 · 365 citations)
- 2024 · RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding (CVPR 2024 · 59 citations)
- 2025 · Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding (CVPR 2025 · 34 citations)
- 2026 · Open-World 3D Scene Graph Generation for Retrieval-Augmented Reasoning (AAAI 2026 · 2 citations)
Earliest papers here
- Understanding Real World Indoor Scenes With Synthetic Data (CVPR 2016 · 243 citations)
- Shading Annotations in the Wild (CVPR 2017 · 76 citations)
- Generating Holistic 3D Scene Abstractions for Text-Based Image Retrieval (CVPR 2017 · 28 citations)
What has been tried
65 papers here report 653 measured changes (a component added, removed or swapped, compared with a baseline): helped 78%, no clear effect 3%, hurt 17%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add regional captions instead of using view- and entity-level language supervision | ScanNet B0/N17 · mIoU† | 21.7 -> 50.6 | Helped | RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding (CVPR 2024 · 59 citations) |
| add scene-graph guidance with node-wise and edge-wise prompts | 30 multi-object scenes (number of objects ≥2) · CLIP score (mean) | 0.3019 -> 0.3326 | Helped | GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs (CVPR 2024 · 43 citations) |
| Add coarse-to-fine mask expansion with r0=0.1 | Functionality segmentation test · AP25 | 0.0 -> 26.6 | Helped | SceneFun3D: Fine-Grained Functionality and Affordance Understanding in 3D Scenes (CVPR 2024 · 36 citations) |
| Use maximum coverage rather than uniform sampling with 8 frames | ScanRefer · Acc@0.25 | 48.93 -> 53.47 | Helped | Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding (CVPR 2025 · 34 citations) |
| Use learnable object identifier tokens instead of plain-text identifiers | ScanRefer validation · Acc@0.5 | 47.2 -> 50.2 | Helped | Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers (NeurIPS 2024 · 31 citations) |
| Use ground-truth rather than estimated depth | iGibson-Synthetic · mAP | 69.09 -> 79.46 | Helped | PanoContext-Former: Panoramic Total Scene Understanding with a Transformer (CVPR 2024 · 18 citations) |
| retain scene memory across navigation subtasks instead of clearing it after each subtask | GOAT-Bench Val Unseen subset · Success Rate | 58.6 -> 69.1 | Helped | 3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning (CVPR 2025 · 16 citations) |
| add viewpoint prediction and viewpoint data augmentation | ScanRefer · Acc@0.25 | 51.7 -> 55.65 | Helped | Viewpoint-Aware Visual Grounding in 3D Scenes (CVPR 2024 · 16 citations) |
| Add assistive object tags to interactive-element detection prompts | SceneFun3D · overall node recall R@3 | 59.3 -> 73.0 | Helped | Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces (CVPR 2025 · 15 citations) |
| Reduce SPAR-7M proportion from 60% to 40% | CV-Bench 2D · score | 73.00 → 74.13 | Helped | From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D (NeurIPS 2025 · 12 citations) |
| replace the spatial encoder with an MLP for direct object location encoding | Nr3D Overall · accuracy | 64.4% -> 45.0% ± 0.4% | Hurt | MiKASA: Multi-Key-Anchor & Scene-Aware Transformer for 3D Visual Grounding (CVPR 2024 · 11 citations) |
| Add depth input | INS testing · PSNR | 29.95 -> 30.38 | Helped | OmniSR: Shadow Removal Under Direct and Indirect Lighting (AAAI 2025 · 9 citations) |
What reviewers will expect
Datasets and baselines named in the 25 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- ScanRefer · 8 of 25 papers
- SQA3D · 7 of 25 papers
- ScanQA · 7 of 25 papers
- ScanNet · 6 of 25 papers
- Scan2Cap · 6 of 25 papers
- Matterport3D · 5 of 25 papers
- Multi3DRefer · 4 of 25 papers
- Structured3D · 3 of 25 papers
Baselines
- LEO · 7 of 25 papers
- ChatScene · 5 of 25 papers
- Scene-LLM · 5 of 25 papers
- 3D-VisTA · 4 of 25 papers
- 3D-LLM · 4 of 25 papers
- LL3DA · 4 of 25 papers
- GPT-4o · 3 of 25 papers
- PQ3D · 3 of 25 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 70 papers; from OpenAlex and the papers' affiliations).
- Google · 5 papers
- ETH Zurich · 4 papers
- The Chinese University of Hong Kong · 4 papers
- Zhejiang University · 3 papers
- Massachusetts Institute of Technology · 3 papers
- Stanford University · 3 papers
- Google DeepMind · 3 papers
- University of Illinois, Urbana Champaign · 3 papers
- Meta · 3 papers
- The University of Sydney · 2 papers
Use it
Download BibTeX of 35 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Generative indoor scene synthesis (131 papers)
- Visual-acoustic scene modeling (12 papers)
Elsewhere in the topic
- Learning 3D reconstruction from images (196 papers)
- Diffusion-based novel view synthesis (156 papers)
- Semantic scene completion (153 papers)
- Hand-object interaction generation (116 papers)
- Feed-forward 3D asset generation and editing (116 papers)
- Score distillation for text-to-3D (106 papers)
- 2D-to-3D pretrained representations (94 papers)
- Monocular 3D object reconstruction (94 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).