Research map / Vision-language and multimodal / Human and embodied modeling
Scene graph generation
Generates open-vocabulary and training-free scene graphs using vision-language models and object-centric binding.
126 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Human and embodied modeling, keyword cluster “retrieval · scene · captioning”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
38 papers in 2023–24 and 54 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 7.1% to 8.1%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Large-Scale Location Recognition and the Geometric Burstiness Problem (CVPR 2016 · 156 citations)
- 2017 · Scene Physics Acquisition via Visual De-animation (NeurIPS 2017)
- 2018 · Diverse Beam Search for Improved Description of Complex Scenes (AAAI 2018 · 229 citations)
- 2019 · Attention Branch Network: Learning of Attention Mechanism for Visual Explanation (CVPR 2019 · 521 citations)
- 2020 · Learning Visual Emotion Representations From Web Data (CVPR 2020 · 42 citations)
- 2021 · Causal Attention for Vision-Language Tasks (CVPR 2021 · 158 citations)
- 2022 · Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation (CVPR 2022 · 134 citations)
- 2023 · AutoAD: Movie Description in Context (CVPR 2023 · 41 citations)
- 2024 · HIPTrack: Visual Tracking with Historical Prompts (CVPR 2024 · 105 citations)
- 2025 · Game4Loc: A UAV Geo-Localization Benchmark from Game Data (AAAI 2025 · 28 citations)
- 2026 · Endowing Vision-Language Models with System 2 Thinking for Fine-grained Visual Recognition (AAAI 2026 · 2 citations)
Earliest papers here
- Large-Scale Location Recognition and the Geometric Burstiness Problem (CVPR 2016 · 156 citations)
- Business-Aware Visual Concept Discovery from Social Media for Multimodal Business Venue Recognition (AAAI 2016 · 12 citations)
- A Task-Oriented Approach for Cost-Sensitive Recognition (CVPR 2016 · 2 citations)
What has been tried
104 papers here report 1,184 measured changes (a component added, removed or swapped, compared with a baseline): helped 76%, no clear effect 3%, hurt 18%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add historical refined foreground masks | LaSOT test · AUC | 71.5 -> 72.7 | Helped | HIPTrack: Visual Tracking with Historical Prompts (CVPR 2024 · 105 citations) |
| use 6 transformer layers rather than 0 in entity grounding | PSG SGDet · mR@100 | 10.3 -> 17.0 | Helped | From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models (CVPR 2024 · 49 citations) |
| replace random word swaps with scene-graph-guided semantic negatives | VG-Attribution · accuracy | 73.9 -> 77.8 | Helped | Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-Modal Structured Representations (AAAI 2024 · 40 citations) |
| Add paraphrased captions to the original captions | VG, COCO Caption · R@50 | 6.60 -> 9.46 | Helped | LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generation (CVPR 2024 · 37 citations) |
| add high-level prompts to global-level prompts | Base-to-new average · harmonic mean | 78.99 -> 79.33 | Helped | Learning Hierarchical Prompt with Structured Linguistic Knowledge for Vision-Language Models (AAAI 2024 · 32 citations) |
| Use oriented instead of horizontal bounding boxes | VRSBench-Ref test, all objects · Acc@0.5 | 49.8% -> 24.3% | Hurt | VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding (NeurIPS 2024 · 29 citations) |
| Use ViT-Base/16 rather than ConvNeXt-Base | GTA-UAV cross-area · R@1 | 55.36% -> 55.91% | Helped | Game4Loc: A UAV Geo-Localization Benchmark from Game Data (AAAI 2025 · 28 citations) |
| add concept-level constraint to proxy learning | Fruit color · NMI | 0.7642 -> 0.8215 | Helped | Multi-Modal Proxy Learning Towards Personalized Visual Multiple Clustering (CVPR 2024 · 24 citations) |
| add backbone optimization | ITSDT-15K · mAP50 | 71.95 -> 76.43 | Helped | Motion Prior Knowledge Learning with Homogeneous Language Descriptions for Moving Infrared Small Target Detection (AAAI 2025 · 21 citations) |
| Add the scene magnifier module | XR-QA · CIDEr | 110.33 -> 113.46(+3.13) | Helped | LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences (CVPR 2025 · 20 citations) |
| Add predicate and triplet cue modeling | VG150 PredCls · R@50 | 56.5/60.4 -> 69.3/71.3 | Helped | Leveraging Predicate and Triplet Learning for Scene Graph Generation (CVPR 2024 · 20 citations) |
| Replace question conditioning with a fixed textual prompt | VQAv2 and VQAT · accuracy | +0.1%; +1.1% | Mixed | Question Aware Vision Transformer for Multimodal Reasoning (CVPR 2024 · 18 citations) |
What reviewers will expect
Datasets and baselines named in the 37 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- Visual Genome · 5 of 37 papers
- COCO · 5 of 37 papers
- ImageNet-1K · 3 of 37 papers
- MSCOCO · 2 of 37 papers
- Wikidata · 2 of 37 papers
- Pascal-5i · 2 of 37 papers
- Pascal VOC 2012 · 2 of 37 papers
- GQA · 2 of 37 papers
Baselines
- BLIP2 · 4 of 37 papers
- CLIP · 3 of 37 papers
- GeoCLIP · 3 of 37 papers
- NegCLIP · 2 of 37 papers
- XVLM · 2 of 37 papers
- Sample4Geo · 2 of 37 papers
- PE-NET · 2 of 37 papers
- SGTR · 2 of 37 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 92 papers; from OpenAlex and the papers' affiliations).
- Tencent · 5 papers
- Zhejiang University · 4 papers
- City University of Hong Kong · 4 papers
- Tsinghua University · 3 papers
- Alibaba Group · 3 papers
- Xiamen University · 3 papers
- Beihang University · 3 papers
- Huazhong University of Science and Technology · 2 papers
- Peng Cheng Laboratory · 2 papers
- Xidian University · 2 papers
Use it
Download BibTeX of 37 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Language-grounded scene text and structure (124 papers)
- Multimodal retrieval-augmented modeling (85 papers)
- Composed and text-based image retrieval (78 papers)
- Concept-guided image captioning (69 papers)
- Neural image captioning (72 papers)
Elsewhere in the topic
- Vision-language pretraining (182 papers)
- Medical vision-language models (130 papers)
- fMRI visual decoding (106 papers)
- Human-like visual cognition in VLMs (97 papers)
- Multimodal and affective interaction modeling (87 papers)
- Language-grounded visual concept learning (83 papers)
- Layout-aware document understanding (73 papers)
- Human perception modeling (71 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).