Research map / Vision-language and multimodal / Human and embodied modeling
Visual grounding transformers
Localizes language queries in images and 3D scenes with decoupled fusion and verification.
63 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Human and embodied modeling, keyword cluster “grounding · referring · expression”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 3 of 3 tracked benchmarks.
26 papers in 2023–24 and 20 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 4.9% to 3.0%.
Open questions the data points to
- RefCOCO val (accuracy): best went from 87.4 (up to 2024) to 92.73 in 2025 — Look Around Before Locating: Considering Content and Structure Information for Visual Grounding (AAAI 2025 · 3 citations)
- RefCOCO testA (accuracy): best went from 88.02 (up to 2024) to 94.04 in 2025 — Look Around Before Locating: Considering Content and Structure Information for Visual Grounding (AAAI 2025 · 3 citations)
- RefCOCO testB (accuracy): best went from 87.11 (up to 2024) to 90.75 in 2025 — Look Around Before Locating: Considering Content and Structure Information for Visual Grounding (AAAI 2025 · 3 citations)
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Generating Descriptions With Grounded and Co-Referenced People (CVPR 2017 · 52 citations)
- 2018 · Grounding Referring Expressions in Images by Variational Context (CVPR 2018 · 273 citations)
- 2019 · Multi-Level Multimodal Common Semantic Space for Image-Phrase Grounding (CVPR 2019 · 72 citations)
- 2020 · Counterfactual Contrastive Learning for Weakly-Supervised Vision-Language Grounding (NeurIPS 2020 · 99 citations)
- 2022 · Improving Visual Grounding With Visual-Linguistic Verification and Iterative Reasoning (CVPR 2022 · 167 citations)
- 2023 · Joint Visual Grounding and Tracking With Natural Language Specification (CVPR 2023 · 139 citations)
- 2024 · Parallel Vertex Diffusion for Unified Visual Grounding (AAAI 2024 · 28 citations)
- 2025 · Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding (CVPR 2025 · 12 citations)
- 2026 · Mono3DVG-EnSD: Enhanced Spatial-aware and Dimension-decoupled Text Encoding for Monocular 3D Visual Grounding (AAAI 2026 · 1 citation)
Earliest papers here
- Generating Descriptions With Grounded and Co-Referenced People (CVPR 2017 · 52 citations)
- Interpretable and Globally Optimal Prediction for Textual Grounding using Image Concepts (NeurIPS 2017 · 12 citations)
- Grounding Referring Expressions in Images by Variational Context (CVPR 2018 · 273 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
RefCOCO val · accuracy (higher is better)
5 papers, 2022–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 85.37 | Multi-Modal Dynamic Graph Transformer for Visual Grounding (CVPR 2022 · 27 citations) |
| 2023 | 87.4 | Dynamic Inference With Grounding Based Vision and Language Models (CVPR 2023 · 6 citations) |
| 2025 | 92.73 | Look Around Before Locating: Considering Content and Structure Information for Visual Grounding (AAAI 2025 · 3 citations) |
RefCOCO testA · accuracy (higher is better)
4 papers, 2022–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 87.69 | Improving Visual Grounding With Visual-Linguistic Verification and Iterative Reasoning (CVPR 2022 · 167 citations) |
| 2024 | 88.02 | Parallel Vertex Diffusion for Unified Visual Grounding (AAAI 2024 · 28 citations) |
| 2025 | 94.04 | Look Around Before Locating: Considering Content and Structure Information for Visual Grounding (AAAI 2025 · 3 citations) |
RefCOCO testB · accuracy (higher is better)
4 papers, 2022–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 87.11 | Multi-Modal Dynamic Graph Transformer for Visual Grounding (CVPR 2022 · 27 citations) |
| 2025 | 90.75 | Look Around Before Locating: Considering Content and Structure Information for Visual Grounding (AAAI 2025 · 3 citations) |
What has been tried
51 papers here report 539 measured changes (a component added, removed or swapped, compared with a baseline): helped 79%, no clear effect 4%, hurt 16%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add center anchor mechanism | RefCOCO val · accuracy (REC) | 79.38 -> 81.24 | Helped | Parallel Vertex Diffusion for Unified Visual Grounding (AAAI 2024 · 28 citations) |
| Use decoupled multimodal fusion with ViLT or BEiT-3 rather than independent dual-stream CLIP encoding | Visual grounding · performance and convergence speed | Helped | SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion (NeurIPS 2024 · 28 citations) | |
| add PAGE geometric feature modeling | ScanRefer Unique, IoU 0.25 · accuracy | 85.67 -> 86.89 | Helped | G^3-LQ: Marrying Hyperbolic Alignment with Explicit Semantic-Geometric Modeling for 3D Visual Grounding (CVPR 2024 · 15 citations) |
| add paraphrases as regular image-text pairs without explicit SelfEQ linkage | Flickr30k · pointing game accuracy | 79.38 -> 78.05 | Hurt | Improved Visual Grounding through Self-Consistent Explanations (CVPR 2024 · 14 citations) |
| Increase localization heads from 1 to 3 | RefCOCO val (RES) · cIoU | 71.7 -> 76.1 | Helped | Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding (CVPR 2025 · 12 citations) |
| fine-tune using text pairs that describe the same object | ARPGrounding Priority · accuracy | 11.24 to 5.63 | Hurt | Investigating Compositional Challenges in Vision-Language Models for Visual Grounding (CVPR 2024 · 10 citations) |
| add random noise to pseudo-language features | ScanRefer val, Overall · Acc@0.25 | 26.88 -> 28.27 | Helped | Towards CLIP-driven Language-free 3D Visual Grounding via 2D-3D Relational Enhancement and Consistency (CVPR 2024 · 10 citations) |
| Add cycle-consistency losses to supervised grounding training | RefCOCO testA · top-1 accuracy | 91.07 -> 91.87 | Helped | Cycle-Consistency Learning for Captioning and Grounding (AAAI 2024 · 9 citations) |
| add Attribute-Text attention | Nr3D · accuracy | 55.2 | Helped | Multi-Attribute Interactions Matter for 3D Visual Grounding (CVPR 2024 · 8 citations) |
| Add gesture input to the language-only model | ScanERU validation set, multiple objects · Acc@0.25IoU | 41.42 -> 66.27 | Helped | ScanERU: Interactive 3D Visual Grounding Based on Embodied Reference Understanding (AAAI 2024 · 6 citations) |
| Use 8-times downsampled images in ACM and ZGM | GigaGrounding test · Precision@0.5 | 66.5% | Helped | When Visual Grounding Meets Gigapixel-level Large-scale Scenes: Benchmark and Approach (CVPR 2024 · 6 citations) |
| Add cross-modal augmentation without LSAD | ScanRefer overall · Acc@0.25 | 50.86 -> 52.79 | Helped | AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referring (AAAI 2025 · 5 citations) |
What reviewers will expect
Datasets and baselines named in the 11 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- RefCOCO · 2 of 11 papers
- RefCOCOg · 2 of 11 papers
Baselines
- BUTD-DETR · 3 of 11 papers
- EDA · 2 of 11 papers
- ViewRefer · 2 of 11 papers
- MDETR · 2 of 11 papers
- Shikra · 2 of 11 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 46 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 4 papers
- Peng Cheng Laboratory · 3 papers
- Sichuan University · 3 papers
- Fudan University · 3 papers
- Guangxi University · 2 papers
- Guangxi Key Laboratory of Multimedia Communications and Network Technology · 2 papers
- Xiamen University · 2 papers
- Tencent · 2 papers
- Hong Kong University of Science and Technology · 2 papers
- Peking University · 2 papers
Use it
Download BibTeX of 38 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Region-level vision-language alignment (32 papers)
- Referring expression comprehension (28 papers)
Elsewhere in the topic
- Vision-language pretraining (182 papers)
- Medical vision-language models (130 papers)
- Scene graph generation (126 papers)
- Language-grounded scene text and structure (124 papers)
- fMRI visual decoding (106 papers)
- Human-like visual cognition in VLMs (97 papers)
- Multimodal and affective interaction modeling (87 papers)
- Multimodal retrieval-augmented modeling (85 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).