Research map / Vision-language and multimodal / Human and embodied modeling
Region-level vision-language alignment
Aligns regions and objects with language through visual experts and hierarchical grounded annotation.
32 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Human and embodied modeling, keyword cluster “grounding · referring · expression”
State History Tried Baselines Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
12 papers in 2023–24 and 11 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 2.3% to 1.6%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Visual Word2Vec (vis-w2v): Learning Visually Grounded Word Embeddings Using Abstract Scenes (CVPR 2016 · 86 citations)
- 2018 · Parallel Attention: A Unified Framework for Visual Object Discovery Through Dialogs and Queries (CVPR 2018 · 154 citations)
- 2019 · Connecting Language to Images: A Progressive Attention-Guided Network for Simultaneous Image Captioning and Language Grounding (AAAI 2019 · 16 citations)
- 2021 · Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separation (CVPR 2021 · 70 citations)
- 2022 · Shifting More Attention to Visual Backbone: Query-Modulated Refinement Networks for End-to-End Visual Grounding (CVPR 2022 · 103 citations)
- 2023 · MNER-QG: An End-to-End MRC Framework for Multimodal Named Entity Recognition with Query Grounding (AAAI 2023 · 60 citations)
- 2024 · GLaMM: Pixel Grounding Large Multimodal Model (CVPR 2024 · 175 citations)
- 2025 · Beyond Human Perception: Understanding Multi-Object World from Monocular View (CVPR 2025 · 6 citations)
- 2026 · Referring Multiple Regions with Large Multimodal Models via Contextual Latent Steering (ICML 2026 · 0 citations)
Earliest papers here
- Visual Word2Vec (vis-w2v): Learning Visually Grounded Word Embeddings Using Abstract Scenes (CVPR 2016 · 86 citations)
- Parallel Attention: A Unified Framework for Visual Object Discovery Through Dialogs and Queries (CVPR 2018 · 154 citations)
- Separating Self-Expression and Visual Content in Hashtag Supervision (CVPR 2018 · 29 citations)
What has been tried
27 papers here report 334 measured changes (a component added, removed or swapped, compared with a baseline): helped 78%, no clear effect 4%, hurt 16%, mixed 1%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Exclude the 1K human-annotated images from GranDf training | GCG test · METEOR | 15.8 -> 14.6 | Hurt | GLaMM: Pixel Grounding Large Multimodal Model (CVPR 2024 · 175 citations) |
| Use deep visual expert fusion instead of shallow adapter-only alignment | COCO Karpathy test · CIDEr | 131.2 -> 142.8 | Helped | CogVLM: Visual Expert for Pretrained Language Models (NeurIPS 2024 · 77 citations) |
| Add OID, OCC, and OSC | ScanRefer overall · Acc@0.25 | 50.59 -> 51.70 | Helped | Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding (AAAI 2024 · 28 citations) |
| add the HOI-Transfer Module and Pixel-Text Fusion Module | ADE20K-Unseen · KLD | 1.707 -> 1.335 | Helped | Weakly Supervised Multimodal Affordance Grounding for Egocentric Images (AAAI 2024 · 10 citations) |
| Use per-token regression instead of raw-string or extra-token localization | RefCOCO · P@0.5 | 84.4 -> 87.6; 86.1 -> 87.6 | Helped | Pixel-Aligned Language Model (CVPR 2024 · 8 citations) |
| Add State-Prompt | MonoMulti3D-ROPE test set (GT bboxes) · F1 | 51.20 -> 64.10 | Helped | Beyond Human Perception: Understanding Multi-Object World from Monocular View (CVPR 2025 · 6 citations) |
| use keyframe sampling rather than clip sampling for temporal downsampling | ScanNet · performance | Helped | Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding (NeurIPS 2024 · 6 citations) | |
| Add LMA to the HAMMER baseline | DGM4 · AUC | 93.16 -> 94.28 | Helped | ASAP: Advancing Semantic Alignment Promotes Multi-Modal Manipulation Detecting and Grounding (CVPR 2025 · 5 citations) |
| Add the LLM adaptation module (VA) | RefCOCO val · Prec@0.5 | 87.62 -> 88.38 | Helped | Task-aware Cross-modal Feature Refinement Transformer with Large Language Models for Visual Grounding (CVPR 2025 · 4 citations) |
| Add PSIL | DGM4 test · IoUmean | 74.56 -> 82.88 | Helped | IDseq: Decoupled and Sequentially Detecting and Grounding Multi-Modal Media Manipulation (AAAI 2025 · 3 citations) |
| add the depth estimation branch | Ego-IRGBench validation and test · cIoU | 31.72 -> 35.14 (validation); 31.90 -> 36.02 (test) | Helped | ANNEXE: Unified Analyzing, Answering, and Pixel Grounding for Egocentric Interaction (CVPR 2025 · 3 citations) |
| Use stochastic 3-view training with image-prior inference instead of cropped-image training | OVAD · mAP (%) | 23.0 -> 28.7 | Helped | DynRefer: Delving into Region-level Multimodal Tasks via Dynamic Resolution (CVPR 2025 · 1 citation) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| GRIT | refCOCOg | CIDEr | 71.6 – 75.8 | 2 papers, e.g. CVPR 2024 and CVPR 2024 |
What reviewers will expect
Datasets and baselines named in the 8 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- RefCOCOg · 3 of 8 papers
- DGM4 · 2 of 8 papers
- COCO · 2 of 8 papers
- RefCOCO · 2 of 8 papers
Baselines
- CLIP · 2 of 8 papers
- ViLT · 2 of 8 papers
- HAMMER · 2 of 8 papers
- UFAFormer · 2 of 8 papers
- X-Decoder · 2 of 8 papers
- BUTD-DETR · 2 of 8 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 23 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 3 papers
- Google · 3 papers
- Carnegie Mellon University · 3 papers
- Australian National University · 2 papers
- Hong Kong University of Science and Technology · 2 papers
Use it
Download BibTeX of 30 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Visual grounding transformers (63 papers)
- Referring expression comprehension (28 papers)
Elsewhere in the topic
- Vision-language pretraining (182 papers)
- Medical vision-language models (130 papers)
- Scene graph generation (126 papers)
- Language-grounded scene text and structure (124 papers)
- fMRI visual decoding (106 papers)
- Human-like visual cognition in VLMs (97 papers)
- Multimodal and affective interaction modeling (87 papers)
- Multimodal retrieval-augmented modeling (85 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).