Research map / Vision-language and multimodal / Human and embodied modeling
Language-grounded scene text and structure
Connects images with text through scene text recognition, language bias correction, and scene-language representations.
124 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Human and embodied modeling, keyword cluster “retrieval · scene · captioning”
State History Benchmarks Tried Baselines Checklist Preprints Who Export
Where it stands
Few recent reports None of the tracked benchmarks has a report from 2025–26.
27 papers in 2023–24 and 43 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 5.1% to 6.4%.
Open questions the data points to
- Papers report NegCLIP on SugarCrepe++ (accuracy) anywhere from 64.9 to 85.64: comparisons against it depend on whose number you take.
No longer reported by papers here: IC13 (accuracy, last in 2024); IIIT5K (accuracy, last in 2022); SVT (accuracy, last in 2022); SVTP (accuracy, last in 2022); IC15 (accuracy, last in 2022); CUTE (accuracy, last in 2022).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Jointly Modeling Embedding and Translation to Bridge Video and Language (CVPR 2016 · 598 citations)
- 2017 · Instance-Aware Image and Sentence Matching With Selective Multimodal LSTM (CVPR 2017 · 254 citations)
- 2018 · AON: Towards Arbitrarily-Oriented Text Recognition (CVPR 2018 · 329 citations)
- 2019 · Show, Attend and Read: A Simple and Strong Baseline for Irregular Text Recognition (AAAI 2019 · 450 citations)
- 2020 · X-Linear Attention Networks for Image Captioning (CVPR 2020 · 699 citations)
- 2021 · Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (CVPR 2021 · 615 citations)
- 2022 · Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality (CVPR 2022 · 214 citations)
- 2023 · Fine-Grained Image-Text Matching by Cross-Modal Hard Aligning Network (CVPR 2023 · 122 citations)
- 2024 · Context-Aware Integration of Language and Visual References for Natural Language Tracking (CVPR 2024 · 58 citations)
- 2025 · ResCLIP: Residual Attention for Training-free Dense Vision-language Inference (CVPR 2025 · 16 citations)
- 2026 · Beyond Cosine Similarity: Magnitude-Aware CLIP for No-Reference Image Quality Assessment (AAAI 2026 · 3 citations)
Earliest papers here
- Jointly Modeling Embedding and Translation to Bridge Video and Language (CVPR 2016 · 598 citations)
- Text Matching as Image Recognition (AAAI 2016 · 460 citations)
- Reading Scene Text in Deep Convolutional Sequences (AAAI 2016 · 320 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
IC13 · accuracy (higher is better)
7 papers, 2018–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2018 | 90.8 | Char-Net: A Character-Aware Neural Network for Distorted Scene Text Recognition (AAAI 2018 · 183 citations) |
| 2020 | 95.5 | Towards Accurate Scene Text Recognition With Semantic Reasoning Networks (CVPR 2020 · 382 citations) |
| 2021 | 97.4 | Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition (CVPR 2021 · 444 citations) |
| 2022 | 97.7 | Context-Based Contrastive Learning for Scene Text Recognition (AAAI 2022 · 59 citations) |
| 2024 | 98.3 | Image as a Language: Revisiting Scene Text Recognition via Balanced, Unified and Synchronized Vision-Language Reasoning Network (AAAI 2024 · 16 citations) |
IIIT5K · accuracy (higher is better)
6 papers, 2018–2022 · not reported since 2022
| Year | Best so far | Set by |
|---|---|---|
| 2018 | 83.6 | Char-Net: A Character-Aware Neural Network for Distorted Scene Text Recognition (AAAI 2018 · 183 citations) |
| 2020 | 94.8 | Towards Accurate Scene Text Recognition With Semantic Reasoning Networks (CVPR 2020 · 382 citations) |
| 2022 | 96.5 | Context-Based Contrastive Learning for Scene Text Recognition (AAAI 2022 · 59 citations) |
SVT · accuracy (higher is better)
6 papers, 2018–2022 · not reported since 2022
| Year | Best so far | Set by |
|---|---|---|
| 2018 | 84.4 | Char-Net: A Character-Aware Neural Network for Distorted Scene Text Recognition (AAAI 2018 · 183 citations) |
| 2020 | 91.5 | Towards Accurate Scene Text Recognition With Semantic Reasoning Networks (CVPR 2020 · 382 citations) |
| 2021 | 93.5 | Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition (CVPR 2021 · 444 citations) |
| 2022 | 94.3 | Context-Based Contrastive Learning for Scene Text Recognition (AAAI 2022 · 59 citations) |
SVTP · accuracy (higher is better)
6 papers, 2018–2022 · not reported since 2022
| Year | Best so far | Set by |
|---|---|---|
| 2018 | 73.5 | Char-Net: A Character-Aware Neural Network for Distorted Scene Text Recognition (AAAI 2018 · 183 citations) |
| 2020 | 85.1 | Towards Accurate Scene Text Recognition With Semantic Reasoning Networks (CVPR 2020 · 382 citations) |
| 2021 | 89.3 | Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition (CVPR 2021 · 444 citations) |
IC15 · accuracy (higher is better)
5 papers, 2020–2022 · not reported since 2022
| Year | Best so far | Set by |
|---|---|---|
| 2020 | 82.7 | Towards Accurate Scene Text Recognition With Semantic Reasoning Networks (CVPR 2020 · 382 citations) |
| 2021 | 86 | Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition (CVPR 2021 · 444 citations) |
CUTE · accuracy (higher is better)
4 papers, 2019–2022 · not reported since 2022
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 78.1 | Scene Text Recognition from Two-Dimensional Perspective (AAAI 2019 · 262 citations) |
| 2020 | 87.8 | Towards Accurate Scene Text Recognition With Semantic Reasoning Networks (CVPR 2020 · 382 citations) |
| 2022 | 91.3 | Context-Based Contrastive Learning for Scene Text Recognition (AAAI 2022 · 59 citations) |
What has been tried
102 papers here report 1,080 measured changes (a component added, removed or swapped, compared with a baseline): helped 72%, no clear effect 6%, hurt 18%, mixed 4%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| add language modulation | TNL2K, NL · AUC | 52.2 -> 53.3 | Helped | Context-Aware Integration of Language and Visual References for Natural Language Tracking (CVPR 2024 · 58 citations) |
| Use ViT-B instead of ViT-S as the backbone | IIIT, SVT, IC13, IC15, SVTP, CUTE (MJ+ST) · word accuracy | 92.8 -> 93.5 | Helped | OTE: Exploring Accurate Scene Text Recognition Using One Token (CVPR 2024 · 30 citations) |
| Remove negatives loss | ARO and VL-Checklist · accuracy | 76.23% -> 57.34%; 67.56% -> 61.98%; 88.58% -> 39.36%; 91.30% -> 44.62%; 80.71% -> 88.37%; 68.69% -> 70.42%; 70.12% -> 61 | Mixed | A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions (CVPR 2024 · 23 citations) |
| use context-rich training sequences with the Transform and Split Strategies | EIST · word accuracy | 26.82 -> 76.77 | Helped | Multi-modal In-Context Learning Makes an Ego-evolving Scene Text Recognizer (CVPR 2024 · 18 citations) |
| Add RCS to NACLIP | VOC20 · mIoU | 79.7 -> 85.5 | Helped | ResCLIP: Residual Attention for Training-free Dense Vision-language Inference (CVPR 2025 · 16 citations) |
| Add balanced language features during training | IC13 · accuracy | 96.7 -> 97.9 | Helped | Image as a Language: Revisiting Scene Text Recognition via Balanced, Unified and Synchronized Vision-Language Reasoning Network (AAAI 2024 · 16 citations) |
| Add probability-of-necessity weighting | Flickr30K · rSum | 504.1 -> 515.1 | Helped | Identification of Necessary Semantic Undertakers in the Causal View for Image-Text Matching (AAAI 2024 · 15 citations) |
| add time-evolving hybrid state space | TNL2K BBOX, NL, and NL&BBOX · AUC | 60.9 -> 62.1; 55.3 -> 56.8; 62.6 -> 64.5 | Helped | MambaVLT: Time-Evolving Multimodal State Space Model for Vision-Language Tracking (CVPR 2025 · 12 citations) |
| Add text-guided feature extraction and fusion | RoadScene test · Qabf | 0.48 -> 0.62 | Helped | Image Fusion via Vision-Language Model (ICML 2024 · 9 citations) |
| Add AEOM instead of cosine similarity with 2 image views | Flickr30K, text retrieval · R@1 | 74.0 -> 76.0 | Helped | Asymmetric Visual Semantic Embedding Framework for Efficient Vision-Language Alignment (AAAI 2025 · 8 citations) |
| Add the linguistic guidance branch and alignment loss to masked reconstruction | Union14M ablation · Avg accuracy | 77.7 -> 81.2 | Helped | Linguistics-aware Masked Image Modeling for Self-supervised Scene Text Recognition (CVPR 2025 · 6 citations) |
| Increase semantic-adjustment λ from 0 to at least 0.3 | Structure personalization · Generation quality | Helped | U-VAP: User-specified Visual Appearance Personalization via Decoupled Self Augmentation (CVPR 2024 · 6 citations) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| NegCLIP | SugarCrepe++ | accuracy | 64.9 – 85.64 | 2 papers, e.g. CVPR 2025 and NeurIPS 2024 |
What reviewers will expect
Datasets and baselines named in the 29 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- MS-COCO · 5 of 29 papers
- Flickr30K · 5 of 29 papers
- CIFAR-10 · 3 of 29 papers
- CIFAR-100 · 3 of 29 papers
- Total-Text · 3 of 29 papers
- CC3M · 2 of 29 papers
- CC12M · 2 of 29 papers
- COCO Captions · 2 of 29 papers
Baselines
- CLIP · 5 of 29 papers
- SCAN · 4 of 29 papers
- CoCa · 3 of 29 papers
- BLIP-2 · 3 of 29 papers
- VSE++ · 2 of 29 papers
- CHAN · 2 of 29 papers
- NAAF · 2 of 29 papers
- HREM · 2 of 29 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 70 papers; from OpenAlex and the papers' affiliations).
- Chinese Academy of Sciences · 4 papers
- Xi’an Jiaotong University · 4 papers
- Carnegie Mellon University · 4 papers
- Tsinghua University · 3 papers
- Stanford University · 3 papers
- City University of Hong Kong · 3 papers
- The Chinese University of Hong Kong · 2 papers
- University of Science and Technology of China · 2 papers
- East China Normal University · 2 papers
- Peking University · 2 papers
Use it
Download BibTeX of 40 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Scene graph generation (126 papers)
- Multimodal retrieval-augmented modeling (85 papers)
- Composed and text-based image retrieval (78 papers)
- Concept-guided image captioning (69 papers)
- Neural image captioning (72 papers)
Elsewhere in the topic
- Vision-language pretraining (182 papers)
- Medical vision-language models (130 papers)
- fMRI visual decoding (106 papers)
- Human-like visual cognition in VLMs (97 papers)
- Multimodal and affective interaction modeling (87 papers)
- Language-grounded visual concept learning (83 papers)
- Layout-aware document understanding (73 papers)
- Human perception modeling (71 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).