Research map / Vision-language and multimodal / Human and embodied modeling
Human-like visual cognition in VLMs
Tests and improves vision-language models on human visual cognition, compositionality, and structure extraction.
97 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Human and embodied modeling, keyword cluster “computer vision · humans · abstract”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
44 papers in 2023–24 and 30 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 8.3% to 4.5%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · Toward a Taxonomy and Computational Models of Abnormalities in Images (AAAI 2016 · 10 citations)
- 2017 · Self-Supervised Learning of Visual Features Through Embedding Images Into Text Topic Spaces (CVPR 2017 · 102 citations)
- 2019 · Grounding Human-To-Vehicle Advice for Self-Driving Vehicles (CVPR 2019 · 121 citations)
- 2020 · Texture Interpolation for Probing Visual Perception (NeurIPS 2020 · 13 citations)
- 2021 · Passive attention in artificial neural networks predicts human visual selectivity (NeurIPS 2021 · 4 citations)
- 2022 · Perceiver IO: A General Architecture for Structured Inputs & Outputs (ICLR 2022 · 205 citations)
- 2023 · Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective (CVPR 2023 · 315 citations)
- 2024 · VMamba: Visual State Space Model (NeurIPS 2024 · 830 citations)
- 2025 · Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detector (CVPR 2025 · 29 citations)
- 2026 · Rethinking Saliency Maps: A Cognitive Human Aligned Taxonomy and Evaluation Framework for Explanations (AAAI 2026 · 3 citations)
Earliest papers here
- Toward a Taxonomy and Computational Models of Abnormalities in Images (AAAI 2016 · 10 citations)
- A Paradigm for Building Generalized Models of Human Image Perception Through Data Fusion (CVPR 2016 · 6 citations)
- Learned Region Sparsity and Diversity Also Predicts Visual Attention (NeurIPS 2016 · 4 citations)
What has been tried
79 papers here report 835 measured changes (a component added, removed or swapped, compared with a baseline): helped 66%, no clear effect 7%, hurt 21%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use Triton for Cross-Scan and Cross-Merge | ImageNet-1K · Throughput | +41 img/s | Helped | VMamba: Visual State Space Model (NeurIPS 2024 · 830 citations) |
| train with discrete text-defined levels instead of scores | KonIQ, SPAQ, LIVE Challenge, AGIQA, and KADID · average SRCC/PLCC | +2.2%; +3.4%; +8.1%; +3.1%; +29.6% | Helped | Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels (ICML 2024 · 33 citations) |
| add the 1st-level feature to the 4th-level feature | IVGaze · angular error | 7.35° -> 7.04° | Helped | What Do You See in Vehicle? Comprehensive Vision Solution for In-Vehicle Gaze Estimation (CVPR 2024 · 31 citations) |
| add UF-prompts | FF++ (c40) · AUC | 91.03 -> 92.57 | Helped | Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detector (CVPR 2025 · 29 citations) |
| add synthetic mesh images during training | Celeb-reID · mAP | 17.9 -> 19.2 | Helped | Distilling CLIP with Dual Guidance for Learning Discriminative Human Body Shape Representation (CVPR 2024 · 25 citations) |
| Add B2T keywords with negative CLIP scores to prompts | Waterbirds · worst-group accuracy | 50.3 -> 45.4 | Hurt | Discovering and Mitigating Visual Biases through Keyword Explanation (CVPR 2024 · 17 citations) |
| Add label-preserving and label-breaking generated images | Circuit Diagram Classification · Top-1 accuracy (%) | 19.3 -> 21.3 | Helped | Low-Resource Vision Challenges for Foundation Models (CVPR 2024 · 11 citations) |
| Use training dynamics as supervision for PLS decomposition rather than unsupervised PCA | CIFAR-100 · cosine similarity | 10.76 -> 22.97 | Helped | Discover and Mitigate Multiple Biased Subgroups in Image Classifiers (CVPR 2024 · 8 citations) |
| add low-level spatial sentences to scene-level sentences during training | NYUv2 supervised test · RMSE | 0.382 -> 0.440 | Hurt | On the Robustness of Language Guidance for Low-Level Vision Tasks: Findings from Depth Estimation (CVPR 2024 · 7 citations) |
| Use only the vision-transformer loss ℓ_vit instead of the combined loss | CUB-200-2011 · Top-1 accuracy | 92.03 -> 91.66 | Hurt | ViTree: Single-Path Neural Tree for Step-Wise Interpretable Fine-Grained Visual Categorization (AAAI 2024 · 4 citations) |
| use 3k rather than 6k steps per generation | Compositional benchmarks · COMP | 34.4 -> 32.1 | Hurt | Iterated Learning Improves Compositionality in Large Vision-Language Models (CVPR 2024 · 4 citations) |
| Add task-conditioned middle and top visual adapters | CortexBench known tasks, 12-task average (test) · performance | 42.5 ± 1.2 -> 54.8 ± 1.2 | Helped | Task-Conditioned Adaptation of Visual Features in Multi-Task Policy Learning (CVPR 2024 · 4 citations) |
What reviewers will expect
Datasets and baselines named in the 20 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- ImageNet-1K · 3 of 20 papers
- COCO · 3 of 20 papers
- ImageNet · 3 of 20 papers
- CelebA · 2 of 20 papers
Baselines
- SAM · 2 of 20 papers
- CLIP · 2 of 20 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 74 papers; from OpenAlex and the papers' affiliations).
- Shanghai Jiao Tong University · 6 papers
- Massachusetts Institute of Technology · 4 papers
- Google · 4 papers
- Korea Advanced Institute of Science and Technology · 3 papers
- Stanford University · 3 papers
- Tsinghua University · 3 papers
- Meta · 2 papers
- University of Rochester · 2 papers
- SenseTime · 2 papers
- Zhejiang University · 2 papers
Use it
Download BibTeX of 35 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Multimodal and affective interaction modeling (87 papers)
- Human perception modeling (71 papers)
- Abstract visual reasoning benchmarks (63 papers)
- Scientific domain vision datasets (47 papers)
Elsewhere in the topic
- Vision-language pretraining (182 papers)
- Medical vision-language models (130 papers)
- Scene graph generation (126 papers)
- Language-grounded scene text and structure (124 papers)
- fMRI visual decoding (106 papers)
- Multimodal retrieval-augmented modeling (85 papers)
- Language-grounded visual concept learning (83 papers)
- Composed and text-based image retrieval (78 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).