Research map / Vision-language and multimodal / Human and embodied modeling
Scientific domain vision datasets
Builds large-scale datasets for ecological, geological, and agricultural imagery plus physical prediction.
47 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2016–2026) · an approach in Human and embodied modeling, keyword cluster “computer vision · humans · abstract”
State History Tried Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
20 papers in 2023–24 and 16 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 3.8% to 2.4%.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2016 · TGIF: A New Dataset and Benchmark on Animated GIF Description (CVPR 2016 · 210 citations)
- 2017 · Robust Joint and Individual Variance Explained (CVPR 2017 · 20 citations)
- 2021 · Nutrition5k: Towards Automatic Nutritional Understanding of Generic Food (CVPR 2021 · 137 citations)
- 2022 · Rethinking Visual Geo-Localization for Large-Scale Applications (CVPR 2022 · 263 citations)
- 2023 · TRAK: Attributing Model Behavior at Scale (ICML 2023 · 9 citations)
- 2024 · BioCLIP: A Vision Foundation Model for the Tree of Life (CVPR 2024 · 124 citations)
- 2025 · Fish-Vista: A Multi-Purpose Dataset for Understanding & Identification of Traits from Images (CVPR 2025 · 12 citations)
- 2026 · Mobility-Embedded POIs: Learning What A Place Is and How It Is Used from Human Movement (ICML 2026 · 0 citations)
Earliest papers here
- TGIF: A New Dataset and Benchmark on Animated GIF Description (CVPR 2016 · 210 citations)
- Robust Joint and Individual Variance Explained (CVPR 2017 · 20 citations)
- Nutrition5k: Towards Automatic Nutritional Understanding of Generic Food (CVPR 2021 · 137 citations)
What has been tried
36 papers here report 357 measured changes (a component added, removed or swapped, compared with a baseline): helped 70%, no clear effect 7%, hurt 21%, mixed 3%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Use taxonomic names rather than scientific names | Rare Species · Zero-shot accuracy | 22.3→26.6; 28.0→30.4 | Helped | BioCLIP: A Vision Foundation Model for the Tree of Life (CVPR 2024 · 124 citations) |
| Add attention pooling | IP102 Classification · Acc@1 | 68.9% to 69.5% | Helped | Insect-Foundation: A Foundation Model and Large-scale 1M Dataset for Visual Insect Understanding (CVPR 2024 · 18 citations) |
| Replace direct regression with a QuadTree classifier | OSV-5M test · Geoscore | + 720 | Helped | OpenStreetView-5M: The Many Roads to Global Visual Geolocation (CVPR 2024 · 15 citations) |
| Add class-balanced re-weighting and focal loss to MaxViT-T | FV-Classification · classification performance | Helped | Fish-Vista: A Multi-Purpose Dataset for Understanding & Identification of Traits from Images (CVPR 2025 · 12 citations) | |
| Add head direction to UWB | Behavior classification OS · average F1 | .707±.051 -> .707±.064 | No clear effect | MmCows: A Multimodal Dataset for Dairy Cattle Monitoring (NeurIPS 2024 · 12 citations) |
| add Camera Parameters Estimator (CPE) | Vi-Fi · SUM | 32.12 -> 27.24 | Helped | OOSTraj: Out-of-Sight Trajectory Prediction With Vision-Positioning Denoising (CVPR 2024 · 10 citations) |
| add 4D Radar to LiDAR | Vehicle Easy, IoU=0.5 · AP | 47.23 -> 53.61 | Helped | V2X-Radar: A Multi-modal Dataset with 4D Radar for Cooperative Perception (NeurIPS 2025 · 7 citations) |
| add environment-guided area representation learning | Forecast, real-world data · AUPRC | 60.2 -> 70.1 | Helped | Sim2Real-Fire: A Multi-modal Simulation Dataset for Forecast and Backtracking of Real-world Forest Fire (NeurIPS 2024 · 4 citations) |
| Add Attention mechanism only | Evaluation Area · accuracy | 92.97 -> 95.41 | Helped | DCV2I: A Practical Approach for Supporting Geographers’ Visual Interpretation in Dune Segmentation with Deep Vision Models (AAAI 2024 · 3 citations) |
| Include more examples from the same target class in the prompt (target-class-only rather than at-least-one-target sampling) | Nanoparticle shapes (test set; N=4) · average accuracy | 0.59±0.12 -> 0.66±0.09 (textual); 0.55±0.11 -> 0.62±0.09 (tabular) | Helped | Unveiling the Potential of AI for Nanomaterial Morphology Prediction (ICML 2024 · 3 citations) |
| replace random deduplication with quality-guided deduplication | DataComp (38 downtasks) · Avg. Perf. | 0.145 -> 0.147 | Helped | Quality over Quantity: Boosting Data Efficiency Through Ensembled Multimodal Data Curation (AAAI 2025 · 2 citations) |
| Remove caption augmentation | Human-created test set · CLIPScore | 26.776 -> 25.703 | Hurt | AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZ (ICLR 2024 · 2 citations) |
What reviewers will expect
Datasets and baselines named in the 13 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Baselines
- MLP · 2 of 13 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 36 papers; from OpenAlex and the papers' affiliations).
- Shanghai Jiao Tong University · 2 papers
- Microsoft · 2 papers
- Tsinghua University · 2 papers
- University of California, Los Angeles · 2 papers
- Nanyang Technological University · 2 papers
- Harvard University · 2 papers
Use it
Download BibTeX of 32 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Multimodal and affective interaction modeling (87 papers)
- Human-like visual cognition in VLMs (97 papers)
- Human perception modeling (71 papers)
- Abstract visual reasoning benchmarks (63 papers)
Elsewhere in the topic
- Vision-language pretraining (182 papers)
- Medical vision-language models (130 papers)
- Scene graph generation (126 papers)
- Language-grounded scene text and structure (124 papers)
- fMRI visual decoding (106 papers)
- Multimodal retrieval-augmented modeling (85 papers)
- Language-grounded visual concept learning (83 papers)
- Composed and text-based image retrieval (78 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).