Research map / Vision-language and multimodal / Human and embodied modeling
Vision-language pretraining
Pretrains unified vision-language models using image-as-language tokens and decoupled visual encoding.
182 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Human and embodied modeling, keyword cluster “pre · generation · unified”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 1 of 6 tracked benchmarks.
67 papers in 2023–24 and 80 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic held steady, from 12.6% to 12.0%.
Open questions the data points to
- GQA (accuracy): best went from 61.8 (up to 2024) to 77 in 2025 — R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts (ICML 2025 · 0 citations)
What looks settled
- GQA (score): best up to 2024 63.9, best in 2025–26 63.1 from 4 papers, at most 0.2% better, if at all.
No longer reported by papers here: ADE20K (mIoU, last in 2024); ImageNet-1K (Top-1 accuracy, last in 2024); COCO captioning (CIDEr, last in 2024); ImageNet-1k (accuracy, last in 2024).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · Learning multiple visual domains with residual adapters (NeurIPS 2017 · 562 citations)
- 2018 · Listening to the World Improves Speech Command Recognition (AAAI 2018 · 37 citations)
- 2019 · Visual Sequence Learning in Hierarchical Prediction Networks and Primate Visual Cortex (NeurIPS 2019 · 3 citations)
- 2020 · Unified Vision-Language Pre-Training for Image Captioning and VQA (AAAI 2020 · 852 citations)
- 2021 · Align before Fuse: Vision and Language Representation Learning with Momentum Distillation (NeurIPS 2021 · 812 citations)
- 2022 · BEiT: BERT Pre-Training of Image Transformers (ICLR 2022 · 915 citations)
- 2023 · Image as a Foreign Language: BEiT Pretraining for Vision and Vision-Language Tasks (CVPR 2023 · 531 citations)
- 2024 · Sequential Modeling Enables Scalable Learning for Large Vision Models (CVPR 2024 · 102 citations)
- 2025 · Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (CVPR 2025 · 44 citations)
- 2026 · Aware Distillation for Robust Vision-Language Tracking Under Linguistic Sparsity (AAAI 2026 · 1 citation)
Earliest papers here
- Learning multiple visual domains with residual adapters (NeurIPS 2017 · 562 citations)
- Listening to the World Improves Speech Command Recognition (AAAI 2018 · 37 citations)
- GLoMo: Unsupervised Learning of Transferable Relational Graphs (NeurIPS 2018 · 21 citations)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
ADE20K · mIoU (higher is better)
8 papers, 2022–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2022 | 57.9 | BEiT: BERT Pre-Training of Image Transformers (ICLR 2022 · 915 citations) |
ImageNet-1K · Top-1 accuracy (higher is better)
7 papers, 2017–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2017 | 59.67 | Learning multiple visual domains with residual adapters (NeurIPS 2017 · 562 citations) |
| 2021 | 82.1 | Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model (NeurIPS 2021 · 13 citations) |
| 2022 | 89.5 | BEiT: BERT Pre-Training of Image Transformers (ICLR 2022 · 915 citations) |
| 2023 | 89.6 | Image as a Foreign Language: BEiT Pretraining for Vision and Vision-Language Tasks (CVPR 2023 · 531 citations) |
GQA · accuracy (higher is better)
6 papers, 2024–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 61.8 | LocCa: Visual Pretraining with Location-aware Captioners (NeurIPS 2024 · 2 citations) |
| 2025 | 77 | R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Experts (ICML 2025 · 0 citations) |
GQA · score (higher is better)
5 papers, 2024–2025 · flat since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2024 | 63.9 | UNIT: Unifying Image and Text Recognition in One Vision Encoder (NeurIPS 2024 · 2 citations) |
COCO captioning · CIDEr (higher is better)
4 papers, 2023–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 130.4 | Write and Paint: Generative Vision-Language Models are Unified Modal Learners (ICLR 2023) |
ImageNet-1k · accuracy (higher is better)
4 papers, 2023–2024 · not reported since 2024
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 87.2 | Uni-Perceiver v2: A Generalist Model for Large-Scale Vision and Vision-Language Tasks (CVPR 2023 · 27 citations) |
What has been tried
137 papers here report 1,565 measured changes (a component added, removed or swapped, compared with a baseline): helped 72%, no clear effect 6%, hurt 20%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add single-image, video, and annotation data components | ImageNet validation set, four prompted tasks · perplexity | Helped | Sequential Modeling Enables Scalable Learning for Large Vision Models (CVPR 2024 · 102 citations) | |
| Use two pre-training scales instead of one | fMoW-Sentinel validation · Top1 Acc. | 61.61 -> 62.86 | Helped | Rethinking Transformers Pre-training for Multi-Spectral Satellite Imagery (CVPR 2024 · 66 citations) |
| Add language features to vision-only features | Harvard-FairVLMed race · ES-AUC | 73.76 -> 76.70 | Helped | FairCLIP: Harnessing Fairness in Vision-Language Learning (CVPR 2024 · 52 citations) |
| decouple visual encoding for understanding and generation | POPE · score | 60.1 -> 87.0 | Helped | Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (CVPR 2025 · 44 citations) |
| Add retrieval-augmented training | In-domain evaluation · average overall score | 0.55 -> 0.79 (+0.25) | Helped | Instruct-Imagen: Image Generation with Multi-modal Instruction (CVPR 2024 · 38 citations) |
| Pretrain Movie-BLIP2 on HowTo-AD | CMD-AD-Eval · CRITIC | 24.5 -> 30.2 | Helped | AutoAD III: The Prequel - Back to the Pixels (CVPR 2024 · 24 citations) |
| Add shared mapping between semantic and pixel codebooks | ImageNet 50k validation · rFID | 8.07 -> 3.96 | Helped | TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation (CVPR 2025 · 22 citations) |
| Add representation alignment regularization | MJHQ FID-10k · FID | 19.84 -> 17.61 | Helped | JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation (CVPR 2025 · 20 citations) |
| Include the drawing's annotation layer | Cabinet test set · reconstruction F1 | 62.65 -> 82.76 | Helped | From 2D CAD Drawings to 3D Parametric Models: A Vision-Language Approach (AAAI 2025 · 16 citations) |
| Add dynamic template patches by setting DTCM top-k to 3 instead of 0 | LaSOT · AUC | 71.0 -> 71.7 | Helped | Dynamic Updates for Language Adaptation in Visual-Language Tracking (CVPR 2025 · 16 citations) |
| Add global tokens to the local-token reconstruction setup | ImageNet-1K val reconstruction · FID | 3.41 -> 2.88 | Helped | Beyond Text: Frozen Large Language Models in Visual Signal Comprehension (CVPR 2024 · 15 citations) |
| Add C4 co-training to CC12M | ImageNet-1K · Top-1 accuracy | 84.4 -> 84.6 | Helped | 4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities (NeurIPS 2024 · 15 citations) |
What reviewers will expect
Datasets and baselines named in the 49 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- ImageNet-1k · 9 of 49 papers
- CC12M · 5 of 49 papers
- ShareGPT4V · 4 of 49 papers
- JourneyDB · 4 of 49 papers
- ImageNet · 4 of 49 papers
- CC3M · 3 of 49 papers
- COYO · 3 of 49 papers
- OpenImages · 3 of 49 papers
Baselines
- Show-o · 10 of 49 papers
- SDXL · 6 of 49 papers
- LLaVA-v1.5 · 6 of 49 papers
- Emu3 · 6 of 49 papers
- Transfusion · 5 of 49 papers
- Chameleon · 5 of 49 papers
- VILA-U · 5 of 49 papers
- Janus · 5 of 49 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 147 papers; from OpenAlex and the papers' affiliations).
- Tsinghua University · 8 papers
- ByteDance · 8 papers
- Shanghai Jiao Tong University · 7 papers
- Google · 7 papers
- Peking University · 7 papers
- Peng Cheng Laboratory · 6 papers
- University of Washington · 6 papers
- Microsoft · 6 papers
- The Chinese University of Hong Kong · 5 papers
- Tencent · 5 papers
Use it
Download BibTeX of 39 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Medical vision-language models (130 papers)
- Layout-aware document understanding (73 papers)
- Text-driven human motion generation (36 papers)
Elsewhere in the topic
- Scene graph generation (126 papers)
- Language-grounded scene text and structure (124 papers)
- fMRI visual decoding (106 papers)
- Human-like visual cognition in VLMs (97 papers)
- Multimodal and affective interaction modeling (87 papers)
- Multimodal retrieval-augmented modeling (85 papers)
- Language-grounded visual concept learning (83 papers)
- Composed and text-based image retrieval (78 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).