Research map / Vision-language and multimodal / Human and embodied modeling
Medical vision-language models
Pretrains medical vision-language models for radiology reports and 3D imaging.
130 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2017–2026) · an approach in Human and embodied modeling, keyword cluster “pre · generation · unified”
State History Benchmarks Tried Checklist Preprints Who Export
Where it stands
Still moving The best reported result is still improving on 4 of 6 tracked benchmarks.
40 papers in 2023–24 and 78 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic grew, from 7.5% to 11.7%.
Open questions the data points to
- IU-Xray test (BLEU-4): best went from 0.185 (up to 2024) to 0.225 in 2025 — DAMPER: A Dual-Stage Medical Report Generation Framework with Coarse-Grained MeSH Alignment and Fine-Grained Hypergraph Matching (AAAI 2025 · 4 citations)
- MIMIC-CXR test (BLEU-1): best went from 0.398 (up to 2024) to 0.402 in 2025 — DAMPER: A Dual-Stage Medical Report Generation Framework with Coarse-Grained MeSH Alignment and Fine-Grained Hypergraph Matching (AAAI 2025 · 4 citations)
- MIMIC-CXR test (BLEU-4): best went from 0.124 (up to 2024) to 0.193 in 2025 — DAMPER: A Dual-Stage Medical Report Generation Framework with Coarse-Grained MeSH Alignment and Fine-Grained Hypergraph Matching (AAAI 2025 · 4 citations)
- MIMIC-CXR test (F1): best went from 0.476 (up to 2024) to 0.507 in 2025 — DAMPER: A Dual-Stage Medical Report Generation Framework with Coarse-Grained MeSH Alignment and Fine-Grained Hypergraph Matching (AAAI 2025 · 4 citations)
No longer reported by papers here: IU-Xray (CIDEr, last in 2023).
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2017 · MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network (CVPR 2017 · 316 citations)
- 2019 · Knowledge-Driven Encode, Retrieve, Paraphrase for Medical Image Report Generation (AAAI 2019)
- 2020 · Learning Conceptual-Contextual Embeddings for Medical Text (AAAI 2020 · 10 citations)
- 2021 · Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation (CVPR 2021 · 346 citations)
- 2022 · Clinical-BERT: Vision-Language Pre-training for Radiograph Diagnosis and Reports Generation (AAAI 2022 · 88 citations)
- 2023 · METransformer: Radiology Report Generation by Transformer With Multiple Learnable Expert Tokens (CVPR 2023 · 172 citations)
- 2024 · SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery (CVPR 2024 · 251 citations)
- 2025 · The OMG dataset: An Open MetaGenomic corpus for mixed-modality genomic language modeling (ICLR 2025 · 44 citations)
- 2026 · PriorRG: Prior-Guided Contrastive Pre-training and Coarse-to-Fine Decoding for Chest X-ray Report Generation (AAAI 2026 · 4 citations)
Earliest papers here
- MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network (CVPR 2017 · 316 citations)
- Condensed Memory Networks for Clinical Diagnostic Inferencing (AAAI 2017 · 77 citations)
- Knowledge-Driven Encode, Retrieve, Paraphrase for Medical Image Report Generation (AAAI 2019)
Benchmark records
The best number each year that beat every earlier one, among papers of this approach reporting their own method. Numbers are extracted by AI from each paper's text; check the paper before citing.
IU X-Ray test · BLEU-1 (higher is better)
4 papers, 2023–2025 · one report since 2024, not better
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 0.525 | KiUT: Knowledge-Injected U-Transformer for Radiology Report Generation (CVPR 2023 · 132 citations) |
IU-Xray · CIDEr (higher is better)
4 papers, 2019–2023 · not reported since 2023
| Year | Best so far | Set by |
|---|---|---|
| 2019 | 0.28 | Knowledge-Driven Encode, Retrieve, Paraphrase for Medical Image Report Generation (AAAI 2019) |
| 2021 | 0.351 | Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation (CVPR 2021 · 346 citations) |
| 2022 | 0.432 | Clinical-BERT: Vision-Language Pre-training for Radiograph Diagnosis and Reports Generation (AAAI 2022 · 88 citations) |
| 2023 | 0.435 | METransformer: Radiology Report Generation by Transformer With Multiple Learnable Expert Tokens (CVPR 2023 · 172 citations) |
IU-Xray test · BLEU-4 (higher is better)
4 papers, 2023–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 0.185 | KiUT: Knowledge-Injected U-Transformer for Radiology Report Generation (CVPR 2023 · 132 citations) |
| 2025 | 0.225 | DAMPER: A Dual-Stage Medical Report Generation Framework with Coarse-Grained MeSH Alignment and Fine-Grained Hypergraph Matching (AAAI 2025 · 4 citations) |
MIMIC-CXR test · BLEU-1 (higher is better)
4 papers, 2023–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 0.393 | KiUT: Knowledge-Injected U-Transformer for Radiology Report Generation (CVPR 2023 · 132 citations) |
| 2024 | 0.398 | PromptMRG: Diagnosis-Driven Prompts for Medical Report Generation (AAAI 2024 · 123 citations) |
| 2025 | 0.402 | DAMPER: A Dual-Stage Medical Report Generation Framework with Coarse-Grained MeSH Alignment and Fine-Grained Hypergraph Matching (AAAI 2025 · 4 citations) |
MIMIC-CXR test · BLEU-4 (higher is better)
4 papers, 2023–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 0.124 | METransformer: Radiology Report Generation by Transformer With Multiple Learnable Expert Tokens (CVPR 2023 · 172 citations) |
| 2025 | 0.193 | DAMPER: A Dual-Stage Medical Report Generation Framework with Coarse-Grained MeSH Alignment and Fine-Grained Hypergraph Matching (AAAI 2025 · 4 citations) |
MIMIC-CXR test · F1 (higher is better)
4 papers, 2023–2025 · still improving
| Year | Best so far | Set by |
|---|---|---|
| 2023 | 0.321 | KiUT: Knowledge-Injected U-Transformer for Radiology Report Generation (CVPR 2023 · 132 citations) |
| 2024 | 0.476 | PromptMRG: Diagnosis-Driven Prompts for Medical Report Generation (AAAI 2024 · 123 citations) |
| 2025 | 0.507 | DAMPER: A Dual-Stage Medical Report Generation Framework with Coarse-Grained MeSH Alignment and Fine-Grained Hypergraph Matching (AAAI 2025 · 4 citations) |
What has been tried
97 papers here report 1,047 measured changes (a component added, removed or swapped, compared with a baseline): helped 74%, no clear effect 5%, hurt 18%, mixed 2%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add Multi-Granularity Contrastive Learning (MGCL) during pre-training | Dyna.-MM · mIoU | 42.2 -> 44.4 | Helped | SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery (CVPR 2024 · 251 citations) |
| Use pathway tokens from both Reactome and Hallmarks instead of a single transcriptomics token | Overall · c-index | 0.586 -> 0.629 | Helped | Modeling Dense Multimodal Interactions Between Biological Pathways and Histology for Survival Prediction (CVPR 2024 · 143 citations) |
| add diagnosis-driven token prompts | MIMIC-CXR test · F1 | 0.370 -> 0.444 | Helped | PromptMRG: Diagnosis-Driven Prompts for Medical Report Generation (AAAI 2024 · 123 citations) |
| Progressively pretrain on fMoW-S2 and BigEarthNet instead of fMoW-S2 alone | BigEarthNet · mAP | 85.59 -> 87.41 | Helped | S2MAE: A Spatial-Spectral Pretraining Foundation Model for Spectral Remote Sensing Data (CVPR 2024 · 41 citations) |
| add CLIP-derived disorder context (CLIP+GPT-3.5 knowledge) to the query prompt | MMQS · ROUGE-1 | 0.451 -> 0.452 | No clear effect | CLIPSyntel: CLIP and LLM Synergy for Multimodal Question Summarization in Healthcare (AAAI 2024 · 36 citations) |
| Add DMM and RFA | All-weather · PSNR | 27.93 -> 29.75 | Helped | Language-driven All-in-one Adverse Weather Removal (CVPR 2024 · 36 citations) |
| Pretrain from scratch rather than distilling ImageNet22K features | BigEarthNet 10% · mAP | 86.4 -> 87.2 | Helped | Bridging Remote Sensors with Multisensor Geospatial Foundation Models (CVPR 2024 · 33 citations) |
| Add Memory Alignment using visual embeddings | MIMIC-CXR · BLEU-1 | 0.357 -> 0.389 | Helped | Automatic Radiology Reports Generation via Memory Alignment Network (AAAI 2024 · 29 citations) |
| remove DCN and use fixed-length time-domain signals | MBHM · accuracy | 99.02% -> 72.15% | Hurt | BearLLM: A Prior Knowledge-Enhanced Bearing Health Management Framework with Unified Vibration Signal Representation (AAAI 2025 · 22 citations) |
| add street-view branch to the satellite branch | CityView-Beijing · R² | 3.56% | Helped | UrbanVLP: Multi-Granularity Vision-Language Pretraining for Urban Socioeconomic Indicator Prediction (AAAI 2025 · 17 citations) |
| add multi-resolution training | ADE20k · mIoU | +2.8 on ADE20k | Helped | RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Models (CVPR 2025 · 17 citations) |
| add SVG-specific data augmentations (rotation, color, and curve noise) | SVG-Emoji and SVG-Icons · performance | Helped | StarVector: Generating Scalable Vector Graphics Code from Images and Text (AAAI 2025 · 16 citations) |
What reviewers will expect
Datasets and baselines named in the 55 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- MIMIC-CXR · 6 of 55 papers
- MoleculeNet · 4 of 55 papers
- MIMIC-IV · 3 of 55 papers
- ChEBI-20 · 3 of 55 papers
- IU-Xray · 2 of 55 papers
- UMLS · 2 of 55 papers
- COCO · 2 of 55 papers
- ADE20K · 2 of 55 papers
Baselines
- MolT5 · 4 of 55 papers
- GPT-4o · 4 of 55 papers
- BiomedCLIP · 3 of 55 papers
- TransMIL · 3 of 55 papers
- DSMIL · 3 of 55 papers
- Text+Chem T5 · 3 of 55 papers
- ABMIL · 3 of 55 papers
- MolFM · 3 of 55 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 118 papers; from OpenAlex and the papers' affiliations).
- Stanford University · 6 papers
- Shanghai Jiao Tong University · 5 papers
- Peking University · 5 papers
- Chinese Academy of Sciences · 5 papers
- Harvard University · 5 papers
- Hong Kong University of Science and Technology · 4 papers
- The University of Hong Kong · 4 papers
- NVIDIA · 4 papers
- Imperial College London · 4 papers
- Amazon · 3 papers
Use it
Download BibTeX of 31 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Vision-language pretraining (182 papers)
- Layout-aware document understanding (73 papers)
- Text-driven human motion generation (36 papers)
Elsewhere in the topic
- Scene graph generation (126 papers)
- Language-grounded scene text and structure (124 papers)
- fMRI visual decoding (106 papers)
- Human-like visual cognition in VLMs (97 papers)
- Multimodal and affective interaction modeling (87 papers)
- Multimodal retrieval-augmented modeling (85 papers)
- Language-grounded visual concept learning (83 papers)
- Composed and text-based image retrieval (78 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).