Research map / Vision-language and multimodal / Human and embodied modeling
MedicalNarratives: Connecting Medical Vision and Language with Localized Narratives
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Curates MedicalNarratives, a 4.7M medical image-text dataset from YouTube pedagogy videos (with cursor traces for 1M samples) and PubMed figures, and trains GenMedCLIP, a CLIP-style model that beats BiomedCLIP on a new 29-dataset medical benchmark.
Problem
Medical vision-language models lack large, spatially grounded image-text data; existing medical datasets either lack spatial annotation or rely on model-generated labels.
Approach
Harvest grounded medical narratives from YouTube pedagogy videos where instructors speak while moving the cursor over image regions (localized-narrative style traces), combined with PubMed figure-caption pairs, and train a CLIP-like model on this mix.
Key results
- GenMedCLIP zero-shot classification average 32.55% vs BiomedCLIP 27.80% across the benchmark.
- GenMedCLIP retrieval average 70.17 vs BiomedCLIP 44.10 on held-out 16K pairs.
- Video data increases average retrieval by 53.6% and classification by 11.65% in the data split ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train on articles and videos instead of articles only (ablation) | GenMedCLIP · Medical benchmark (11 domains; shorter fine-tuning) | overall: 20.84 -> 32.49 | Helped |
| train on articles and videos instead of articles only (ablation) | GenMedCLIP · MedicalNarratives held-out set (shorter fine-tuning) | retrieval Avg: 12.70 -> 66.30 | Helped |
| pretrain on MedicalNarratives and use GenMedCLIP rather than Biomed | medical benchmark (11 domains) · Medical benchmark (11 domains) | overall: 27.43 -> 31.99 | Helped |
| pretrain on MedicalNarratives and use GenMedCLIP rather than Biomed | MedicalNarratives held-out set · MedicalNarratives held-out set | retrieval Avg: 44.10 -> 70.17 | Helped |
| pretrain on MedicalNarratives and use GenMedCLIP rather than PubMedCLIP | linear probing · Linear probing (Isic, Til, Pcam, Mhist, Nck, Mammo) | accuracy Avg: 82.06 -> 83.97 | Helped |
Surprising findings
- Generic CLIP-ViT-B-16 outperforms the in-domain PubMedCLIP on classification and retrieval, per the authors.
- Models trained on article-only data are comparable to video-inclusive models on classification, except BiomedCLIP.
Limitations
- Dataset lacks human-annotated bounding boxes, so trace quality is not validated against expert annotations.
- Dataset overrepresents abnormal cases, introducing bias.
- Traces are not used to train any spatially-aware model in this work.
Experimental setup
Datasets: MedicalNarratives PMC-OA MedICaT ImageCLEF 2016 RadImageNet VinDr-CXR VinDr-Mammo Dresden GastroVision DDI ISIC 2018 G1020
Benchmarks: RadImageNet VinDr-CXR VinDr-Mammo Dresden GastroVision DDI ISIC 2018 G1020 OCTDL PatchCamelyon NCT-CRC-HE-100K BACH
Metrics: accuracy mean average precision recall@5 recall@50 recall@200
Baselines: CLIP-ViT-B-16 PubMedCLIP BiomedCLIP PMC-CLIP
Builds on: CLIP Localized Narratives Quilt-1M Quilt-LLaVA BiomedCLIP
Closest papers by meaning
- MedHorizon: Towards Long-context Medical Video Understanding in the Wild (NeurIPS 2026 · similarity 0.87)
- MedMax: Mixed-Modal Instruction Tuning for Training Biomedical Assistants (NeurIPS 2025 · similarity 0.87)
- MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine (ICLR 2025 · similarity 0.86)
- BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literature (CVPR 2025 · similarity 0.86)
- OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM (CVPR 2024 · similarity 0.86)
- MedUnifier: Unifying Vision-and-Language Pre-training on Medical Data with Vision Generation Task using Discrete Visual Representations (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.