Research map / Vision-language and multimodal / CLIP and zero-shot
BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literature
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds BIOMEDICA, a streaming ETL framework and 24M image-caption dataset from PubMed Central Open Access with expert-annotated concepts, and continually pretrains OpenCLIP into BMC-CLIP models evaluated on 40 biomedical tasks.
Problem
Generalist biomedical vision-language models are limited by the lack of large, annotated, publicly accessible image-caption datasets spanning the full breadth of biomedical literature, and existing biomedical CLIP evaluations are narrow in domain.
Approach
Extract the entire PMC Open Access subset into a domain-agnostic, metadata-rich WebDataset archive, label image clusters (DINOv2 features, PCA, k-means) with expert-derived hierarchical concepts propagated to all images, and continually pretrain OpenCLIP on it via streaming with concept filtering/balancing.
Key results
- BMC-CLIP (concept-filtered) achieves 6.56% average improvement over BioMedCLIP on general biomedical classification, with +29.8% in dermatology and +17.5% in ophthalmology.
- BMC-CLIP concept-filtered reaches 38.30 image-to-text recall@100 on BioMed-Flickr vs 36.27 for BioMedCLIP.
- Models reportedly use 10x less compute and 2.5x less data than BioMedCLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use concept filtering rather than full-dataset continual pretraining (ablation) | BMC-CLIP · General bioimaging classification (35 tasks) | average accuracy: 41.35 -> 47.72 | Helped |
| Use concept filtering rather than concept balancing (ablation) | BMC-CLIP · General bioimaging classification (35 tasks) | average accuracy: 44.89 -> 47.72 | Helped |
| Use concept filtering rather than full-dataset continual pretraining (ablation) | BMC-CLIP · BioMed-Flickr | image-to-text recall@100: 31.93 -> 38.30 | Helped |
| Use concept filtering rather than full-dataset continual pretraining (ablation) | BMC-CLIP · BioMed-Flickr | text-to-image recall@100: 30.03 -> 36.10 | Helped |
| Apply WiSE-FT weight merging to BMC-CLIPCF (ablation) | BMC-CLIPCF · Microscopy composition identification | average accuracy: 50.37 -> 58.53 | Helped |
| Apply WiSE-FT weight merging to BMC-CLIPCF (ablation) | BMC-CLIPCF · General bioimaging classification | average accuracy | Mixed |
Surprising findings
- Concept-balancing and full-dataset pretraining were outperformed by concept filtering, which drops over-represented topics such as plots and charts.
- WiSE-FT model merging improves some tasks but decreases performance in dermatology and multimodal retrieval recall.
Limitations
- CLIP ViT-L-14 context length of 77 tokens truncates long captions (median 64 tokens, but tail is longer).
- Images are uniformly resized without handling varied sizes and aspect ratios, losing detail in high-resolution figures.
Experimental setup
Datasets: BIOMEDICA (PMC-OA derived, 24M image-caption pairs) BioMed-Flickr (7K retrieval set) Micro-Bench MedMNIST RSNA CheXpert Dresden surgical dataset
Benchmarks: 40-task biomedical zero-shot classification suite BioMed-Flickr image-text retrieval Micro-Bench microscopy composition identification
Metrics: average accuracy recall@1 recall@10 recall@100
Baselines: OpenCLIP CoCa PMC-CLIP BioMedCLIP
Builds on: OpenCLIP DINOv2 WiSE-FT DataComp
Closest papers by meaning
- BiomedCoOp: Learning to Prompt for Biomedical Vision-Language Models (CVPR 2025 · similarity 0.89)
- LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding (CVPR 2026 · similarity 0.88)
- Micro-Bench: A Microscopy Benchmark for Vision-Language Understanding (NeurIPS 2024 · similarity 0.88)
- PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaboration (ICLR 2025 · similarity 0.87)
- MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine (ICLR 2025 · similarity 0.87)
- SlideChat: A Large Vision-Language Assistant for Whole-Slide Pathology Image Understanding (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.