Research map / Vision-language and multimodal / CLIP and zero-shot
BiomedCoOp: Learning to Prompt for Biomedical Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BiomedCoOp, a prompt-learning framework that adapts BiomedCLIP for few-shot and base-to-novel biomedical image classification using GPT-4-generated prompt ensembles, LLM-prompt semantic consistency loss, and statistics-based (MAD) prompt selection with knowledge distillation.
Problem
Adapting vision-language models to few-shot biomedical image classification is hard: prompt engineering is expertise-intensive, full fine-tuning is costly, and existing prompt learning methods built on natural-image CLIP generalize poorly to unseen biomedical classes.
Approach
Learnable context vectors on BiomedCLIP are regularized toward the mean embedding of GPT-4-generated class-specific visual descriptions (semantic consistency loss), and a KL distillation term uses only outlier-filtered LLM prompts selected by a median-absolute-deviation modified z-score.
Key results
- Averaged over 11 datasets, BiomedCoOp reaches 57.03% at 1-shot and 72.42% at 16-shot, versus 50.16% and 69.62% for CoOp and 51.88% and 67.13% for ProGrad at 1- and 16-shot (ProGrad at 67.13 is 16-shot).
- In base-to-novel evaluation, BiomedCoOp achieves an average harmonic mean of 75.07 versus 67.23 for CoOp and 67.43 for ProGrad over 10 datasets.
- Ablation: adding both SCCM and KDSP to BiomedCLIP raises 16-shot accuracy from 54.80 to 72.42 in the component table's setting, and base-to-novel HM from 53.81 (zero-shot baseline row) to 74.64 in the full model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SCCM (LLM prompt ensemble semantic consistency) on BiomedCLIP (ablation) | BiomedCoOp · 11 biomedical benchmarks (few-shot and base-to-novel) | HM / K=16 accuracy: HM 67.23 -> 70.19; K=16 69.62 -> 69.99 (per Table 3 rows) | Helped |
| Add KDSP (statistics-based outlier-excluded prompt selection with KD) on BiomedCLIP (ablation) | BiomedCoOp · 11 biomedical benchmarks (few-shot and base-to-novel) | HM / K=16 accuracy: HM 67.23 -> 71.79 (without SCCM, 74.30 with SCCM); K=16 69.62 -> 71.79 | Helped |
| Combine SCCM and KDSP (ablation) | BiomedCoOp · 11 biomedical benchmarks | HM: HM 74.64 with both vs 70.19 (SCCM only) and 74.30 (KDSP only) | Helped |
| Use BiomedCLIP instead of CLIP backbone (ablation) | BiomedCoOp · few-shot K=1 to 16 | accuracy (%): BiomedCoOp reaches 72.42 at K=16 with BiomedCLIP, outperforming CLIP, PubMedCLIP and PMC-CLIP backbones | Helped |
| Increase number of LLM prompts from 10 to 50 (ablation) | BiomedCoOp · few-shot K=0 to 16 | accuracy (%): K=0: 47.55 -> 52.27; K=1: 52.43 -> 57.03; gains diminish at K=8 and 16 | Helped |
| Use BiomedCoOp instead of prior prompt learning methods | BiomedCLIP-based prompt learning · 11 biomedical benchmarks few-shot K=1 | accuracy (%): 57.03 vs 51.88 for ProGrad (+5.2) | Helped |
| Use BiomedCoOp instead of prior prompt learning methods | base-to-novel generalization · 11 biomedical benchmarks | HM (%): 75.07 vs 67.43 for ProGrad | Helped |
| Use BiomedCoOp instead of prior prompt learning methods | few-shot K=16 · 11 biomedical benchmarks | accuracy (%): 72.42 vs 70.91 for Tip-Adapter-F (best baseline) | Helped |
Surprising findings
- Using KDSP alone on the CLIP-only setup hurt generalization, indicating outlier pruning needs domain knowledge from a biomedical VLM to be effective.
Limitations
- Evaluated only on 11 medical image classification datasets with shallow text prompts; deep prompt and vision-side prompt methods were excluded from comparison.
- Relies on GPT-4 generated descriptions, whose quality and cost are not analyzed in depth.
- Some reported gains are small or within standard deviation on several datasets (e.g., Kvasir novel classes, CHMNIST).
Experimental setup
Datasets: CTKidney DermaMNIST Kvasir RETINA LC25000 CHMNIST BTMRI OCTMNIST BUSI COVID-QU-Ex KneeXray
Benchmarks: CTKidney DermaMNIST Kvasir RETINA LC25000 CHMNIST BTMRI OCTMNIST BUSI COVID-QU-Ex KneeXray
Metrics: classification accuracy harmonic mean (HM) standard deviation over 3 support sets
Baselines: CoOp CoCoOp KgCoOp ProGrad CLIP-Adapter Tip-Adapter Tip-Adapter-F Standard LP LP++ BiomedCLIP zero-shot
Builds on: CoOp CoCoOp BiomedCLIP KgCoOp ProGrad
Closest papers by meaning
- BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literature (CVPR 2025 · similarity 0.89)
- LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding (CVPR 2026 · similarity 0.89)
- Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for zero-shot medical detection (ICLR 2025 · similarity 0.88)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.87)
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.87)
- KPL: Training-Free Medical Knowledge Mining of Vision-Language Models (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.