Research map / Vision-language and multimodal / CLIP and zero-shot
KPL: Training-Free Medical Knowledge Mining of Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KPL, a training-free CLIP inference method that retrieves image-relevant LLM-generated medical descriptions to refine text proxies and then learns multimodal proxies via a log-space Stable Greenkhorn algorithm for zero-shot medical image classification.
Problem
Zero-shot image classification with CLIP performs poorly on medical images because single class names are inadequate, descriptions are not image-relevant, and the modal gap between visual and text spaces hurts proxy quality; existing Sinkhorn-based proxy learning is unstable on medical data.
Approach
Builds an LLM-generated knowledge-enhanced base of descriptions per class, retrieves the top-k descriptions most similar to the dataset's mean image feature (visual-based retrieval) to form text proxies, and uses these as guidance for multimodal proxy learning with Stable Greenkhorn, a log-domain greedy variant of Sinkhorn that updates the most imbalanced row or column.
Key results
- On medical datasets with CLIP ViT-L/14@336px, KPL reaches 78.55 (Shenzhen), 47.09 (IDRiD), 80.86 (MalariaCell), 44.43 (Cataract), 70.29 (Montgomery), outperforming InMaP and VCD baselines.
- On MalariaCell with CLIP ViT-L/14, KPL raises accuracy from 48.53 (CLIP) to 77.64.
- Applying KPL to BioMedCLIP ViT-B/16 improves average medical accuracy, e.g. MalariaCell 72.13 to 91.61 and Cataract 16.47 to 51.25.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use KPL instead of CLIP inference | CLIP ViT-L/14 · MalariaCell | accuracy: 48.53 -> 77.64 | Helped |
| Use KPL instead of CLIP inference | CLIP ViT-L/14@336px · MalariaCell | accuracy: 50.83 -> 80.86 | Helped |
| Add KEB while retaining retrieval and SG (ablation) | KPL · MalariaCell | accuracy: 60.98 -> 80.86 | Helped |
| Add KEB while retaining retrieval and SG (ablation) | KPL · Cataract | accuracy: 31.11 -> 44.43 | Helped |
| Add KEB while retaining retrieval and SG (ablation) | KPL · IDRiD | accuracy: 46.51 -> 47.09 | Helped |
| Add SG instead of Sinkhorn (ablation) | KPL · Shenzhen | accuracy: 72.81 -> 78.55 | Helped |
| Add SG instead of Sinkhorn (ablation) | KPL · IDRiD | accuracy: 32.36 -> 47.09 | Helped |
| Add SG instead of Sinkhorn (ablation) | KPL · MalariaCell | accuracy: 60.02 -> 80.86 | Helped |
| Add SG instead of Sinkhorn (ablation) | KPL · Cataract | accuracy: 32.45 -> 44.43 | Helped |
| Add SG instead of Sinkhorn (ablation) | KPL · Montgomery | accuracy: 68.84 -> 70.29 | Helped |
| Use visually retrieved descriptions rather than randomly selected descriptions (ablation) | KPL with KEB · Medical datasets | accuracy | Helped |
| Use KEB descriptions rather than VCD descriptions (ablation) | KPL · Medical datasets | accuracy | Helped |
Surprising findings
- Sinkhorn-based pseudo labels did not consistently improve medical image classification, and replacing SG with Sinkhorn lowered performance across datasets.
- CLIP ViT-L/14 with KPL surpassed BioMedCLIP with KPL on most medical datasets despite BioMedCLIP being trained on medical data.
Limitations
- Sinkhorn and SG-based proxy learning results on natural datasets use KPL with Sinkhorn rather than SG, indicating SG did not uniformly help there.
- Only the ViT-B/16 BioMedCLIP variant was publicly available for testing.
- Hyperparameters k, N_max and tau are selected by grid search, which may depend on the evaluation data.
Experimental setup
Datasets: Shenzhen IDRiD MalariaCell Cataract Montgomery CUB Places365 Oxford Pets ImageNet
Benchmarks: Shenzhen IDRiD MalariaCell Cataract Montgomery CUB Places365 Oxford Pets ImageNet
Metrics: classification accuracy
Baselines: CLIP VCD CMD InMaP Hierarchical
Builds on: CLIP InMaP Sinkhorn Greenkhorn VCD
Closest papers by meaning
- Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for zero-shot medical detection (ICLR 2025 · similarity 0.88)
- MLIP: Enhancing Medical Visual Representation with Divergence Encoder and Knowledge-guided Contrastive Learning (CVPR 2024 · similarity 0.87)
- Bringing CLIP to the Clinic: Dynamic Soft Labels and Negation-Aware Learning for Medical Analysis (CVPR 2025 · similarity 0.87)
- Interpreting and Analysing CLIP's Zero-Shot Image Classification via Mutual Knowledge (NeurIPS 2024 · similarity 0.87)
- BiomedCoOp: Learning to Prompt for Biomedical Vision-Language Models (CVPR 2025 · similarity 0.87)
- FOCUS: Knowledge-enhanced Adaptive Visual Compression for Few-shot Whole Slide Image Classification (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.