Research map / Vision-language and multimodal / CLIP and zero-shot
CPLIP: Zero-Shot Learning for Histopathology with Comprehensive Vision-Language Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes PLIP (CLIP-style) on unpaired, LLM- and retrieval-augmented histology image and text bags using many-to-many MIL-NCE contrastive learning, improving zero-shot tile, WSI and segmentation performance.
Problem
Zero-shot vision-language transfer in histopathology is limited by scarce paired image-text data and by models aligning each image with only one short textual prompt.
Approach
Builds bags of comprehensive textual descriptions (from a pathology dictionary, GPT-3 expansions, and PLIP retrieval) and bags of retrieved histology images, then aligns the two bags with a many-to-many MIL-NCE loss, without ground-truth annotations or paired captions.
Key results
- On CRC100K tile-level zero-shot, CPLIP reaches 0.701 balanced accuracy versus 0.566 for CONCH and 0.544 for MI-Zero (single prompt).
- On WSI-level BRCA subtyping, CPLIP2 (in-domain) reaches 82.4% balanced accuracy, 18.1% above CONCH and 4.3% above MI-Zero.
- Bag pruning at 90% matching gave the best results across datasets; using only the visual bag or only the text bag degraded performance versus the joint bags.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CPLIP instead of the strongest listed baseline | zero-shot tile classification · CRC100K | weighted F1: 0.542 -> 0.681 | Helped |
| Use CPLIP instead of the strongest listed baseline | zero-shot tile classification · DigestPath | weighted F1: 0.832 -> 0.856 | Helped |
| Use CPLIP instead of the strongest listed baseline | zero-shot tile classification · SICAP | weighted F1: 0.255 -> 0.388 | Helped |
| Use CPLIP instead of the strongest listed baseline | zero-shot tile classification · WSSS4LUAD | weighted F1: 0.742 -> 0.791 | Helped |
| Use CPLIP instead of the strongest listed baseline | zero-shot tile classification · PanNuke | weighted F1: 0.688 -> 0.757 | Helped |
| Use CPLIP2 instead of the strongest listed baseline | zero-shot WSI classification · CAM16 | weighted F1: 0.461 -> 0.587 | Helped |
| Use CPLIP2 instead of the strongest listed baseline | zero-shot WSI classification · TCGA-BRCA | weighted F1: 0.600 -> 0.786 | Helped |
| Use CPLIP2 instead of the strongest listed baseline | zero-shot WSI classification · TCGA-RCC | weighted F1: 0.797 -> 0.855 | Helped |
| Use CPLIP2 instead of the strongest listed baseline | zero-shot WSI classification · TCGA-NSCLC | weighted F1: 0.803 -> 0.835 | Helped |
| Use a cleaned D200 dictionary instead of uncleaned D500 (ablation) | CPLIP · CRC100K | weighted F1: 0.804 -> 0.844 | Helped |
| Use ARCH paired text instead of the unsupervised dictionary (ablation) | CPLIP textual-bag construction · CRC100K | weighted F1: 0.844 -> 0.861 | Helped |
| Add GPT-3 and PLIP textual and visual enrichment to D200 (ablation) | CPLIP · CRC100K | weighted F1: 0.697 -> 0.844 | Helped |
Surprising findings
- Cleaning the pathology dictionary from 500 to 200 prompts improved zero-shot performance over the larger uncleaned set.
Limitations
- Relies on the single public ARCH image-caption dataset and noisy GPT-3 and MI-Zero/PLIP-derived text, with bag sizes capped at 17 texts and 21 images for compute reasons.
- CONCH results on some datasets are missing or use different splits, complicating comparison.
Experimental setup
Datasets: ARCH CRC100K WSSS4LUAD PanNuke DigestPath SICAP CAMELYON16 TCGA-BRCA TCGA-RCC TCGA-NSCLC
Benchmarks: CRC100K DigestPath SICAP WSSS4LUAD PanNuke CAMELYON16 TCGA-BRCA TCGA-RCC TCGA-NSCLC
Metrics: balanced accuracy weighted F1 AUROC Dice score precision recall
Baselines: CLIP BiomedCLIP PLIP MI-Zero CONCH
Builds on: CLIP PLIP MI-Zero MIL-NCE
Closest papers by meaning
- Multi-Resolution Pathology-Language Pre-training Model with Text-Guided Visual Representation (CVPR 2025 · similarity 0.90)
- Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Framework (CVPR 2024 · similarity 0.88)
- CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathology (CVPR 2025 · similarity 0.88)
- Prompt as Knowledge Bank: Boost Vision-language model via Structural Representation for zero-shot medical detection (ICLR 2025 · similarity 0.87)
- PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaboration (ICLR 2025 · similarity 0.87)
- PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology (AAAI 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.