Research map / Vision-language and multimodal / CLIP and zero-shot
CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathology
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds CPath-Omni, a 15B unified large multimodal model for pathology that handles both patch-level and whole-slide-image tasks, with a new pathology CLIP (CPath-CLIP) as its patch vision encoder.
Problem
Pathology LMMs are trained separately for patch-level and WSI-level analysis, which limits knowledge sharing across levels and produces redundant models.
Approach
A single LMM couples a CLIP-style patch encoder (CPath-CLIP, concatenating OpenAI CLIP-L and Virchow2 features, with an LLM as text encoder) and a multi-scale WSI encoder (SlideParser with CoCa-style query compression) with Qwen2.5-14B, trained in four stages mixing patch and WSI data.
Key results
- CPath-Omni scores 72.4 overall on PathMMU, exceeding PathGen-LLaVA (60.1) by 12.3 points and matching the 71.8% expert pathologist score.
- CPath-CLIP reaches 83.2% average zero-shot accuracy across nine pathology datasets, versus 79.7% for PathGen-CLIP-L.
- On WSI VQA, CPath-Omni reaches 67.3% open-ended and 70.8% closed-ended accuracy, versus 20.5% and 35.5% for GPT-4o.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CPath-CLIP instead of PathGen-CLIP-L | Zero-shot pathology image classification · LC-Lung | accuracy (%): 89.8 -> 97.1; +7.3% | Helped |
| Use CPath-CLIP instead of PathGen-CLIP-L | Zero-shot pathology image classification · PatchCamelyon | accuracy (%): 88.2 -> 95.9; +7.7% | Helped |
| Use CPath-CLIP instead of PathGen-CLIP-L | Zero-shot pathology image classification · Osteo | accuracy (%): 74.6 -> 80.7; 6.1% | Helped |
| Use CPath-CLIP instead of PathGen-CLIP-L | Zero-shot pathology image classification · CRC100K | accuracy (%): 78.0 -> 78.0 | No clear effect |
| Use CPath-CLIP instead of PathGen-CLIP-L | Zero-shot pathology image classification · SICAPv2 | accuracy (%): 63.5 -> 63.1 | Hurt |
| Use CPath-Omni instead of PathGen-LLaVA | Pathology VQA · PathMMU overall test | accuracy: 60.1 -> 72.4; 13.8% | Helped |
| Use CPath-Omni instead of GPT-4o | WSI VQA · Open-ended questions | accuracy: 20.5 -> 67.3 | Helped |
| Use CPath-Omni instead of GPT-4o | WSI VQA · Closed-ended questions | accuracy: 35.5 -> 70.8 | Helped |
| Use CPath-Omni instead of HistGen | WSI report generation · WSI report generation | BLEU4: 8.4 -> 8.7 | Helped |
| Use CPath-Omni instead of PathGen-LLaVA | Patch-level VQA · PathMMU PathCLS | accuracy: 54.2 -> 75.7; 30.1% | Helped |
Surprising findings
- CPath-Omni slightly exceeds the human pathologist accuracy on PathMMU, which the authors attribute to pathologists not being experts in every disease category.
- CPath-Omni beats GPT-4o even on out-of-distribution patch classification datasets.
Limitations
- WSI training data is limited (about 5,850 reports and 33,830 instruction samples).
- Win-rate comparisons for captioning and VPR rely on GPT-4o and 50 manually annotated images, a small evaluation set.
Experimental setup
Datasets: CPath-PatchCaption PathCap Quilt-1M OpenPath PathInstruct-200K HistGen WSI captions TCGA subtyping datasets PathMMU PatchCamelyon CRC-100K BACH SICAPv2
Benchmarks: PathMMU PathCLS LC-Lung LC-Colon CRC100K SkinCancer Pcam BACH Osteo WSSSLUAD SICAPv2 TCGA subtyping
Metrics: accuracy BLEU-1-4 ROUGE-L win rate zero-shot accuracy linear probing accuracy
Baselines: PathGen-LLaVA Quilt-LLaVA LLaVA-Med GPT-4V GPT-4o Gemini-1.5-Pro PRISM HistGen WsiCaption ABMIL DSMIL PathGen-CLIP-L
Builds on: LLaVA-NeXT CoCa CLIP Virchow2
Closest papers by meaning
- PathView-Bench: Can Multimodal Large Language Models Achieve Fine-grained Multiscale Understanding of Pathology Images? (NeurIPS 2026 · similarity 0.91)
- PathVQ: Reforming Computational Pathology Foundation Model for Whole Slide Image Analysis via Vector Quantization (NeurIPS 2025 · similarity 0.91)
- SlideChat: A Large Vision-Language Assistant for Whole-Slide Pathology Image Understanding (CVPR 2025 · similarity 0.91)
- MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding (CVPR 2026 · similarity 0.90)
- PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaboration (ICLR 2025 · similarity 0.90)
- CPathAgent: An Agent-based Foundation Model for Interpretable High-Resolution Pathology Image Analysis Mimicking Pathologists' Diagnostic Logic (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.