Research map / Vision-language and multimodal / Multimodal LLMs
PathAsst: A Generative Foundation AI Assistant towards Artificial General Intelligence of Pathology
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds PathCap/PathInstruct pathology datasets, a pathology CLIP (PathCLIP), and PathAsst, a Vicuna-13B-based multimodal assistant that can invoke eight pathology sub-models and a PubMed paper retrieval tool.
Problem
General MLLMs focus on natural images, leaving pathology without high-quality image-text data and pathology-specific multimodal models that can interpret slides and support diagnosis.
Approach
Combine a pathology-dedicated CLIP trained on 207K curated pathology image-caption pairs with Vicuna-13B, instruction-tune on ChatGPT-generated pathology instructions plus tool-invocation samples, and let the assistant call pathology CV sub-models and a 5.3M-abstract paper retrieval database.
Key results
- PathAsst (w/ PathCLIP) reaches 90.9% closed-ended and 38.4% open-ended accuracy/F1 on PathVQA, versus LLaVA's 81.0 and 19.2.
- PathCLIP improves zero-shot F1 over PLIP by 1.1%, 11.6%, 2.7%, and 7.3% on CRC100K, WSSS4LUAD, LC-lung, and LC-colon.
- PathCLIP achieves 10.71x and 11.07x the R@10 retrieval on PubMed compared to OpenAI CLIP and PLIP, respectively.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace OpenAI CLIP with PathCLIP as PathAsst vision encoder (ablation) | PathAsst (Vicuna-13B) · PathVQA | Closed accuracy: +1.2 | Helped |
| Replace OpenAI CLIP with PathCLIP as PathAsst vision encoder (ablation) | PathAsst (Vicuna-13B) · PathVQA | Open F1: +0.8 | Helped |
| Instruction tuning on PathInstruct (PathAsst vs LLaVA) | PathAsst · PathVQA | Closed accuracy: +8.7 over LLaVA | Helped |
| Instruction tuning on PathInstruct (PathAsst vs LLaVA) | PathAsst · PathVQA | Open F1: +18.4 over LLaVA | Helped |
| Domain-specific contrastive fine-tuning of CLIP on PathCap (PathCLIP vs OpenAI CLIP) | CLIP zero-shot classification · CRC100K / WSSS4LUAD / LC-lung / LC-colon | zero-shot F1: +32%, +19.5%, +57.2%, +18.6% | Helped |
| Domain-specific CLIP (PathCLIP vs PLIP) | CLIP zero-shot classification · CRC100K / WSSS4LUAD / LC-lung / LC-colon | zero-shot F1: +1.1, +11.6, +2.7, +7.3 | Helped |
| Domain-specific CLIP (PathCLIP vs OpenAI CLIP) on cross-modal retrieval | CLIP retrieval · PubMed and books test sets | R@10: 10.71-fold (PubMed vs CLIP); 5.78x (books vs CLIP) | Helped |
| Adding PathAsst pathology sub-model invocation and paper retrieval tools | PathAsst · PathVQA | Closed accuracy / Open F1 | No clear effect |
| PathAsst vs prior SOTA on open-ended PathVQA questions | PathAsst · PathVQA | Open F1: CLIP-ViT w/ GPT2 40.0 vs PathAsst 38.4 | Hurt |
Surprising findings
- PathAsst slightly underperforms the prior SOTA on open-ended PathVQA questions despite gains on closed-ended questions.
- LLaVA fails to recognize some pathology images as pathological, while MiniGPT-4 gives simplistic descriptions.
Limitations
- Open-ended VQA performance trails the prior SOTA model.
- Training data partly from PubMed with limited pathology share and lower image clarity; relies on ChatGPT-generated instruction data.
Experimental setup
Datasets: PathCap PathInstruct PubMed image-text pairs CRC100K WSSS4LUAD LC25000 PathVQA PubMed abstracts (5.3M)
Benchmarks: PathVQA CRC100K WSSS4LUAD LC25000 PubMed retrieval test set
Metrics: R@k F1 accuracy
Baselines: OpenAI CLIP PLIP LLaVA MiniGPT-4 BLIP-2 Flan-T5 XXL M2I2 CLIP-ViT w/ GPT2 MMQ
Builds on: LLaVA MiniGPT-4 OpenCLIP Vicuna PLIP
Closest papers by meaning
- PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaboration (ICLR 2025 · similarity 0.91)
- Patho-R1: A Multimodal Reinforcement Learning-Based Pathology Expert Reasoner (AAAI 2026 · similarity 0.90)
- PathView-Bench: Can Multimodal Large Language Models Achieve Fine-grained Multiscale Understanding of Pathology Images? (NeurIPS 2026 · similarity 0.89)
- PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology (AAAI 2026 · similarity 0.88)
- CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathology (CVPR 2025 · similarity 0.87)
- CPathAgent: An Agent-based Foundation Model for Interpretable High-Resolution Pathology Image Analysis Mimicking Pathologists' Diagnostic Logic (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.