Research map / Vision-language and multimodal / Multimodal LLMs
SlideChat: A Large Vision-Language Assistant for Whole-Slide Pathology Image Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SlideChat, a LLaVA-style vision-language assistant for gigapixel whole-slide pathology images, trained on the new SlideInstruction dataset and evaluated on the new SlideBench benchmark.
Problem
Existing pathology MLLMs operate on patches or ROIs and lack whole-slide context, and public WSI instruction datasets are scarce.
Approach
A slide-level encoder (LongNet with sparse attention) sits on top of frozen patch-level CONCH features, feeding a projector into an LLM (Qwen2.5-7B-Instruct), trained in two stages (caption alignment, then VQA instruction tuning) on GPT-4-generated WSI instruction data from TCGA reports.
Key results
- SlideChat reaches 81.17% overall accuracy on SlideBench-VQA (TCGA), 13.47 points above the second-best model.
- SlideChat reaches 54.14% on SlideBench-VQA (BCNB) in a zero-shot setting, and 60.18% on WSI-VQA*.
- On captioning, SlideChat's GPT score of 4.11 is close to MI-Gen's 4.14, with BLEU-1 of 0.37 for both.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SlideChat's whole-slide model instead of the strongest reported caption baseline | SlideChat · SlideBench-Caption | GPT score: 4.14 -> 4.11 | No clear effect |
| Use SlideChat's whole-slide model instead of MedDr with patch input | SlideChat · SlideBench-VQA (TCGA) | accuracy (Overall): 67.70 -> 81.17 | Helped |
| Use SlideChat's whole-slide model instead of Quilt-LLaVA with patch input | SlideChat · SlideBench-VQA (BCNB) | accuracy: 32.19 -> 54.14 | Helped |
| Use SlideChat's whole-slide model instead of MedDr with patch input | SlideChat · WSI-VQA* | accuracy: 54.36 -> 60.18 | Helped |
| Remove the slide-level encoder (ablation) | SlideChat · SlideBench-VQA (BCNB) | accuracy: 54.14 -> 45.49 | Hurt |
| Remove the slide-level encoder (ablation) | SlideChat · SlideBench-VQA (TCGA) | accuracy: 81.17 -> 81.21 | No clear effect |
| Replace Qwen2.5-7B-Instruct with Qwen2.5-14B-Instruct (ablation) | SlideChat · SlideBench-VQA (TCGA) | accuracy: +2.57% | Helped |
| Replace Qwen2.5-7B-Instruct with Qwen2.5-3B-Instruct (ablation) | SlideChat · SlideBench-VQA (TCGA) | accuracy: 81.17 -> 80.32 | Hurt |
Surprising findings
- Removing the slide-level encoder left TCGA and WSI-VQA performance roughly unchanged but dropped BCNB overall accuracy by over 10 points, indicating the encoder matters mainly for cross-domain generalization.
- GPT-4o and LLaVA-Med score zero on SlideBench-VQA (BCNB) with slide thumbnails.
Limitations
- Patch-level MLLM baselines had to be adapted by sampling 30 random patches with majority voting, since existing MLLMs cannot ingest gigapixel slides.
- Benchmark VQA answers were filtered using LLMs and pathologist review, and the training data are GPT-4-generated from TCGA reports.
Experimental setup
Datasets: TCGA BCNB WSI-VQA
Benchmarks: SlideBench-Caption SlideBench-VQA (TCGA) SlideBench-VQA (BCNB) WSI-VQA
Metrics: BLEU ROUGE-L GPT score accuracy
Baselines: GPT-4o MedDr LLaVA-Med Quilt-LLaVA MI-Gen GPT-4
Builds on: LLaVA LongNet CONCH PathChat Quilt-LLaVA
Closest papers by meaning
- MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding (CVPR 2026 · similarity 0.93)
- PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology (AAAI 2026 · similarity 0.92)
- PathVQ: Reforming Computational Pathology Foundation Model for Whole Slide Image Analysis via Vector Quantization (NeurIPS 2025 · similarity 0.92)
- CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathology (CVPR 2025 · similarity 0.91)
- PathView-Bench: Can Multimodal Large Language Models Achieve Fine-grained Multiscale Understanding of Pathology Images? (NeurIPS 2026 · similarity 0.90)
- PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaboration (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.