Research map / Vision-language and multimodal / CLIP and zero-shot
Generalizable Whole Slide Image Classification with Fine-Grained Visual-Semantic Interaction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FiVE, a WSI classification framework that trains a vision-language model on fine-grained, GPT-4-standardized pathology report descriptions using a TFS module with learnable diagnosis prompts and patch sampling.
Problem
WSI classification with vision-language models relies on coarse category-level text labels, which miss fine-grained pathological details and limit generalization; processing high-resolution WSIs is also computationally expensive.
Approach
Extract fine-grained diagnostic answers from non-standardized TCGA pathology reports via GPT-4 with expert-designed queries, recombine sampled answers into varied-granularity text labels, and align bag-level features (aggregated by a Task-specific Fine-grained Semantics module with learnable diagnosis prompts) with these labels; also sample a subset of patches during training.
Key results
- On TCGA Lung Cancer, FiVE reaches 94.62% ACC, 96.33% AUC and 93.89% F1 in full supervision, versus 89.93% ACC for the best baseline MHIM-MIL.
- In 16-shot transfer on TCGA Lung Cancer, FiVE achieves 91.25% accuracy, 9.19 points above the prior SOTA TOP.
- Zero-shot LUAD histological subtype classification reaches 65.23% top-1 and 95.18% top-5 accuracy when pre-trained on TCGA-LUAD labels.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FiVE instead of MHIM-MIL for slide classification | WSI classification · Camelyon16 | ACC: 92.48±0.35 -> 94.25±0.33 | Helped |
| Use FiVE instead of MHIM-MIL for slide classification | WSI classification · TCGA Lung Cancer | ACC: 89.93±3.37 -> 94.62±2.13 | Helped |
| Use FiVE instead of TOP | Few-shot WSI classification · TCGA Lung Cancer 1-shot | accuracy: 71.01 -> 83.91 | Helped |
| Use FiVE instead of TOP | Few-shot WSI classification · TCGA Lung Cancer 16-shot | accuracy: 82.06 -> 91.25 | Helped |
| Pre-train FiVE on TCGA-LUAD fine-grained labels rather than use attention-pooling | Zero-shot subtype classification · TCGA-LUAD histological subtypes | Top-1 accuracy: 58.36 -> 65.23 | Helped |
| Add fine-grained labels (ablation) | FiVE · TCGA Lung Cancer | ACC: 89.77 -> 91.21 | Helped |
| Add fine-grained guidance (ablation) | FiVE with fine-grained labels · TCGA Lung Cancer | ACC: 91.21 -> 93.56 | Helped |
| Add learnable diagnosis prompts (ablation) | FiVE with fine-grained guidance · TCGA Lung Cancer | ACC: 93.56 -> 94.62 | Helped |
| Sample a subset of visual instances during training (ablation) | FiVE · WSI classification | accuracy | No clear effect |
Limitations
- Relies on a frozen or lightly tuned ResNet image encoder, which constrains performance upper bound.
- Evaluation of zero-shot subtype classification limited to LUAD subtype labels from a single database.
- Reports are sourced from TCGA only, so pre-training data is narrow in institution and format.
Experimental setup
Datasets: TCGA Lung Cancer Camelyon16 TCGA pathology reports cBioPortal LUAD histological subtype labels
Benchmarks: TCGA Lung Cancer Camelyon16 TCGA-LUAD histological subtypes
Metrics: ACC AUC F1-score top-1 accuracy top-5 accuracy
Baselines: ABMIL DSMIL CLAM-SB CLAM-MB TransMIL DTFD-MIL MHIM-MIL CoOp TOP Max-pooling Mean-pooling Attention-pooling
Builds on: CLIP CoOp LLaMA-Adapter FLIP MHIM-MIL
Closest papers by meaning
- PathVQ: Reforming Computational Pathology Foundation Model for Whole Slide Image Analysis via Vector Quantization (NeurIPS 2025 · similarity 0.90)
- ViLa-MIL: Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classification (CVPR 2024 · similarity 0.89)
- PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology (AAAI 2026 · similarity 0.89)
- Few-Shot Learning from Gigapixel Images via Hierarchical Vision-Language Alignment and Modeling (NeurIPS 2025 · similarity 0.89)
- MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding (CVPR 2026 · similarity 0.88)
- CaMIL: Causal Multiple Instance Learning for Whole Slide Image Classification (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.