Research map / Vision-language and multimodal / CLIP and zero-shot
ViLa-MIL: Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ViLa-MIL, a few-shot WSI classification framework that aligns dual-scale (low/high magnification) LLM-generated visual descriptive text prompts with CLIP image features via prototype-guided patch and context-guided text decoders.
Problem
MIL-based whole slide image classification needs many bag-level labels and generalizes poorly, while existing pathology VLMs rely on costly image-text pretraining and generic class-name text prompts that lack diagnostic detail.
Approach
Generate dual-scale visual descriptive text prompts with a frozen LLM (GPT-3.5) for low- and high-magnification WSIs, and transfer frozen CLIP to WSIs via a prototype-guided patch decoder (image side) and a context-guided text decoder (text side) trained end-to-end on few labels.
Key results
- On 16-shot TIHD-RCC, ViLa-MIL reaches 84.3 AUC / 68.7 F1 / 68.8 ACC, versus 77.1 AUC for the best prior MIL baseline (GTMIL).
- Abstract and text report 1.7-7.2% AUC and 2.1-7.3% F1 improvements over the strongest MIL methods across the three datasets.
- Cross-center TIHD->TCGA evaluation gives ViLa-MIL 83.4 AUC versus 78.2 for the best baseline (IBMIL).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ViLa-MIL instead of the strongest AUC baseline | WSI classification · TIHD-RCC (16-shot) | AUC: 77.1 ± 3.5 -> 84.3 ± 4.6 | Helped |
| use ViLa-MIL instead of the strongest AUC baseline | WSI classification · TCGA-RCC (16-shot) | AUC: 90.9 ± 4.1 -> 92.6 ± 3.0 | Helped |
| use ViLa-MIL instead of the strongest AUC baseline | WSI classification · TCGA-Lung (16-shot) | AUC: 69.2 ± 7.4 -> 74.7 ± 3.5 | Helped |
| add the prototype-guided patch decoder instead of ABMIL aggregation (ablation) | ViLa-MIL · TIHD-RCC (16-shot), low-scale prompt | AUC: 76.8 ± 3.1 -> 79.4 ± 2.1 | Helped |
| add the prototype-guided patch decoder instead of ABMIL aggregation (ablation) | ViLa-MIL · TIHD-RCC (16-shot), high-scale prompt | AUC: 79.3 ± 3.2 -> 82.9 ± 2.6 | Helped |
| add a dual-scale prompt instead of a single-scale prompt (ablation) | ViLa-MIL with the prototype-guided patch decoder · TIHD-RCC (16-shot) | AUC: 82.9 ± 2.6 -> 83.6 ± 2.7 | Helped |
| add the context-guided text decoder (ablation) | ViLa-MIL · TIHD-RCC (16-shot) | AUC: 83.6 ± 2.7 -> 84.3 ± 4.6 | Helped |
| use prototype-guided patch decoding instead of mean pooling (ablation) | ViLa-MIL · TIHD-RCC (16-shot) | AUC: 80.8 ± 4.5 -> 84.3 ± 4.6 | Helped |
| use prototype-guided patch decoding instead of attention-based pooling (ablation) | ViLa-MIL · TIHD-RCC (16-shot) | AUC: 81.1 ± 2.9 -> 84.3 ± 4.6 | Helped |
| use prototype-guided patch decoding instead of self-attention-based pooling (ablation) | ViLa-MIL · TIHD-RCC (16-shot) | AUC: 80.1 ± 3.2 -> 84.3 ± 4.6 | Helped |
| use a dual-scale visual descriptive prompt instead of a class-name-replacement prompt (ablation) | ViLa-MIL · TIHD-RCC (16-shot) | AUC: 66.4 ± 2.5 -> 84.3 ± 4.6 | Helped |
| use a dual-scale visual descriptive prompt instead of a diagnostic-guideline prompt (ablation) | ViLa-MIL · TIHD-RCC (16-shot) | AUC: 78.0 ± 2.8 -> 84.3 ± 4.6 | Helped |
Surprising findings
- Class-name-replacement text prompts perform poorly (66.4 AUC) relative to the LLM-generated visual descriptive prompt (80.5 AUC in the ablation without multi-scale).
- GPT-4 gives higher AUC (85.6) than GPT-3.5 (84.3), but the gain is small and the method is robust across different LLMs.
Limitations
- Evaluated only under a 16-shot setting on three subtyping datasets (renal and lung).
- Relies on the in-house TIHD-RCC dataset, which is not fully public, and on a ResNet-50 CLIP encoder rather than pathology-specific VLMs.
Experimental setup
Datasets: TIHD-RCC TCGA-RCC TCGA-Lung
Benchmarks: TIHD-RCC TCGA-RCC TCGA-Lung
Metrics: AUC F1 ACC
Baselines: Max-pooling Mean-pooling ABMIL CLAM-SB CLAM-MB TransMIL DSMIL GTMIL DTMIL IBMIL
Builds on: CLIP CoOp ABMIL CLAM DSMIL
Closest papers by meaning
- Queryable Prototype Multiple Instance Learning with Vision-Language Models for Incremental Whole Slide Image Classification (AAAI 2025 · similarity 0.91)
- Few-Shot Learning from Gigapixel Images via Hierarchical Vision-Language Alignment and Modeling (NeurIPS 2025 · similarity 0.90)
- Generalizable Whole Slide Image Classification with Fine-Grained Visual-Semantic Interaction (CVPR 2024 · similarity 0.89)
- SlideChat: A Large Vision-Language Assistant for Whole-Slide Pathology Image Understanding (CVPR 2025 · similarity 0.88)
- CaMIL: Causal Multiple Instance Learning for Whole Slide Image Classification (AAAI 2024 · similarity 0.88)
- PathVQ: Reforming Computational Pathology Foundation Model for Whole Slide Image Analysis via Vector Quantization (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.