Research map / Vision-language and multimodal / CLIP and zero-shot
Queryable Prototype Multiple Instance Learning with Vision-Language Models for Incremental Whole Slide Image Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QPMIL-VL, the first vision-language framework for incremental WSI classification, combining a queryable prototype pool (prompt-based MIL aggregation over CONCH features) with class feature enhancement, evaluated on four TCGA datasets.
Problem
Incremental (continual) classification of whole slide images, where new datasets arrive over time and models must avoid catastrophic forgetting without storing past data in buffers.
Approach
Replace visual-only MIL and replay buffers with a pathology VLM (CONCH): a key-prompt prototype pool queried by bag max-pooled features with a frequency-based penalty to diversify prototype selection across datasets, plus class ensemble text prompts, a tunable class vector, and a class similarity loss.
Key results
- On forward-order training, QPMIL-VL reaches ACC 0.890 ± 0.021 and forgetting 0.027 ± 0.014, versus 0.839 for MI-Zero and 0.749 for DER++ (the best rehearsal baseline).
- On reverse-order training, QPMIL-VL achieves ACC 0.859 ± 0.032 and Upper-bound Ratio 0.946, outperforming all compared methods including buffer-based ones.
- Ablation: adding the QPMIL prototype module lifts ACC from 0.308 (FineTune) to 0.720; the full CFE and class similarity loss bring it to 0.890.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use QPMIL-VL instead of MI-Zero | Incremental WSI classification · TCGA WSI, forward-order class-incremental | ACC: 0.839 ± 0.034 -> 0.890 ± 0.021 | Helped |
| Use QPMIL-VL instead of MI-Zero | Incremental WSI classification · TCGA WSI, reverse-order class-incremental | ACC: 0.839 ± 0.034 -> 0.859 ± 0.032 | Helped |
| Add QPMIL to FineTune (ablation) | QPMIL-VL · TCGA WSI, forward-order class-incremental | ACC: 0.308 ± 0.045 -> 0.720 ± 0.029 | Helped |
| Add tunable vector (ablation) | QPMIL · TCGA WSI, forward-order class-incremental | ACC: 0.720 ± 0.029 -> 0.747 ± 0.039 | Helped |
| Add class ensemble (ablation) | QPMIL + tunable vector · TCGA WSI, forward-order class-incremental | ACC: 0.747 ± 0.039 -> 0.874 ± 0.033 | Helped |
| Add class similarity loss (ablation) | QPMIL + tunable vector + class ensemble · TCGA WSI, forward-order class-incremental | ACC: 0.874 ± 0.033 -> 0.890 ± 0.021 | Helped |
| Add prototype keys (ablation) | QPMIL-VL · TCGA WSI, forward-order class-incremental | ACC: 0.675 ± 0.029 -> 0.890 ± 0.021 | Helped |
| Add matching penalty (ablation) | QPMIL-VL · TCGA WSI, forward-order class-incremental | ACC: 0.659 ± 0.029 -> 0.890 ± 0.021 | Helped |
| Use MaxPooling instead of MeanPooling for query vector generation (ablation) | QPMIL-VL · TCGA WSI, forward-order class-incremental | ACC: 0.877 ± 0.034 -> 0.890 ± 0.021 | Helped |
| Use MaxPooling instead of MeanPooling for query vector generation (ablation) | QPMIL-VL · TCGA WSI, reverse-order class-incremental | ACC: 0.859 ± 0.042 -> 0.859 ± 0.032 | No clear effect |
| Train in reverse-order rather than forward-order | QPMIL-VL · TCGA WSI, class-incremental | ACC: 0.890 ± 0.021 -> 0.859 ± 0.032 | Hurt |
| Apply class feature enhancement (ablation) | QPMIL-VL class features · ESCA-ESCC | cosine similarity: 0.078 -> 0.447 | Helped |
Surprising findings
- Regularization-based methods (EWC, LwF) are nearly ineffective for WSI class-incremental learning, with ACC close to the fine-tuning lower bound.
- Reverse-order training yields lower performance for most methods, which the authors attribute to the larger NSCLC dataset interfering with earlier-learned ESCA.
Limitations
- Results are reported on only four TCGA datasets with two classes added per dataset, limiting generality.
- Performance depends on training order; reverse-order training drops ACC noticeably, and forgetting still occurs (forgetting 0.027 on forward order, larger on reverse).
Experimental setup
Datasets: TCGA NSCLC TCGA BRCA TCGA RCC TCGA ESCA
Benchmarks: TCGA NSCLC/BRCA/RCC/ESCA incremental WSI benchmark
Metrics: ACC Upper-bound Ratio Forgetting BWT Masked ACC
Baselines: FineTune JointTrain EWC LwF A-GEM ER DER++ ER-ACE ConSlide AttriCLIP MI-Zero
Builds on: L2P CONCH ABMIL TaskRes ConSlide
Closest papers by meaning
- ViLa-MIL: Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classification (CVPR 2024 · similarity 0.91)
- CaMIL: Causal Multiple Instance Learning for Whole Slide Image Classification (AAAI 2024 · similarity 0.90)
- Dynamic Policy-Driven Adaptive Multi-Instance Learning for Whole Slide Image Classification (CVPR 2024 · similarity 0.89)
- Boosting Multiple Instance Learning Models for Whole Slide Image Classification: A Model-Agnostic Framework Based on Counterfactual Inference (AAAI 2024 · similarity 0.89)
- Generalizable Whole Slide Image Classification with Fine-Grained Visual-Semantic Interaction (CVPR 2024 · similarity 0.88)
- Transformer-Based Video-Structure Multi-Instance Learning for Whole Slide Image Classification (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.