Research map / Label-efficient and robust learning / Recognition and long-tail
Single GPU Task Adaptation of Pathology Foundation Models for Whole Slide Image Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TAPFM, a single-GPU method that uses ViT CLS attention for MIL aggregation and fine-tunes pathology foundation models with detached feature and attention gradient losses, evaluated on BLCA FGFR3 and LUAD mutation prediction.
Problem
Adapting large pathology foundation models (PFMs) end-to-end for WSI-level weakly supervised tasks (mutation prediction) is memory-prohibitive on a single GPU, and jointly training PFMs with MIL aggregators is unstable.
Approach
Use the ViT CLS-token attention as the MIL aggregation weights, keep separate computation graphs for the aggregator and the PFM by detaching features and attention, and update the PFM with a Task Adaptation Loss built from detached feature gradients and attention gradients.
Key results
- H-Optimus-0 (TAPFM) achieves AUC 0.8647 on institutional BLCA FGFR3 and 0.9021 on TCGA-BLCA FGFR3, the best among all reported models.
- H-Optimus-0 (TAPFM) reaches AUC 0.8491 on institutional LUAD EGFR and 0.8553 on TCGA-LUAD EGFR.
- On multi-label LUAD mutation prediction, H-Optimus-0 (TAPFM) reports macro-average AUC 0.8510 (institutional) and 0.8439 (TCGA).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TAPFM instead of the strongest reported fine-tuned MIL baseline | H-Optimus-0 on BLCA FGFR3 institutional cohort · BLCA FGFR3 institutional cohort | AUC: 0.8526 -> 0.8647 | Helped |
| Use TAPFM instead of the strongest reported fine-tuned MIL baseline | H-Optimus-0 on BLCA FGFR3 TCGA · BLCA FGFR3 TCGA | AUC: 0.8873 -> 0.9021 | Helped |
| Use TAPFM instead of the strongest reported fine-tuned MIL baseline | H-Optimus-0 on LUAD EGFR institutional cohort · LUAD EGFR institutional cohort | AUC: 0.8189 -> 0.8491 | Helped |
| Use TAPFM instead of the strongest reported fine-tuned MIL baseline | H-Optimus-0 on LUAD EGFR TCGA · LUAD EGFR TCGA | AUC: 0.8543 -> 0.8553 | Helped |
| Use H-Optimus-0 instead of GigaPath with TAPFM | LUAD multi-label institutional cohort · LUAD multi-label institutional cohort | Macro-average AUC: 0.8439 -> 0.8510 | Helped |
| Use H-Optimus-0 instead of GigaPath with TAPFM | LUAD multi-label TCGA cohort · LUAD multi-label TCGA cohort | Macro-average AUC: 0.8392 -> 0.8439 | Helped |
| Increase tile sampling from 25 to 50 tiles per WSI per epoch (ablation) | H-Optimus-0 (TAPFM) on BLCA FGFR3 · BLCA FGFR3 validation | validation AUC: 0.8527 -> 0.8764 | Helped |
| Increase tile sampling from 50 to 75 tiles per WSI per epoch (ablation) | H-Optimus-0 (TAPFM) on BLCA FGFR3 · BLCA FGFR3 validation | validation AUC: 0.8764 -> 0.8960 | Helped |
| Increase attention-loss weight from λ = 0.25 to λ = 1.0 (ablation) | H-Optimus-0 (TAPFM) on BLCA FGFR3 · BLCA FGFR3 training | training loss convergence rate b: -0.0701 -> -0.0810 | Helped |
| Add cosine regularization to the feature-alignment loss (ablation) | H-Optimus-0 (TAPFM) on BLCA FGFR3 · BLCA FGFR3 prediction | performance | No clear effect |
Surprising findings
- The paper reports that a cosine regularization term for feature alignment produced no significant change in FGFR3 performance.
- Fine-tuning with standard MIL aggregators (FT) often underperforms fixed-PFM variants for UNI (e.g., UNI + ABMIL FT 0.8193 vs fixed 0.7904 is higher, but UNI + ABMIL FT on TCGA LUAD is lower than TAPFM), and TAPFM's gains are reported as larger than end-to-end fine-tuning.
Limitations
- Number of tiles per WSI per epoch is limited (75-300) by single-GPU memory, so only a subset of tiles is used in training.
- Validation limited to two cancer types and TCGA external cohorts; the authors note need for multi-institutional validation.
- Layer-selective updating and distributed multi-GPU scaling are not explored.
Experimental setup
Datasets: institutional BLCA cohort institutional LUAD cohort TCGA-BLCA TCGA-LUAD
Benchmarks: TCGA-BLCA FGFR3 TCGA-LUAD EGFR institutional BLCA FGFR3 institutional LUAD EGFR LUAD multi-label EGFR/KRAS/MET/ALK
Metrics: AUC macro-average AUC
Baselines: UNI + ABMIL UNI + CLAM UNI + DSMIL UNI + VarMIL GigaPath + ABMIL H-Optimus-0 + ABMIL fine-tuned PFM with MIL aggregators
Builds on: ABMIL CLAM DSMIL VarMIL AdamW
Closest papers by meaning
- Free Lunch in Pathology Foundation Model: Task-specific Model Adaptation with Concept-Guided Feature Enhancement (NeurIPS 2024 · similarity 0.87)
- PathVQ: Reforming Computational Pathology Foundation Model for Whole Slide Image Analysis via Vector Quantization (NeurIPS 2025 · similarity 0.85)
- Do Multiple Instance Learning Models Transfer? (ICML 2025 · similarity 0.85)
- Unsupervised Foundation Model-Agnostic Slide-Level Representation Learning (CVPR 2025 · similarity 0.85)
- Dynamic Policy-Driven Adaptive Multi-Instance Learning for Whole Slide Image Classification (CVPR 2024 · similarity 0.85)
- Fast and Accurate Gigapixel Pathological Image Classification with Hierarchical Distillation Multi-Instance Learning (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.