Research map / Vision-language and multimodal / Human and embodied modeling
Prompting Vision Foundation Models for Pathology Image Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QAP, a visual prompting method for liver pathology images that generates task-specific prompts from K-function spatial and histogram morphological attributes via an attribute-conditioned prompt generator on a frozen ViT-S/16.
Problem
Adapting vision foundation models to small-scale liver pathology image analysis for NAFLD diagnosis, where existing task-agnostic visual prompts designed for natural images give insufficient pathology-specific cues.
Approach
Derives quantitative spatial attributes (K-function over nuclei and white regions) and morphological attributes (histograms of area, eccentricity, perimeter) from a segmenter's output, and uses a transformer-decoder prompt generator conditioned on these attributes to produce instance-specific visual prompts for a ViT.
Key results
- QAP reaches 97.75±0.77 F1 on NAFLD abnormality classification with 10% training data, versus 96.58±0.60 for VPT-DEEP.
- QAP attains 83.37 macro-average F1 on histological findings recognition, above VQT's 81.17.
- On histological scoring generalization, QAP achieves Spearman rho 0.793 for steatosis, above VQT's 0.766.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use QAP quantitative-attribute prompts instead of VPT-DEEP | ViT-S/16 for NAFLD abnormalities classification · NAFLD abnormalities, 10% training | F1 score: 96.58±0.60 -> 97.75±0.77 | Helped |
| Use QAP quantitative-attribute prompts instead of VPT-DEEP | ViT-S/16 for NAFLD abnormalities classification · NAFLD abnormalities, 50% training | F1 score: 99.30±0.23 -> 99.37±0.23 | Helped |
| Use QAP quantitative-attribute prompts instead of SAR | ViT-S/16 for NAFLD abnormalities classification · NAFLD abnormalities, 100% training | F1 score: 99.54±0.19 -> 99.58±0.17 | Helped |
| Use QAP quantitative-attribute prompts instead of strongest baseline | ViT-S/16 for histological findings recognition · Liver-NAS Macro-Average | F1 score: 81.17±7.99 -> 83.37±6.99 | Helped |
| Use QAP quantitative-attribute prompts instead of VQT | ViT-S/16 for histological findings recognition · Liver-NAS Steatosis | F1 score: 98.90±0.41 -> 98.79±0.25 | Hurt |
| Use QAP quantitative-attribute prompts instead of VPT-DEEP | ViT-S/16 for histological scoring · Liver-NAS Inflammation | Spearman correlation coefficient ρ: 0.422±0.022 -> 0.410 ±0.008 | Hurt |
| Use QAP quantitative-attribute prompts instead of VQT | ViT-S/16 for histological scoring · Liver-NAS Ballooning | Spearman correlation coefficient ρ: 0.316±0.012 -> 0.332±0.018 | Helped |
| Use QAP quantitative-attribute prompts instead of VQT | ViT-S/16 for histological scoring · Liver-NAS Steatosis | Spearman correlation coefficient ρ: 0.766±0.006 -> 0.793±0.008 | Helped |
| Add quantitative attributes to prompts instead of using random initial prompt embeddings (ablation) | QAP for histological findings recognition · Liver-NAS | F1 score | Helped |
| Condition prompts on explicit quantitative attributes rather than task-agnostic visual prompts (ablation) | QAP for histological findings recognition · Liver-NAS | F1 score | Helped |
| Add spatial and morphological quantitative attributes to visual prompts (ablation) | QAP for histological findings recognition · Liver-NAS | F1 score | Helped |
Surprising findings
- Fabian et al. and SAR fall to about 80.7 F1 with only 10% training data, while prompting methods stay near 96 F1.
Limitations
- Gains over VPT-DEEP and VQT are small on several tasks and differences are within reported standard deviations.
- Relies on a tissue segmenter whose quality affects the attributes, and the attributes are limited to nuclei and white regions.
- Evaluated only on liver H&E images from a mouse dataset and a single human cohort.
Experimental setup
Datasets: NAFLD-Anomaly Liver-NAS TCGA-BRCA
Benchmarks: NAFLD-Anomaly Liver-NAS
Metrics: F1 score Spearman correlation coefficient p-value
Baselines: Fabian et al. SAR VPT-DEEP VQT
Builds on: Visual Prompt Tuning VQT Transformer decoder
Closest papers by meaning
- Memory-Efficient Prompt Tuning for Incremental Histopathology Classification (AAAI 2024 · similarity 0.84)
- iDPA: Instance Decoupled Prompt Attention for Incremental Medical Object Detection (ICML 2025 · similarity 0.83)
- Prototype-Based Image Prompting for Weakly Supervised Histopathological Image Segmentation (CVPR 2025 · similarity 0.83)
- Free Lunch in Pathology Foundation Model: Task-specific Model Adaptation with Concept-Guided Feature Enhancement (NeurIPS 2024 · similarity 0.83)
- CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathology (CVPR 2025 · similarity 0.83)
- CAT: Coordinating Anatomical-Textual Prompts for Multi-Organ and Tumor Segmentation (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.