Research map / Vision-language and multimodal / Human and embodied modeling

PathVQ: Reforming Computational Pathology Foundation Model for Whole Slide Image Analysis via Vector Quantization

NeurIPS 2025 · Vision-language and multimodal › Human and embodied modeling · approach: Medical vision-language models · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Compresses all spatial patch tokens of a pathology tile encoder (UNI) via multi-scale vector quantization distillation and uses the resulting offline tokenizer for slide-level SSL and ABMIL/Transformer WSI fine-tuning.

Problem

Pathology foundation models for WSI analysis use only the [CLS] token of tile encoders, discarding spatial patch-token information; using all patch tokens is prohibitively expensive in storage and training.

Approach

Vector-quantize the 196 patch tokens of each tile (1024 to 16 dims, 8192 codes) with a multi-scale residual VQ whose tile-level scale doubles as a tokenizer for slide-level masked-token and ABMIL soft-target SSL, with a conv module that turns compressed features into slide-level inputs.

Key results

• On BRACS tumor classification, UNI + PathVQ + SPT with Roformer reaches F1 0.754 and AUC 0.910, versus ABMIL 0.692 F1 and 0.875 AUC.
• PathVQ with ABMIL improves TCGA survival c-index over the ABMIL reproduction, e.g. KIRC 0.691 to 0.760 and BLCA 0.540 to 0.608.
• The paper reports that VQ compresses patch tokens from 1024 to 16 dimensions, a 64x compression rate, and that UNI-2 gives only marginal gains over UNI.

Measured effects reported in the paper

Surprising findings

• Scaling to the larger UNI-2 encoder yields only marginal gains over UNI, which the authors attribute to [CLS] token information loss.
• Roformer (large transformer) overfits on survival prediction without pretraining, and PathVQ alone sometimes underperforms the ABMIL baseline on specific tasks.

Limitations

• Evaluation limited to BRACS for tumor classification; authors note binary WSI tasks are nearly saturated and did not test harder datasets.
• Many reported gains are within standard deviations across folds, and the code is promised only 'soon'.
• Only UNI-based tile encoder is used in main results, with Conch-v1.5 only in appendix.

Experimental setup

Datasets: TCGA BRACS CRC-100K Camelyon17-WILDS

Benchmarks: BRACS TCGA-BRCA TCGA-CRC TCGA-BLCA TCGA-UCEC TCGA-KIRC CRC-100K Camelyon17-WILDS

Metrics: macro-F1 macro-AUC c-index

Baselines: ABMIL DSMIL DTFD-MIL TransMIL GigaPath TITAN UNI-2 CHIEF PANTHER

Builds on: UNI VQ-VAE ABMIL iBOT LoRA

Closest papers by meaning

• MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding (CVPR 2026 · similarity 0.92)
• FOCUS: Knowledge-enhanced Adaptive Visual Compression for Few-shot Whole Slide Image Classification (CVPR 2025 · similarity 0.92)
• SlideChat: A Large Vision-Language Assistant for Whole-Slide Pathology Image Understanding (CVPR 2025 · similarity 0.92)
• PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology (AAAI 2026 · similarity 0.91)
• CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathology (CVPR 2025 · similarity 0.91)
• PathView-Bench: Can Multimodal Large Language Models Achieve Fine-grained Multiscale Understanding of Pathology Images? (NeurIPS 2026 · similarity 0.91)