Research map / Vision-language and multimodal / CLIP and zero-shot
Few-Shot Learning from Gigapixel Images via Hierarchical Vision-Language Alignment and Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HiVE-MIL, a hierarchical heterogeneous graph over 5x/20x visual and textual nodes with text-guided filtering and hierarchical text contrastive loss, for few-shot weakly supervised WSI classification with pathology VLMs.
Problem
Few-shot weakly supervised WSI classification with vision-language MIL models fails to model hierarchical interactions across scales within each modality and to align visual and textual features on the same scale.
Approach
Builds a unified hierarchical heterogeneous graph with parent-child edges between coarse (5x) and fine (20x) visual and textual nodes, intra-scale edges filtered by a two-stage text-guided dynamic thresholding, a modality-scale attention aggregator, and a hierarchical text contrastive loss.
Key results
- Under 16-shot with CONCH, HiVE-MIL reports 90.39% ACC / 96.49% AUC / 90.37% macro F1 on NSCLC, improving over the second-best method by +1.99 / +1.13 / +2.00.
- Gains of up to +4.37% ACC, +4.22% AUC, and +4.14% macro F1 on BRCA (QuiltNet setting), per the paper's reported results.
- Ablation reports a 2.04% macro F1 drop on BRCA when HTCL is removed.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HiVE-MIL instead of the strongest baseline with PLIP | 16-shot WSI classification · TCGA NSCLC | Macro F1: +3.26% | Helped |
| Use HiVE-MIL instead of the strongest baseline with PLIP | 16-shot WSI classification · TCGA BRCA | Macro F1: +2.41% | Helped |
| Use HiVE-MIL instead of the strongest baseline with PLIP | 16-shot WSI classification · TCGA RCC | Macro F1: +1.64% | Helped |
| Use HiVE-MIL instead of the strongest baseline with QuiltNet | 16-shot WSI classification · TCGA NSCLC | Macro F1: +2.96% | Helped |
| Use HiVE-MIL instead of the strongest baseline with QuiltNet | 16-shot WSI classification · TCGA BRCA | Macro F1: +4.14% | Helped |
| Use HiVE-MIL instead of the strongest baseline with QuiltNet | 16-shot WSI classification · TCGA RCC | Macro F1: +0.75% | Helped |
| Use HiVE-MIL instead of the strongest baseline with CONCH | 16-shot WSI classification · TCGA NSCLC | Macro F1: +2.00% | Helped |
| Use HiVE-MIL instead of the strongest baseline with CONCH | 16-shot WSI classification · TCGA BRCA | Macro F1: +3.76% | Helped |
| Use HiVE-MIL instead of the strongest baseline with CONCH | 16-shot WSI classification · TCGA RCC | Macro F1: +0.43% | Helped |
| Add HTCL to the model (ablation) | HiVE-MIL · TCGA BRCA | Macro F1: 2.04% decrease | Helped |
| Add HHG hierarchical message passing (ablation) | HiVE-MIL without HTCL (c) · TCGA RCC | Macro F1: 1.64% drop | Helped |
| Add TGDF, HHG, and HTCL (ablation) | HiVE-MIL without these modules · All reported datasets | accuracy and Macro F1 | Helped |
Limitations
- TGDF uses non-learnable, per-WSI similarity thresholds which may limit generalization across datasets or backbones.
- Hit Ratio metric assumes correctness of LLM-generated parent-child text hierarchies, which have not been validated by pathologists.
- LLM-generated descriptions were not verified by pathologists.
Experimental setup
Datasets: TCGA-NSCLC TCGA-BRCA TCGA-RCC
Benchmarks: TCGA-NSCLC TCGA-BRCA TCGA-RCC
Metrics: ACC AUC macro F1 Hit Ratio@2
Baselines: Max Pooling Mean Pooling ABMIL DSMIL CLAM-SB CLAM-MB TransMIL DTFD-MIL WiKG ViLa-MIL MSCPT FOCUS
Builds on: CoOp GraphSAGE CLIP ViLa-MIL MSCPT FOCUS
Closest papers by meaning
- ViLa-MIL: Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classification (CVPR 2024 · similarity 0.90)
- Generalizable Whole Slide Image Classification with Fine-Grained Visual-Semantic Interaction (CVPR 2024 · similarity 0.89)
- FOCUS: Knowledge-enhanced Adaptive Visual Compression for Few-shot Whole Slide Image Classification (CVPR 2025 · similarity 0.88)
- SlideChat: A Large Vision-Language Assistant for Whole-Slide Pathology Image Understanding (CVPR 2025 · similarity 0.88)
- MAPLE: Multi-scale Attribute-enhanced Prompt Learning for Few-shot Whole Slide Image Classification (NeurIPS 2025 · similarity 0.88)
- MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.