Research map / Vision-language and multimodal / Multimodal LLMs

PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PathFLIP, a fine-grained language-image pretraining framework for whole slide images that aligns region embeddings from a Region Q-Former with LLM-decomposed subcaptions, combined with slide-level contrastive alignment and LLM instruction tuning.

Problem

Existing pathology vision-language models align WSIs with slide-level captions globally and fail to capture fine-grained correspondences between text descriptions and the thousands of patches/regions in gigapixel slides.

Approach

Randomly sample subsets of sentences from slide captions as subcaptions, use them as text queries in cross-attention over region embeddings (text-conditioned region alignment with in-batch negatives), and combine with a global slide-caption contrastive loss; region and slide embeddings are fed to a Qwen3-0.6B LLM via LoRA for instruction following.

Key results

• Zero-shot CPTAC gene mutation prediction reaches average AUC 0.6634, versus 0.6424 for CPath-Omni and 0.6364 for PathAlign.
• SlideBench VQA average accuracy 0.7761 vs 0.7483 for SlideChat; on BCNB VQA 0.5865 vs 0.5414.
• Few-shot CPTAC (ratio 0.3) average AUC 0.6899, the best among compared models.

Limitations

• No quantitative evaluation of region-level visual grounding due to lack of expert-annotated pathology region-text datasets.
• Pretraining data limited to about 4.9K WSI-caption pairs from TCGA.
• Several comparisons use only 30 randomly sampled patches for LLM-based baselines, which may not reflect their full-slide capability.

Experimental setup

Datasets: SlideInstruct SlideBench CPTAC Quilt-1M BCNB

Benchmarks: CPTAC SlideBench Quilt BCNB

Metrics: AUC Recall@K BLEU ROUGE-L accuracy Qwen embedding similarity

Baselines: MI-Zero PLIP CONCH Prov-GigaPath PathGen PathAlign LLaVA-Med MedDr Quilt-LLaVA PathGen-LLaVA CPath-Omni SlideChat

Builds on: CLIP BLIP-2 CONCH LLaVA

Closest papers by meaning

• PathView-Bench: Can Multimodal Large Language Models Achieve Fine-grained Multiscale Understanding of Pathology Images? (NeurIPS 2026 · similarity 0.92)
• SlideChat: A Large Vision-Language Assistant for Whole-Slide Pathology Image Understanding (CVPR 2025 · similarity 0.92)
• MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding (CVPR 2026 · similarity 0.92)
• PathVQ: Reforming Computational Pathology Foundation Model for Whole Slide Image Analysis via Vector Quantization (NeurIPS 2025 · similarity 0.91)
• PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaboration (ICLR 2025 · similarity 0.90)
• CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathology (CVPR 2025 · similarity 0.90)