Research map / Vision-language and multimodal / Multimodal LLMs

Patho-R1: A Multimodal Reinforcement Learning-Based Pathology Expert Reasoner

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds a pathology textbook-derived image-text corpus and CoT SFT data, then trains Patho-R1 (Qwen2.5-VL-3B/7B) via continued pretraining, SFT, and GRPO/DAPO RL, plus a CLIP variant Patho-CLIP that is evaluated on retrieval, zero-shot classification, and pathology VQA/MCQ benchmarks.

Problem

Pathology VLMs lack diagnostic accuracy and interpretable reasoning because existing pathology datasets are mostly short image-description pairs without structured diagnostic reasoning.

Approach

Mine pathology textbooks and expert notes into a 3.5M image-text corpus and 500k difficulty-leveled chain-of-thought SFT samples (generated by DeepSeek-R1), then apply GRPO and DAPO reinforcement learning on 10k diagnostic MCQs with format, accuracy, and length-aware rewards.

Key results

• Patho-CLIP-L reaches 76.14% mean zero-shot accuracy across five pathology classification datasets, above CONCH and MUSK (best baseline PathGen-CLIP-L at 74.65%).
• Patho-CLIP-L achieves 62.28% i2t and 60.33% t2i average recall on ARCH, versus 50.71% and 52.73% for CONCH.
• Patho-R1-7B reaches 63.37% overall on PathMMU-test, versus 58.79% for PathGen-LLaVA-13B, and 62.98% on PathMMU-val.

Surprising findings

• Providing VLMs with image-caption-CoT triplets did not outperform caption-only prompts to a strong text-only model during data generation.
• Patho-CLIP-B outperforms Patho-CLIP-L on the BMT dataset in zero-shot accuracy (49.33% vs 38.44%).

Limitations

• Performance degrades on out-of-domain modalities such as MRI and CT since training data is pathology-only.
• Continued pretraining may degrade instruction following; more advanced CPT methods were not explored due to compute cost.
• SFT data generated by DeepSeek-R1 has issues such as language mixing in pathology terminology and repetition, mitigated only by rule-based filtering and selective manual checks.

Experimental setup

Datasets: PubMed Quilt-1M PathGen-1.6M PathCap ARCH Quilt-VQA Path-VQA

Benchmarks: ARCH Archive LC-Lung LC-Colon WSSSLUAD SICAPv2 BMT PathMMU Quilt-VQA Path-VQA MedXpertQA OmniMedVQA

Metrics: Recall@K accuracy

Baselines: PLIP CONCH PathGen-CLIP MUSK PathGen-LLaVA-13B HuatuoGPT-Vision-7B Qwen2.5VL-7B InternVL3-8B

Builds on: GRPO DAPO DeepSeek-R1 CLIP Qwen2.5-VL

Closest papers by meaning

• PathView-Bench: Can Multimodal Large Language Models Achieve Fine-grained Multiscale Understanding of Pathology Images? (NeurIPS 2026 · similarity 0.90)
• Patho-AgenticRAG: Towards Multimodal Agentic Retrieval-Augmented Generation for Pathology VLMs via Reinforcement Learning (AAAI 2026 · similarity 0.90)
• PathAsst: A Generative Foundation AI Assistant towards Artificial General Intelligence of Pathology (AAAI 2024 · similarity 0.90)
• PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaboration (ICLR 2025 · similarity 0.89)
• PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology (AAAI 2026 · similarity 0.88)
• MedReasoner: Reinforcement Learning Drives Reasoning Grounding from Clinical Thought to Pixel-Level Precision (AAAI 2026 · similarity 0.88)