Research map / Vision-language and multimodal / Multimodal LLMs

Patho-AgenticRAG: Towards Multimodal Agentic Retrieval-Augmented Generation for Pathology VLMs via Reinforcement Learning

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Patho-AgenticRAG, a multimodal agentic RAG framework over a pathology-textbook page-image database, with a GRPO-trained router that decides retrieval, query rewriting, and tissue classifier use for pathology VLM diagnosis.

Problem

Pathology VLMs hallucinate and text-only RAG ignores diagnostic visual cues; existing medical RAG lacks fine-grained tissue-system organization and robust agentic retrieval planning.

Approach

Joint text-image retrieval over page-level embeddings of pathology textbooks (ColQwen2 + Milvus/HNSW), a fusion re-ranking score combining text and image token similarity statistics, and a Qwen3-4B router trained with hierarchical-reward GRPO to choose whether and how to invoke RAG and tissue classifiers.

Key results

• On PathMMU-test-tiny, Patho-AgenticRAG reaches 79.33 to 81.73 range per subset vs Patho-R1-7B, with PathCLS 57.22 vs 44.63 (per-subset gains are reported in Table 2)
• On MedXpertQA pathology subset, Patho-AgenticRAG scores 60.00% vs 22.00% for Patho-R1-7B; on OmniMedVQA Bright 90.11% vs 70.79%
• Retrieval fusion method achieves Rec@1 0.720 and NDCG@5 0.824 on 50 test pairs, versus 0.640 and 0.804 for CoPaLi text-only

Surprising findings

• Image-only retrieval (CoPaLi Image, WeiMoCIR) performs far worse than text-only retrieval on pathology queries (Rec@1 0.060 vs 0.640)
• Large-scale SFT before GRPO hurt generalization, with a small 400-sample cold start performing best

Limitations

• Retrieval fusion evaluated on only 100 expert-curated image-question pairs split 50/50
• Router training data derived from Quilt-VQA and Path-VQA with pseudo ground-truth paths distilled from QwenMax, and the PathMMU and MedXpertQA subsets are small or filtered

Experimental setup

Datasets: Quilt-VQA Path-VQA PathMMU MedXpertQA OmniMedVQA pathology textbooks (600+ books, 200k+ pages)

Benchmarks: PathMMU-test PathMMU-test-tiny Quilt-VQA Path-VQA MedXpertQA (pathology subset) OmniMedVQA BRIGHT

Metrics: accuracy Recall@k MRR@k NDCG@k

Baselines: CoPaLi (Text) CoPaLi (Image) WeiMoCIR InternVL2-8B InternVL2.5-8B InternVL3-8B Qwen2.5VL-7B Patho-R1-7B

Builds on: GRPO CoPaLi VRAG agent ColQwen2 Patho-R1

Closest papers by meaning

• Patho-R1: A Multimodal Reinforcement Learning-Based Pathology Expert Reasoner (AAAI 2026 · similarity 0.90)
• PathView-Bench: Can Multimodal Large Language Models Achieve Fine-grained Multiscale Understanding of Pathology Images? (NeurIPS 2026 · similarity 0.88)
• CPathAgent: An Agent-based Foundation Model for Interpretable High-Resolution Pathology Image Analysis Mimicking Pathologists' Diagnostic Logic (NeurIPS 2025 · similarity 0.87)
• Navigating Massive Visual Context in Retrieval-Augmented Generation via Multimodal Memory Graph (ICML 2026 · similarity 0.86)
• MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflow (ICLR 2026 · similarity 0.86)
• PathAsst: A Generative Foundation AI Assistant towards Artificial General Intelligence of Pathology (AAAI 2024 · similarity 0.86)