Research map / Vision-language and multimodal / CLIP and zero-shot
PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaboration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds PathGen-1.6M, a 1.6M pathology image-caption dataset generated from TCGA WSIs by a multi-agent pipeline, and uses it to train PathGen-CLIP and PathGen-LLaVA models.
Problem
Pathology vision-language models are limited by small, noisy image-text pairs from PubMed, YouTube, and Twitter, which are unscalable and low quality.
Approach
Mine representative WSI patches via prompt-based CLIP retrieval plus k-means, then generate, revise, and summarize captions with a chain of specialized pathology LMM and LLM agents, yielding a large high-quality image-text corpus from unpaired WSIs and reports.
Key results
- PathGen-CLIP-L reaches 79.7% average zero-shot accuracy across nine pathology classification datasets, versus 66.2% for the best prior model PathCLIP.
- PathGen-LLaVA scores 58.4% overall on the PathMMU test set, above GPT-4V at 49.8%, and well above Quilt-LLaVA at 41.5%.
- On CAMELYON16 with ABMIL, PathGen-CLIP achieves an average AUC of 96.9 versus 90.0 for PLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the CLIP model using PathGen-1.6M data and then fine-tune with PathGen_init | PathGen-CLIP · Zero-shot pathology classification (Average) | accuracy (%): 66.2 -> 74.3 | Helped |
| Use the larger PathGen-CLIP-L variant | PathGen-CLIP · Zero-shot pathology classification (Average) | accuracy (%): 74.3 -> 79.7 | Helped |
| Use PathGen-CLIP-L as the image encoder | ABMIL for WSI classification · WSI classification (Average) | F1-score: 71.4 -> 73.2 | Helped |
| Use PathGen-CLIP-L as the image encoder | ACMIL for WSI classification · WSI classification (Average) | AUC: 91.2 -> 92.6 | Helped |
| Train PathGen-LLaVA using PathGen-based instruction-tuning data and the PathGen-CLIP backbone | PathGen-LLaVA · PathMMU test set (Test Overall, ALL) | accuracy (%): 49.8 -> 58.4 | Helped |
Surprising findings
- A pathology CLIP trained on synthetic captions from a multimodal model outperforms GPT-4V on PathMMU for the LMM variant, despite the LMM being much smaller in its training signal.
Limitations
- Approach partially relies on WSI pathology reports, which are difficult to obtain.
- Generated captions depend on GPT-4V/GPT-4 outputs and LMM hallucinations, mitigated only by a revision agent.
- Evaluated only on TCGA-derived and public pathology datasets; PathGen-LLaVA evaluated mainly on PathMMU.
Experimental setup
Datasets: TCGA PathCap Quilt-1M OpenPath PathMMU PubMed-derived pairs CAMELYON16 CAMELYON17 BRACS PatchCamelyon CRC-100K SICAPv2
Benchmarks: PathMMU CAMELYON16 CAMELYON17 BRACS PatchCamelyon CRC-100K SICAPv2 BACH Osteo SkinCancer WSSSLUAD LC-Lung
Metrics: accuracy F1-score AUC
Baselines: OpenAI-CLIP OpenAI-CLIP-L PLIP PubMedCLIP PMC-CLIP QuiltNet PathCLIP BiomedCLIP LLaVA-Med Quilt-LLaVA GPT-4V Gemini Pro Vision
Builds on: CLIP LLaVA-v1.5 Quilt-1M ABMIL ACMIL HistGen
Closest papers by meaning
- PathAsst: A Generative Foundation AI Assistant towards Artificial General Intelligence of Pathology (AAAI 2024 · similarity 0.91)
- PathView-Bench: Can Multimodal Large Language Models Achieve Fine-grained Multiscale Understanding of Pathology Images? (NeurIPS 2026 · similarity 0.90)
- PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathology (AAAI 2026 · similarity 0.90)
- CPath-Omni: A Unified Multimodal Foundation Model for Patch and Whole Slide Image Analysis in Computational Pathology (CVPR 2025 · similarity 0.90)
- PathVQ: Reforming Computational Pathology Foundation Model for Whole Slide Image Analysis via Vector Quantization (NeurIPS 2025 · similarity 0.90)
- Patho-R1: A Multimodal Reinforcement Learning-Based Pathology Expert Reasoner (AAAI 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.