Research map / Large language models / Instruction and fine-tuning
CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks ~20 X-ray report generation models and 16 LLMs/VLMs on CheXpert Plus without truncation, and proposes MambaXray-VL, a Mamba-based vision encoder trained with autoregressive pre-training, image-text contrastive learning, and SFT.
Problem
X-ray medical report generation lacks comparative baselines and standardized results on the newly released large-scale CheXpert Plus dataset, and existing vision encoders are costly (Transformer O(N^2)) and pre-trained in a single stage.
Approach
Combine a Mamba (Vim) vision encoder pre-trained with autoregressive next-token generation on 1.27M X-ray images, then align it with reports via contrastive learning, then fine-tune with an LLM decoder; also benchmark many existing MRG models and LLMs on CheXpert Plus using full (non-truncated) ground truth.
Key results
- MambaXray-VL-L achieves BLEU-4 0.112, ROUGE-L 0.276, METEOR 0.157, CIDEr 0.139 and clinical F1 0.335 on CheXpert Plus, the best among the benchmarked models in Table 1.
- Autoregressive pre-training (ARG) outperforms MAE on all metrics on MIMIC-CXR and CheXpert Plus, with a +45% relative CIDEr gain on MIMIC-CXR (0.224 vs 0.154).
- Adding image-text contrastive learning improves ROUGE-L by over 11% on CheXpert Plus, and Bio ClinicalBERT outperforms Llama2 as the contrastive text encoder (CheXpert Plus CIDEr 0.139 vs 0.043).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use autoregressive generation (ARG) instead of MAE pre-training (ablation) | MambaXray-VL · MIMIC-CXR | BLEU-4: 0.104 -> 0.130 | Helped |
| Use autoregressive generation (ARG) instead of MAE pre-training (ablation) | MambaXray-VL · CheXpert Plus | BLEU-4: 0.094 -> 0.089 | Hurt |
| Add image-text contrastive learning (CTL) (ablation) | MambaXray-VL · MIMIC-CXR | F1: 0.255 -> 0.340 | Helped |
| Add image-text contrastive learning (CTL) (ablation) | MambaXray-VL · CheXpert Plus | F1: 0.248 -> 0.335 | Helped |
| Replace the Transformer encoder with a Mamba encoder using ARG pre-training (ablation) | MambaXray-VL · MIMIC-CXR | BLEU-4: 0.125 -> 0.133 | Helped |
| Replace the Transformer encoder with a Mamba encoder using ARG pre-training (ablation) | MambaXray-VL · CheXpert Plus | BLEU-4: 0.101 -> 0.112 | Helped |
| Use Vim-PTD weights rather than Vim-IN1K weights (ablation) | MambaXray-VL · MIMIC-CXR | BLEU-4: 0.105 -> 0.133 | Helped |
| Use Vim-PTD weights rather than Vim-IN1K weights (ablation) | MambaXray-VL · CheXpert Plus | BLEU-4: 0.082 -> 0.112 | Helped |
| Use Bio ClinicalBERT instead of Llama2 for contrastive learning (ablation) | MambaXray-VL · MIMIC-CXR | BLEU-4: 0.122 -> 0.133 | Helped |
| Use Bio ClinicalBERT instead of Llama2 for contrastive learning (ablation) | MambaXray-VL · CheXpert Plus | BLEU-4: 0.066 -> 0.112 | Helped |
| Use Vim-large instead of Vim-base (ablation) | MambaXray-VL · CheXpert Plus | F1: 0.273 -> 0.335 | Helped |
Surprising findings
- Vision-language models InternVL-2 and MiniCPM-V2.5 perform worse than most text-only LLM-based report generators, suggesting natural-image VLM pre-training transfers poorly to X-ray reports.
- On MIMIC-CXR, the Mamba encoder's CIDEr is not significantly better than the Transformer baseline despite gains on BLEU-4.
Limitations
- Pre-training uses a limited set of paired report data; the paper notes aligned X-ray-report data are scarce.
- Text encoder choice shows Llama2 underperforms in contrastive alignment, suggesting general LLMs are poorly suited without medical pre-training.
- Results for IU X-ray and MIMIC-CXR are relegated to supplementary material.
Experimental setup
Datasets: CheXpert Plus MIMIC-CXR IU X-ray 1.27M chest X-ray image corpus from Wang et al.
Benchmarks: CheXpert Plus IU X-ray MIMIC-CXR
Metrics: BLEU-4 ROUGE-L METEOR CIDEr clinical efficacy precision clinical efficacy recall clinical efficacy F1
Baselines: R2GenGPT XProNet R2GenCSR R2Gen PromptMRG Token-Mixer ORGan WCL
Builds on: ARM Vim R2GenGPT CLIP R2GenCSR
Closest papers by meaning
- BenchX: A Unified Benchmark Framework for Medical Vision-Language Pretraining on Chest X-Rays (NeurIPS 2024 · similarity 0.86)
- CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rays (NeurIPS 2025 · similarity 0.86)
- Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generation (CVPR 2025 · similarity 0.85)
- HC-LLM: Historical-Constrained Large Language Models for Radiology Report Generation (AAAI 2025 · similarity 0.84)
- Reporting Practice Matters: The Impact of Reference Choice on Chest X-ray Report Evaluation (NeurIPS 2026 · similarity 0.84)
- PromptMRG: Diagnosis-Driven Prompts for Medical Report Generation (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.