Research map / Vision-language and multimodal / Human and embodied modeling
BenchX: A Unified Benchmark Framework for Medical Vision-Language Pretraining on Chest X-Rays
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BenchX, a unified benchmark framework that standardizes data, preprocessing, and task-adaptation protocols to fairly compare nine medical vision-language pretraining methods on chest X-ray classification, segmentation, report generation, and retrieval.
Problem
Existing medical vision-language pretraining (MedVLP) methods are evaluated with differing datasets, preprocessing, and finetuning setups, making fair head-to-head comparison on chest X-ray tasks impossible.
Approach
Pretrain all MedVLP methods on the same frontal MIMIC-CXR training split, standardize preprocessing and hyperparameter grid search, and use unified task heads (linear classifier, frozen-encoder UperNet for segmentation, frozen-encoder R2Gen for report generation) so that differences reflect the pretrained encoder; also shows that classification training tricks (LayerNorm, truncated normal init, di
Key results
- On NIH multi-label classification with 100% data, MRM achieves 85.3 AUROC, the best of the compared methods.
- After applying LN, TNI and DLR training strategies, ConVIRT's NIH AUROC rises from 71.7 to 77.0 and becomes the third-best method, showing configuration sensitivity.
- Averaged across tasks, MRM (avg. rank 2.00) and MGCA-ViT (avg. rank 2.38) rank highest, while M-FLAG ranks last (10.00).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LN, TNI, and DLR instead of no refinements (ablation) | ConVIRT · NIH classification | AUROC: 71.7 -> 77.0 | Helped |
| Apply DLR (ablation) | ConVIRT · NIH classification | AUROC: 71.7 -> 76.9 | Helped |
| Add LN after DLR (ablation) | ConVIRT · NIH classification | AUROC: 76.9 -> 74.5 | Hurt |
| Apply all refinements (ablation) | MGCA-ViT · NIH classification | AUROC: 78.2 (=) | No clear effect |
| Add LN, TNI, and DLR instead of no refinements (ablation) | MRM · NIH classification | AUROC: 77.9 -> 80.1 | Helped |
| Add LN, TNI, and DLR instead of no refinements (ablation) | REFERS · NIH classification | AUROC: 76.8 -> 75.6 | Hurt |
| Use TNI alone (ablation) | MedVLP classification models · Classification | AUROC | Hurt |
| Use ViT-based rather than ResNet-based encoder | MedCLIP and MGCA · Classification | AUROC or F1 | Helped |
| Use 10% rather than 100% of training data | MedVLP methods · All classification datasets except SIIM | AUROC or F1 | No clear effect |
| Use MedVLP pretraining rather than the from-scratch baseline | Report generation · IU-Xray | BLEU4: 0.133$\pm$0.018 -> 0.148$\pm$0.006 | Helped |
| Use PTUnifier rather than the second-best method | Image-text retrieval · MIMIC 5x200 | H@1: 63.3 -> 78.7 | Helped |
| Use PTUnifier rather than MGCA-ViT | Image-text retrieval · MIMIC 5x200 | P@5: 56.4 -> 38.4 | Hurt |
Surprising findings
- Early method ConVIRT, once finetuned with refined training strategies, matches or beats recent MedVLP methods such as MedCLIP, MedKLIP, and REFERS in several settings.
- Naive linear-head finetuning is suboptimal for nearly all MedVLP methods, and the best training configuration differs per method.
Limitations
- Focuses on frozen or pretrained image-encoder performance on classification and segmentation; cross-modal tasks such as VQA are not covered.
- Only public chest X-ray datasets are used, so generalization to other medical imaging modalities is untested.
- Hyperparameter grid search may be incomplete, so some methods could be suboptimal.
Experimental setup
Datasets: MIMIC-CXR NIH ChestX-ray VinDr-CXR COVIDx CXR-4 SIIM-ACR Pneumothorax RSNA Pneumonia Object CXR TBX11K IU X-ray MIMIC 5x200
Benchmarks: NIH ChestX-ray VinDr-CXR COVIDx CXR-4 SIIM RSNA Pneumonia Object CXR TBX11K IU X-ray MIMIC 5x200
Metrics: AUROC F1 mDice BLEU ROUGE-L METEOR Hit@K Precision@K
Baselines: ConVIRT GLoRIA MedCLIP-R50 MedCLIP-ViT MedKLIP M-FLAG MGCA-R50 MGCA-ViT MRM PTUnifier REFERS
Builds on: ConVIRT GLoRIA MedCLIP MGCA MRM
Closest papers by meaning
- CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rays (NeurIPS 2025 · similarity 0.90)
- Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models (NeurIPS 2026 · similarity 0.89)
- RadZero: Similarity-Based Cross-Attention for Explainable Vision-Language Alignment in Chest X-ray with Zero-Shot Multi-Task Capability (NeurIPS 2025 · similarity 0.89)
- Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-training (ICML 2024 · similarity 0.88)
- MedSG-Bench: A Benchmark for Medical Image Sequences Grounding (NeurIPS 2025 · similarity 0.87)
- How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.