Research map / Large language models / Language, safety and interpretability
Bootstrapping Large Language Models for Radiology Report Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Bootstraps MiniGPT-4 (Vicuna-13B) for radiology report generation via contrastive in-domain instance induction on retrieved reports and coarse-to-fine decoding with refinement prompts, reaching SOTA on IU X-Ray and MIMIC-CXR.
Problem
Radiology report generation from chest X-rays is hard for LLMs pre-trained on general data because of a domain gap between general and medical text and limited in-domain labeled data for aligning visual and textual features.
Approach
Two components on top of MiniGPT-4: in-domain instance induction, which uses a MedCLIP-based retrieval of RRG training reports and PubMed documents plus a contrastive pairwise Euclidean-distance ranking loss on LLM hidden states; and coarse-to-fine decoding, which regenerates a final report from an intermediate report, visual features and a refinement prompt.
Key results
- On MIMIC-CXR, MiniGPT-4 (FT)+I3+C2FD reaches BLEU-4 0.128, ROUGE-L 0.291 and CheXpert F1 0.473, versus 0.109 and 0.287 for R2GENRL (best prior BLEU-4 baseline).
- On IU X-Ray, the full model reaches BLEU-4 0.184 and ROUGE-L 0.390, above the prior best R2GENRL (0.181, 0.384).
- Ablation: I3 alone raises average NLG score by 14.9% over MiniGPT-4 (FT) on MIMIC-CXR; C2FD alone raises it by 9.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add I3 to MiniGPT-4 (FT) (ablation) | MiniGPT-4 (FT) · IU X-RAY test | BLEU-1: 0.389 -> 0.458 | Helped |
| Add C2FD to MiniGPT-4 (FT) (ablation) | MiniGPT-4 (FT) · IU X-RAY test | BLEU-1: 0.389 -> 0.437 | Helped |
| Add both I3 and C2FD to MiniGPT-4 (FT) (ablation) | MiniGPT-4 (FT) · IU X-RAY test | BLEU-1: 0.389 -> 0.499 | Helped |
| Add I3 to MiniGPT-4 (FT) (ablation) | MiniGPT-4 (FT) · MIMIC-CXR test | BLEU-1: 0.323 -> 0.358 | Helped |
| Add C2FD to MiniGPT-4 (FT) (ablation) | MiniGPT-4 (FT) · MIMIC-CXR test | BLEU-1: 0.323 -> 0.345 | Helped |
| Add both I3 and C2FD to MiniGPT-4 (FT) (ablation) | MiniGPT-4 (FT) · MIMIC-CXR test | BLEU-1: 0.323 -> 0.402 | Helped |
| Use both training data and public medical corpora for retrieval rather than training data only (ablation) | I3 · MIMIC-CXR test | BLEU-1: 0.339 -> 0.358 | Helped |
| Use both training data and public medical corpora for retrieval rather than public medical data only (ablation) | I3 · MIMIC-CXR test | BLEU-1: 0.310 -> 0.358 | Helped |
| Refine the intermediate report with C2FD (ablation) | MiniGPT-4 (FT)+I3 · MIMIC-CXR test | BLEU-1: 0.364 -> 0.402 | Helped |
Surprising findings
- Using only the in-domain training reports as retrieval source (I3 (TD)) or only PubMed (I3 (PMD)) both underperform combining the two sources.
Limitations
- Relies on MiniGPT-4 with a 13B Vicuna backbone, which is computationally heavy.
- Reports that LLaVA can replace MiniGPT-4 but only lists results for MiniGPT-4; the method's dependence on a specific multimodal LLM is not fully explored.
- Evaluation limited to NLG and CheXpert-based CE metrics on two datasets; no human or radiologist evaluation.
Experimental setup
Datasets: IU X-Ray MIMIC-CXR PubMed
Benchmarks: IU X-Ray MIMIC-CXR
Metrics: BLEU-1 BLEU-2 BLEU-3 BLEU-4 METEOR ROUGE-L CheXpert precision CheXpert recall CheXpert F1
Baselines: R2GEN R2GENCMN R2GENRL RGRG XRAYGPT MED-PALM WARMSTART MiniGPT-4 (FT)
Builds on: MiniGPT-4 MedCLIP R2Gen
Closest papers by meaning
- HC-LLM: Historical-Constrained Large Language Models for Radiology Report Generation (AAAI 2025 · similarity 0.93)
- LLM-RG4: Flexible and Factual Radiology Report Generation Across Diverse Input Contexts (AAAI 2025 · similarity 0.92)
- S2D-Align: Shallow-to-Deep Auxiliary Learning for Anatomically-Grounded Radiology Report Generation (AAAI 2026 · similarity 0.89)
- Instance-level Expert Knowledge and Aggregate Discriminative Attention for Radiology Report Generation (CVPR 2024 · similarity 0.87)
- Radiology Report Generation via Multi-objective Preference Optimization (AAAI 2025 · similarity 0.87)
- AHIVE: Anatomy-aware Hierarchical Vision Encoding for Interactive Radiology Report Retrieval (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.