Research map / Vision-language and multimodal / Multimodal LLMs
LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Instruction-tunes a text-only LLM (dolly-v2-3b) on MIMIC-CXR with VQ-GAN image tokens expanded into its vocabulary, enabling CXR report generation, CXR image generation, and CXR VQA in one model.
Problem
Adapter-based multimodal LLMs leave a bottleneck between image and text representations, limiting vision-language alignment for medical images such as chest X-rays, which need fine-grained visual detail for both understanding and generation.
Approach
Treat VQ-GAN image tokens as added vocabulary of a pretrained instruction-tuned LLM and fine-tune it on bidirectional CXR tasks (report-to-CXR, CXR-to-report, CXR-VQA, NL instruction following) with a conditional response-only loss, plus a clinical feature-preserving L2 loss in the VQ-GAN and LLM-generated synthetic VQA data.
Key results
- LLM-CXR (2.8B dolly-v2-3b base) achieves micro AUROC 0.597 and micro F1 0.360 on CXR-to-report CheXpert labels, matching or exceeding UniXGen-512 and XrayGPT at 256px in several metrics (e.g., weighted AUROC 0.597 vs 0.577 for XrayGPT).
- On CXR-VQA, LLM-CXR reaches 44.8% accuracy versus 25.2% for XrayGPT and 32.7% for RadFM, though below ELIXR's 54.8%.
- For report-to-CXR, LLM-CXR gets FID 22.75 (inception) and 0.7136 (txv-all-1024), better than UniXGen (78.19 / 7.894) and RoentGen (42.38 / 6.039); micro AUROC 0.8065 vs 0.7061 for RoentGen.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add clinical information-preserving loss (ablation) | LLM-CXR · CXR-to-report | AUROC weighted: 0.5853 -> 0.5969 | Helped |
| add simultaneous CXR-VQA training (ablation) | LLM-CXR · CXR-to-report | AUROC weighted: 0.5846 -> 0.5969 | Helped |
| include stage 1 training on the full dataset (ablation) | LLM-CXR · CXR-to-report | AUROC weighted: 0.5435 -> 0.5969 | Helped |
| use instruction-tuning loss instead of joint loss (ablation) | LLM-CXR · CXR-to-report | AUROC weighted: 0.5745 -> 0.5969 | Helped |
| add clinical information-preserving loss (ablation) | LLM-CXR · Report-to-CXR | AUROC weighted: 0.8223 -> 0.8054 | Hurt |
| add clinical information-preserving loss (ablation) | LLM-CXR · Report-to-CXR | F1 weighted: 0.7977 -> 0.7991 | Helped |
| add simultaneous CXR-VQA training (ablation) | LLM-CXR · Report-to-CXR | AUROC weighted: 0.7465 -> 0.8054 | Helped |
| include stage 1 training on the full dataset (ablation) | LLM-CXR · Report-to-CXR | AUROC weighted: 0.7167 -> 0.8054 | Helped |
| use instruction-tuning loss instead of joint loss (ablation) | LLM-CXR · Report-to-CXR | AUROC weighted: 0.5808 -> 0.8054 | Helped |
| add clinical information-preserving loss (ablation) | LLM-CXR · Report-to-CXR | FID txv-all-1024: 0.931 -> 0.714 | Helped |
| add simultaneous CXR-VQA training (ablation) | LLM-CXR · Report-to-CXR | FID txv-all-1024: 1.539 -> 0.714 | Helped |
| include stage 1 training on the full dataset (ablation) | LLM-CXR · Report-to-CXR | FID txv-all-1024: 1.226 -> 0.714 | Helped |
Surprising findings
- Removing the CIP loss raised report-to-CXR AUROC (0.8227 vs 0.8065) while lowering F1, showing a mixed effect.
- Inception-v3 FID disagreed with the CXR-specific txv FID; some ablations improved Inception FID while worsening the CXR-specific one.
Limitations
- Generated reports still include false positives and missed diagnoses.
- Fixed 256 image tokens per image make generation slow (30-60 s per image).
- Reports referencing prior studies add noise; longitudinal information is not used.
Experimental setup
Datasets: MIMIC-CXR-JPG Indiana University chest X-ray dataset synthetic CXR VQA (Llama2-generated)
Benchmarks: MIMIC-CXR-JPG ELIXR VQA question set
Metrics: AUROC F1 Jaccard similarity index BLEU METEOR ROUGE-L CIDEr FID VQA accuracy
Baselines: UniXGen XrayGPT RadFM IFCC R2Gen ELIXR RoentGen
Builds on: VQ-GAN Alpaca instruction tuning Dolly-v2 TorchXRayVision
Closest papers by meaning
- ExGra-Med: Extended Context Graph Alignment for Medical Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- AOR: Anatomical Ontology-Guided Reasoning for Medical Large Multimodal Model in Chest X-Ray Interpretation (NeurIPS 2025 · similarity 0.88)
- CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rays (NeurIPS 2025 · similarity 0.88)
- MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Text (AAAI 2026 · similarity 0.88)
- LLaDA-MedV: Exploring Large Language Diffusion Models for Biomedical Image Understanding (CVPR 2026 · similarity 0.87)
- MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.