Research map / Large language models / Language, safety and interpretability
HC-LLM: Historical-Constrained Large Language Models for Radiology Report Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HC-LLM, which adds similarity, multimodal contrastive, and structural constraints on time-shared and time-specific features of longitudinal chest X-rays and reports to adapt LLM-based radiology report generation.
Problem
Radiology report generation usually uses only the current chest X-ray, ignoring historical images and reports needed to describe disease progression; LLMs given longitudinal data often fail to capture that progression.
Approach
Extract time-shared and time-specific features from current/prior X-rays and reports, then constrain them with intra-modality MSE similarity, InfoNCE-style multimodal contrastive loss, and distance/angle structural (triangle) constraints to guide LLM report generation; the constraints can be plugged into different multimodal LLM frameworks.
Key results
- On Longitudinal-MIMIC with the MiniGPT4 framework, HC-LLM reports BLEU-4 0.142 and CE F1 0.498, versus 0.119 and 0.366 for MiniGPT4 with report and image history.
- Within the R2GenGPT framework on Longitudinal-MIMIC, HC-LLM reaches BLEU-1 0.404 and ROUGE-L 0.287 with both historical report and image, improving over R2GenGPT's 0.389 and 0.278.
- Without historical data at test time, HC-LLM (R2GenGPT framework) achieves BLEU-4 0.107 versus 0.103 for R2GenGPT and 0.098 for PromptMRG on Longitudinal-MIMIC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HC-LLM instead of R2GenGPT + report & image | R2GenGPT framework · Longitudinal-MIMIC | BLEU-4: 0.117 -> 0.128 | Helped |
| Use HC-LLM instead of PromptMRG | Longitudinal-MIMIC report generation · Longitudinal-MIMIC | F-1: 0.482 -> 0.498 | Helped |
| Use HC-LLM instead of R2GenGPT + report | R2GenGPT framework · MS-CXR-T | BLEU-4: 0.129 -> 0.136 | Helped |
| Use HC-LLM instead of MiniGPT4 + report & image | MiniGPT4 framework · MS-CXR-T | F-1: 0.491 -> 0.514 | Helped |
| Add the similarity constraint (ablation) | HC-LLM with contrastive and structural constraints · Longitudinal-MIMIC | BLEU-4: 0.119 -> 0.128 | Helped |
| Add the contrastive constraint (ablation) | HC-LLM with similarity and structural constraints · Longitudinal-MIMIC | BLEU-4: 0.118 -> 0.128 | Helped |
| Add the structural constraint (ablation) | HC-LLM with similarity and contrastive constraints · Longitudinal-MIMIC | BLEU-4: 0.121 -> 0.128 | Helped |
| Add the contrastive and structural constraints (ablation) | HC-LLM without similarity constraint · Longitudinal-MIMIC | BLEU-4: 0.117 -> 0.119 | Helped |
| Increase the structural-constraint coefficient beyond its optimal value (ablation) | HC-LLM on MS-CXR-T · MS-CXR-T | report-generation performance | Hurt |
| Use a larger LLM | HC-LLM · Longitudinal-MIMIC | BLEU-4 | Helped |
| Test with only the current chest X-ray rather than historical data | HC-LLM · Longitudinal-MIMIC without historical data at test time | BLEU-4: 0.098 -> 0.107 | Helped |
Surprising findings
- Standard LLM-based longitudinal baselines (R2GenGPT, BioMedGPT, MiniGPT4) perform worse than the single-image PromptMRG when tested with only the current chest X-ray.
- Removing the similarity constraint causes only a minor drop, while removing the contrastive constraint causes a significant drop.
Limitations
- Only two time points of longitudinal data are used; multi-historical-point data is not explored.
- Evaluated only on Longitudinal-MIMIC and MS-CXR-T, as no standard longitudinal RRG benchmark exists.
- On the small MS-CXR-T dataset, conventional LLM baselines often underperform traditional single-image models.
Experimental setup
Datasets: Longitudinal-MIMIC MS-CXR-T MIMIC-CXR
Benchmarks: Longitudinal-MIMIC MS-CXR-T
Metrics: BLEU-1 BLEU-2 BLEU-3 BLEU-4 METEOR ROUGE-L CE precision CE recall CE F1
Baselines: AoANet R2Gen R2CMN R2GenRL CvT2DistilGPT2 PromptMRG Prefilling R2GenGPT BioMedGPT MiniGPT4
Builds on: R2GenGPT MiniGPT-4 Prefilling Swin Transformer CheXbert
Closest papers by meaning
- Bootstrapping Large Language Models for Radiology Report Generation (AAAI 2024 · similarity 0.93)
- LLM-RG4: Flexible and Factual Radiology Report Generation Across Diverse Input Contexts (AAAI 2025 · similarity 0.90)
- Enhanced Contrastive Learning with Multi-view Longitudinal Data for Chest X-ray Report Generation (CVPR 2025 · similarity 0.88)
- S2D-Align: Shallow-to-Deep Auxiliary Learning for Anatomically-Grounded Radiology Report Generation (AAAI 2026 · similarity 0.87)
- Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generation (NeurIPS 2025 · similarity 0.86)
- Unlocking the Power of Spatial and Temporal Information in Medical Multimodal Pre-training (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.