Research map / Vision-language and multimodal / Human and embodied modeling
PromptMRG: Diagnosis-Driven Prompts for Medical Report Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PromptMRG, which converts a disease classification branch's outputs into token prompts to guide a chest X-ray report decoder, and adds retrieval-based feature enhancement and self-adaptive logit-adjusted loss to improve clinical efficacy on MIMIC-CXR and IU X-Ray.
Problem
Medical report generation from chest X-rays often produces diagnostically inaccurate reports, and rare diseases are underrepresented, so their diagnostic performance is unreliable.
Approach
Diagnostic outputs of an auxiliary classification branch are turned into special token prompts ([POS], [NEG], [BLA], [UNC]) that explicitly condition the text decoder; retrieval of similar reports via a frozen CLIP enhances classification features, and an adaptive logit-adjusted loss, driven by per-disease prediction scores, rebalances rare diseases.
Key results
- On MIMIC-CXR, PromptMRG reaches 0.476 CE F1, versus 0.447 for the best prior method RGRG and 0.373 for DCL.
- On IU X-Ray (re-evaluated by the authors), PromptMRG reaches 0.211 CE F1 versus 0.180 for RGRG.
- Ablation on MIMIC: DDP raises CE F1 from 0.370 to 0.444; adding all modules (ADL, CFE, SDL) yields 0.476.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add diagnosis-driven token prompts (ablation) | PromptMRG · MIMIC-CXR test | F1: 0.370 -> 0.444 | Helped |
| add auxiliary disease labeling (ablation) | PromptMRG with DDP · MIMIC-CXR test | F1: 0.444 -> 0.451 | Helped |
| add cross-modal feature enhancement (ablation) | PromptMRG with DDP and ADL · MIMIC-CXR test | F1: 0.451 -> 0.464 | Helped |
| add self-adaptive disease-balanced learning (ablation) | PromptMRG with DDP and ADL · MIMIC-CXR test | F1: 0.451 -> 0.468 | Helped |
| add diagnosis-driven prompts (ablation) | PromptMRG · MIMIC-CXR test | BLEU-4: 0.116 -> 0.106 | Hurt |
| replace token prompts with text prompts (ablation) | PromptMRG · MIMIC-CXR test | F1: 0.451 -> 0.446 | Hurt |
| replace token prompts with feature prompts (ablation) | PromptMRG · MIMIC-CXR test | F1: 0.370 -> 0.357 | Hurt |
| replace token prompts with embedding prompts (ablation) | PromptMRG · MIMIC-CXR test | F1: 0.370 -> 0.434 | Helped |
| add SDL (ablation) | PromptMRG · MIMIC test | Macro-averaged F1: 0.319 -> 0.381 | Helped |
| add SDL (ablation) | PromptMRG · IU test | Macro-averaged F1: 0.193 -> 0.246 | Helped |
| use token prompts instead of text prompts (ablation) | PromptMRG · MIMIC-CXR test | F1: 0.446 -> 0.451 | Helped |
Surprising findings
- Token-based diagnosis prompts reduce NLG scores (about 0.8% on average) on MIMIC while improving clinical efficacy, which the authors attribute to less repetition of frequent training phrases.
- Feature prompts (raw pooled features) degrade CE F1 by 1.3% relative to no prompt, while explicit token or text prompts help.
Limitations
- Requires disease labels for training the classification branch; applicability to other domains with limited image-report pairs is unverified.
- Average example-based F1 improves only modestly (1.2%) from SDL despite large gains on rare diseases, since the metric weights frequent diseases more.
- The IU X-Ray test split was re-defined by the authors (normal images subsampled), so results are not directly comparable to prior work.
Experimental setup
Datasets: MIMIC-CXR IU X-Ray
Benchmarks: MIMIC-CXR IU X-Ray
Metrics: CE precision CE recall CE F1 BLEU-1 BLEU-4 METEOR ROUGE-L
Baselines: R2Gen M2TR MKSG CliBert CVT2Dis. M2KT METrans. KiUT DCL RGRG
Builds on: logit-adjusted loss CLIP CheXbert R2Gen
Closest papers by meaning
- Bootstrapping Large Language Models for Radiology Report Generation (AAAI 2024 · similarity 0.86)
- DAMPER: A Dual-Stage Medical Report Generation Framework with Coarse-Grained MeSH Alignment and Fine-Grained Hypergraph Matching (AAAI 2025 · similarity 0.85)
- From Complexity to Clarity: Transforming Chest X-ray Reports with Chained Prompting (Student Abstract) (AAAI 2025 · similarity 0.85)
- LLM-RG4: Flexible and Factual Radiology Report Generation Across Diverse Input Contexts (AAAI 2025 · similarity 0.84)
- Large Language Models Are Clinical Reasoners: Reasoning-Aware Diagnosis Framework with Prompt-Generated Rationales (AAAI 2024 · similarity 0.84)
- Instance-level Expert Knowledge and Aggregate Discriminative Attention for Radiology Report Generation (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.