Research map / Label-efficient and robust learning / Recognition and long-tail
Instance-level Expert Knowledge and Aggregate Discriminative Attention for Radiology Report Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EKAGen, a radiology report generator that uses instance-level expert knowledge retrieved from a unified embedding space, weakly supervised aggregate discriminative attention maps, and global self-distillation to reach SOTA on IU X-Ray and MIMIC-CXR.
Problem
Automatic chest X-ray report generation struggles with feature degradation when injecting prior medical knowledge (due to mismatched embedding networks) and with focusing attention on pivotal abnormal regions without pixel-level annotations.
Approach
Unifies the prior-knowledge embedding network with the language generation model, selects rectified knowledge support via cosine similarity to disease prototypes, builds per-image expert knowledge from diagnosed diseases, uses Grad-CAM-based multi-disease attention maps as foreground masks, and distills from an iteratively updated teacher whose weights are copied from the student when it improves.
Key results
- On MIMIC-CXR, EKAGen (RN-101) reports a 2.6% BLEU-1 gain over prior SOTA as stated in the text, and clinical efficacy F1 of 0.499 versus 0.373 for the best listed baseline (DCL).
- On IU X-Ray, the full EKAGen (RN-101) reaches BLEU-1 0.526 and BLEU-4 0.203 versus BASE 0.463 and 0.149 in the ablation.
- Replacing a separate BERT prior-knowledge encoder with the unified embedding lowers IU X-Ray BLEU-1 from 0.526 to 0.507 (BERT).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Instance-level Expert Knowledge (IEK) to the BASE model (ablation) | Radiology report generation · IU X-Ray | BLEU-1: 0.463 -> 0.501 | Helped |
| add preliminary expert knowledge support (PKS) (ablation) | EKAGen · IU X-Ray | BLEU-1: 0.463 -> 0.475 | Helped |
| use nearest original-space features as rectified knowledge support (RKS) (ablation) | EKAGen with PKS · IU X-Ray | BLEU-1: 0.475 -> 0.494 | Helped |
| retrieve instance-level expert knowledge based on patient diagnosis (EKN) (ablation) | EKAGen with RKS · IU X-Ray | BLEU-1: 0.494 -> 0.501 | Helped |
| add Aggregate Discriminative Attention Map (ADM) (ablation) | BASE + IEK · IU X-Ray | BLEU-4: 0.170 -> 0.198 | Helped |
| add Global Information Self-Distillation (GID) (ablation) | BASE + IEK + ADM · IU X-Ray | BLEU-1: 0.509 -> 0.526 | Helped |
| use a separate BERT prior-knowledge embedding network instead of the unified embedding network (ablation) | EKAGen · IU X-Ray | BLEU-1: 0.526 -> 0.507 | Hurt |
| use a separate BERT prior-knowledge embedding network instead of the unified embedding network (ablation) | EKAGen · MIMIC-CXR | BLEU-1: 0.419 -> 0.409 | Hurt |
| use ViT-B/16 instead of ResNet-101 as the visual extractor (ablation) | EKAGen · IU X-Ray and MIMIC-CXR | BLEU-1: 0.526 -> 0.517; 0.419 -> 0.415 | Mixed |
Surprising findings
- The ViT-B/16 visual extractor underperforms ResNet-101 on both datasets, which the authors attribute to CNNs capturing local fine-grained features.
- EKAGen is slightly below KiUT and METransformer on a few NLG metrics, which the authors attribute to KiUT's synonym handling and METransformer's word voting.
Limitations
- Performance gap to ViT-B/16 backbone suggests sensitivity to visual extractor choice.
- Some NLG metrics remain below KiUT and METransformer.
Experimental setup
Datasets: IU X-Ray MIMIC-CXR
Benchmarks: IU X-Ray MIMIC-CXR
Metrics: BLEU-1 BLEU-2 BLEU-3 BLEU-4 METEOR ROUGE-L CE Precision CE Recall CE F1 alignment score
Baselines: R2GenCMN GSKET Clinical-BERT KiUT METransformer DCL PPKED R2GenRL M2transformer BLIP CA Grounded
Builds on: Grad-CAM Pre-Norm Transformer knowledge distillation prototype networks
Closest papers by meaning
- Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation (CVPR 2021 · similarity 0.90)
- Bootstrapping Large Language Models for Radiology Report Generation (AAAI 2024 · similarity 0.87)
- AHIVE: Anatomy-aware Hierarchical Vision Encoding for Interactive Radiology Report Retrieval (CVPR 2024 · similarity 0.87)
- MEPNet: Medical Entity-Balanced Prompting Network for Brain CT Report Generation (AAAI 2025 · similarity 0.85)
- LLM-RG4: Flexible and Factual Radiology Report Generation Across Diverse Input Contexts (AAAI 2025 · similarity 0.85)
- Automatic Radiology Reports Generation via Memory Alignment Network (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.