Research map / Vision-language and multimodal / Human and embodied modeling
Automatic Radiology Reports Generation via Memory Alignment Network
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Memory Alignment module that queries a learnable memory matrix with visual embeddings plus positional embeddings to guide cross-modal alignment in a BERT-based radiology report generator.
Problem
Radiology report generation lacks image-text correspondence annotations, so aligning visual and textual features is hard; prior memory-based alignment (CMN) couples representation and alignment and is costly at inference.
Approach
Use a memory matrix only as a key/value store for alignment: queries formed from visual embeddings plus positional embeddings produce a memory alignment embedding computed once per report, which is added to the BERT encoder input, instead of re-mapping every generated word through the memory as in Cross-Modal Memory.
Key results
- On MIMIC-CXR the model reaches BLEU-1 0.396, BLEU-4 0.115, METEOR 0.151, ROUGE-L 0.274, outperforming generation-based baselines on most NLG metrics.
- On CE metrics on MIMIC-CXR, precision 0.411, recall 0.398, F1 0.389, the best among compared methods (best CE F1 vs KGAE 0.355).
- Compared with Cross-Modal Memory on the same backbone, the method uses 100 memory vectors (about 4.9% of CMM's 2,048) and inference is 0.19 s per radiograph versus 0.48 s for CMM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Memory Alignment using visual embeddings (ablation) | Baseline report generator · MIMIC-CXR | BLEU-1: 0.357 -> 0.389 | Helped |
| Add Memory Alignment using visual embeddings (ablation) | Baseline report generator · MIMIC-CXR | BLEU-4: 0.101 -> 0.110 | Helped |
| Add positional embeddings to the Memory Alignment query (ablation) | Memory Alignment with visual embeddings · MIMIC-CXR | BLEU-1: 0.389 -> 0.396 | Helped |
| Add positional embeddings to the Memory Alignment query (ablation) | Memory Alignment with visual embeddings · MIMIC-CXR | BLEU-4: 0.110 -> 0.115 | Helped |
| Replace Cross-Modal Memory with Memory Alignment using visual and positional embeddings (ablation) | Same report-generation backbone · MIMIC-CXR | BLEU-1: 0.369 -> 0.396 | Helped |
| Replace Cross-Modal Memory with Memory Alignment using visual and positional embeddings (ablation) | Same report-generation backbone · MIMIC-CXR | BLEU-4: 0.106 -> 0.115 | Helped |
| Use Memory Alignment | Radiology report generation model · MIMIC-CXR test | BLEU-1: 0.386 -> 0.396 | Helped |
| Use Memory Alignment | Radiology report generation model · IU X-Ray test | BLEU-2: 0.325 -> 0.328 | Helped |
| Use Memory Alignment | Radiology report generation model · MIMIC-CXR | F1: 0.355 -> 0.389 | Helped |
| Set memory matrix size to 100 vectors rather than 16 (ablation) | Memory Alignment report-generation model · MIMIC-CXR | BLEU-1: 0.346 -> 0.396 | Helped |
| Increase memory matrix size from 100 to 256 vectors (ablation) | Memory Alignment report-generation model · MIMIC-CXR | BLEU-1: 0.396 -> 0.384 | Hurt |
Surprising findings
- Retrieval-based KERP and KGAE are competitive or better on some metrics such as BLEU-4, which the authors attribute to generating long phrases less well than retrieval methods.
- Very small memory sizes (16 or 32 vectors) degrade performance below the no-memory baseline on MIMIC-CXR.
Limitations
- Underperforms retrieval-based methods on BLEU-4 and generates long phrases less well.
- Performance is sensitive to memory matrix size, with a peak at 100 vectors and degradation for very small or large sizes.
Experimental setup
Datasets: MIMIC-CXR IU X-Ray ImageNet (pretraining of DenseNet)
Benchmarks: MIMIC-CXR IU X-Ray
Metrics: BLEU-1 BLEU-2 BLEU-3 BLEU-4 METEOR ROUGE-L CheXpert clinical efficacy precision recall F1 inference time
Baselines: R2Gen CMN PPKED AlignTransformer M2KT HRGR KERP KGAE TOPDOWN RL-CMN
Builds on: Cross-Modal Memory Networks (CMN) BERT DenseNet
Closest papers by meaning
- DART: Disease-aware Image-Text Alignment and Self-correcting Re-alignment for Trustworthy Radiology Report Generation (CVPR 2025 · similarity 0.90)
- S2D-Align: Shallow-to-Deep Auxiliary Learning for Anatomically-Grounded Radiology Report Generation (AAAI 2026 · similarity 0.89)
- DAMPER: A Dual-Stage Medical Report Generation Framework with Coarse-Grained MeSH Alignment and Fine-Grained Hypergraph Matching (AAAI 2025 · similarity 0.88)
- CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classification (CVPR 2024 · similarity 0.86)
- Bootstrapping Large Language Models for Radiology Report Generation (AAAI 2024 · similarity 0.86)
- Image-aware Evaluation of Generated Medical Reports (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.