Research map / Image generation / Autoregressive and latent generation
Image-aware Evaluation of Generated Medical Reports
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VLScore, a image-aware metric for X-ray report evaluation that scores the triangle area formed by image, generated report, and reference report embeddings in a LIMITR shared space, and introduces a perturbed MIMIC-CXR dataset for metric sensitivity.
Problem
Existing evaluation metrics for generated radiology reports either measure only textual overlap or only a limited set of pathologies (CE metrics), and correlate poorly with radiologist judgment.
Approach
Score report quality using the joint image-report embedding space: the area of the triangle formed by the image, generated report and reference report embeddings (lower area means higher similarity), normalized by a constant C, instead of comparing two texts alone.
Key results
- VLScore reaches Kendall tau 0.718 with radiologist rankings on ReXVal, versus 0.515 for RadGraph F1 and 0.345 for BLEU-4.
- On the perturbed set, VLScore shows the largest score drop for pathology removal (delta 0.15) versus insignificant sentence removal.
- Image-centered cosine similarity (0.678) outperforms plain cosine similarity (0.555) in the ablation; LIMITR gives the best tau among embedding models tested.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VLScore instead of RadGraph F1 for metric evaluation | medical report generation evaluation · ReXVal | Kendall τ with radiologist judgments: 0.515 -> 0.718 | Helped |
| Remove a pathology sentence rather than an insignificant sentence | VLScore · Perturbed dataset | similarity score difference: 0.84 -> 0.69; 0.15 | Helped |
| Change a pathology location word rather than a non-informative word | VLScore · Perturbed dataset | similarity score difference: 0.91 -> 0.72; 0.19 | Helped |
| Change a pathology severity word rather than a non-informative word | VLScore · Perturbed dataset | similarity score difference: 0.91 -> 0.79; 0.12 | Helped |
| Use triangle area instead of report-report cosine similarity (ablation) | VLScore · ReXVal | Kendall τ: 0.555 -> 0.718 | Helped |
| Use triangle area instead of minimal bounding sphere (ablation) | VLScore · ReXVal | Kendall τ: 0.525 -> 0.718 | Helped |
| Use triangle area instead of image-centered cosine similarity (ablation) | VLScore · ReXVal | Kendall τ: 0.678 -> 0.718 | Helped |
| Use image-centered cosine similarity instead of report-report cosine similarity (ablation) | embedding similarity measurement · ReXVal | Kendall τ: 0.555 -> 0.678 | Helped |
| Use LIMITR embeddings instead of ConVIRT, BioViL, MedCLIP, or GLoRIA (ablation) | VLScore · ReXVal | Kendall τ: 0.526; 0.580; 0.636; 0.698 -> 0.718 | Helped |
| Use VLScore rather than BLEU-4 on equivalent normal reports with standardized wording | medical report evaluation metrics · Perturbed dataset, normal reports | average similarity score: 0.15 -> 0.85 | Helped |
Surprising findings
- Traditional NLG metrics and CE metrics show near-identical scores for significant and insignificant perturbations, and location/severity word changes often yield scores near 1 for most baselines.
- BLEU-4 and RadGraph F1 score no-finding report pairs low (0.15 and 0.23), and CheXbert averages only 0.63 on them despite expectations.
Limitations
- Normalization constant C is set from a representative dataset and must be re-determined for other domains or imaging sources.
- The triangle area is unbounded, so normalization to [0,1] depends on C.
- Evaluation on ReXVal relies on only 200 report pairs annotated by radiologists.
Experimental setup
Datasets: MIMIC-CXR ReXVal
Benchmarks: ReXVal perturbed MIMIC-CXR dataset
Metrics: Kendall tau score deltas under perturbations
Baselines: BLEU METEOR ROUGE-L BERTScore CheXpert CheXbert RadGraph F1 ConVIRT BioViL MedCLIP GLoRIA
Builds on: LIMITR RadGraph F1 BERTScore CheXbert
Closest papers by meaning
- DART: Disease-aware Image-Text Alignment and Self-correcting Re-alignment for Trustworthy Radiology Report Generation (CVPR 2025 · similarity 0.89)
- Reporting Practice Matters: The Impact of Reference Choice on Chest X-ray Report Evaluation (NeurIPS 2026 · similarity 0.88)
- ReFINE: A Reward-Based Framework for Interpretable and Nuanced Evaluation of Radiology Report Generation (AAAI 2026 · similarity 0.88)
- DAMPER: A Dual-Stage Medical Report Generation Framework with Coarse-Grained MeSH Alignment and Fine-Grained Hypergraph Matching (AAAI 2025 · similarity 0.87)
- GEMA-Score: Granular Explainable Multi-Agent Scoring Framework for Radiology Report Evaluation (AAAI 2026 · similarity 0.86)
- Automatic Radiology Reports Generation via Memory Alignment Network (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.