Research map / Large language models / AI and society

Reporting Practice Matters: The Impact of Reference Choice on Chest X-ray Report Evaluation

NeurIPS 2026 · Large language models › AI and society · approach: AI for health and public decisions · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that RRG metrics are sensitive to radiologist reporting-practice variations in the reference, and introduces ReRef (o3-based generator-verifier) to rewrite references along a taxonomy, reordering model rankings on MIMIC-CXR and other CXR datasets.

Problem

Automated RRG evaluation compares generated reports to a single human reference, but radiologists vary in reporting style, detail and terminology, so metric scores and model rankings may reflect reference style rather than clinical accuracy.

Approach

A radiologist-informed taxonomy of reporting-practice variation (organization, completeness, granularity, language) and ReRef, an LLM generator-verifier loop that rewrites reference Findings/Impression along one axis while preserving clinical content, used to measure metric and ranking sensitivity.

Key results

• Radiologist annotations: 92.7% (445/480) of alternative references rated as realistic as the original, and 98.8% rated Equivalent or Minor Discrepancy for clinical equivalence.
• On MIMIC-CXR with RadCliQ-v1, Minimal findings-scope perturbation drops Libra from first to second while CheXOne rises from third to first.
• Libra significantly underperforms CheXOne in 6 of 12 perturbation settings under RadCliQ-v1 relative to significantly outperforming in the original setting; CRIMSON is most stable (mean IQR 0.083).

Surprising findings

• CRIMSON is the most stable metric overall, while RadGraph-F1 and RadCliQ-v1 show the highest sensitivity to reporting practice.
• Model rankings under RadCliQ-v1 change for pairs whose bootstrap CI-based comparisons also change, showing ranking conclusions are not robust to reference style.

Limitations

• ReRef relies on the proprietary, closed-source o3 model, which may hinder reproducibility.
• Rewrites may contain hallucinations or subtle clinical deviations despite radiologist validation.
• LLM judge metrics (CRIMSON, GREEN) depend on the chosen LLM, so trends may change with other judges; evaluation is limited to Findings-only and single-reference, single-axis perturbations.

Experimental setup

Datasets: MIMIC-CXR CheXpert Plus IU X-ray ReXGradient-160K

Benchmarks: MIMIC-CXR CheXpert Plus IU X-ray ReXGradient-160K

Metrics: RadCliQ-v1 CRIMSON GREEN RadGraph-F1 RaTEScore CheXbert-F1 BLEU-2 ROUGE-L BERTScore model ranking reversal rate

Baselines: MAIRA-2 MedGemma-4B CheXOne MedVersa CheXagent-2-3B Med-CXRGen Libra LLaVA-Rad CheX-MIMIC

Builds on: RadCliQ GREEN CRIMSON RadGraph-F1 ReXrank

Closest papers by meaning

• ReFINE: A Reward-Based Framework for Interpretable and Nuanced Evaluation of Radiology Report Generation (AAAI 2026 · similarity 0.90)
• Image-aware Evaluation of Generated Medical Reports (NeurIPS 2024 · similarity 0.88)
• LLM-RG4: Flexible and Factual Radiology Report Generation Across Diverse Input Contexts (AAAI 2025 · similarity 0.87)
• From Complexity to Clarity: Transforming Chest X-ray Reports with Chained Prompting (Student Abstract) (AAAI 2025 · similarity 0.86)
• GEMA-Score: Granular Explainable Multi-Agent Scoring Framework for Radiology Report Evaluation (AAAI 2026 · similarity 0.85)
• DART: Disease-aware Image-Text Alignment and Self-correcting Re-alignment for Trustworthy Radiology Report Generation (CVPR 2025 · similarity 0.85)