Research map / Vision-language and multimodal / Multimodal LLMs

ReFINE: A Reward-Based Framework for Interpretable and Nuanced Evaluation of Radiology Report Generation

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains a Llama-3-based multi-head reward model with a margin-based reward enforcement loss on GPT-4-generated scored radiology reports to produce interpretable sub-scores whose sum is a human-correlated radiology report evaluation metric (ReFINE).

Problem

Automated radiology report generation evaluation: existing metrics (BLEU, BERTScore, CheXbert/RadGraph, RadCliQ) correlate poorly with radiologist judgments and give no per-criterion breakdown, while GPT-4 judging raises privacy and cost concerns.

Approach

Generate GPT-4 scored report pairs from the same ground-truth report under two scoring systems, pair accepted/rejected reports with per-criterion score margins, and train a LoRA-tuned LLM with multiple reward heads whose sum is the final score, using a margin-based ranking loss on both sub-scores and totals.

Key results

• On ReXVal, ReFINE reaches Kendall's Tau 0.751 and Spearman 0.910, above GREEN (0.640 Kendall) and RadCliQ-v1 (0.631 Kendall).
• On Rad-100, ReFINE achieves Kendall's Tau 0.23 and Spearman 0.29 (p=0.003), while all NLG and clinical baselines are non-significant (p>0.1).
• Ablation shows dropping either L_tot or L_ind lowers ReXVal Kendall's Tau from 0.751 to 0.740 and 0.738 respectively.

Surprising findings

• Sub-criterion correlation for incorrect location or position of a finding is low (Kendall's Tau 0.246), suggesting location details are hard to capture.
• Rad-100 sub-criterion accuracies vary widely, with completeness at 0.38 and clinical history at 0.41, below chance for binary labels.

Limitations

• Explainability limited to numeric sub-scores without free-text rationales.
• Human-evaluation test sets are small (200 ReXVal pairs; 100 Rad-100 reports).
• Rad-100 correlations are modest (Kendall 0.23) and the dataset is not released.

Experimental setup

Datasets: MIMIC-CXR ReXVal

Benchmarks: ReXVal Rad-100

Metrics: Kendall's Tau Spearman correlation accuracy

Baselines: BLEU-4 ROUGE-L METEOR CIDEr BERTScore RadGraph F1 RadCliQ-v1 GREEN semb_score

Builds on: RadCliQ MRScore GREEN LoRA Llama 3

Closest papers by meaning

• Reporting Practice Matters: The Impact of Reference Choice on Chest X-ray Report Evaluation (NeurIPS 2026 · similarity 0.90)
• GEMA-Score: Granular Explainable Multi-Agent Scoring Framework for Radiology Report Evaluation (AAAI 2026 · similarity 0.88)
• Radiology Report Generation via Multi-objective Preference Optimization (AAAI 2025 · similarity 0.88)
• Image-aware Evaluation of Generated Medical Reports (NeurIPS 2024 · similarity 0.88)
• DART: Disease-aware Image-Text Alignment and Self-correcting Re-alignment for Trustworthy Radiology Report Generation (CVPR 2025 · similarity 0.86)
• LLM-RG4: Flexible and Factual Radiology Report Generation Across Diverse Input Contexts (AAAI 2025 · similarity 0.86)