Research map / Multi-agent systems and games / LLM agents

GEMA-Score: Granular Explainable Multi-Agent Scoring Framework for Radiology Report Evaluation

AAAI 2026 · Multi-agent systems and games › LLM agents · approach: Agent data synthesis and behavior studies · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GEMA-Score, an LLM-based multi-agent framework that combines objective entity-level F1 (disease, location, severity, uncertainty) with subjective linguistic scoring to evaluate radiology reports with explanations, and distills it into an 8B open model.

Problem

Existing radiology report evaluation metrics (overlap, BERT-based, NER-F1, diagnostic classifiers, single-step LLM judges) miss fine-grained clinical details, are limited by fixed vocabularies, and lack interpretability, so they poorly reflect clinical reliability of generated reports.

Approach

A four-agent LLM workflow (entity extraction, objective clinical accuracy, subjective expressiveness, score evaluation) that parses reports into disease/location/severity/uncertainty tuples, computes harmonic-mean F1 for objective accuracy, penalizes linguistic errors for subjective quality, and outputs explanatory feedback; a LLaMA-3.1-8B distilled variant enables local deployment.

Key results

• On ReXVal, GEMA-Score (Distilled LLaMA-3.1-8B) reaches Kendall tau 0.69 (clinically significant errors, 0.678) and Spearman 0.845, exceeding GREEN (Spearman 0.811 for significant errors).
• On RadEvalX, GEMA-Score (Chat-GPT-o1) achieves Kendall 0.466 and Spearman 0.597 for significant errors, versus 0.347 Kendall for GREEN.
• On CT-RATE (60 studies), GEMA-Score (Chat-GPT-o1) reaches Kendall 0.755 and Spearman 0.794 against expert annotations, compared with 0.310 Kendall for GREEN.

Surprising findings

• The distilled LLaMA-3.1-8B version matches or exceeds several proprietary LLM backbones on ReXVal correlation, despite its small size.
• Several standard metrics (e.g., MOVERScore on ReXVal, BERTScore on RadEvalX) show negative or non-significant correlations with expert judgments.

Limitations

• Subjective score uses discrete rounding and agent-defined weights and penalty set heuristically (lambda=0.05, alpha=0.8).
• Human correlation evaluated on small expert-annotated sets (50 ReXVal studies, 100 RadEvalX studies, 60 CT studies), and the RadEvalX insignificant-error correlations are weak and not significant.
• Relies heavily on proprietary LLM APIs for the main results, with reproducibility depending on model access.

Experimental setup

Datasets: MIMIC-CXR ReXVal RadEvalX CT-RATE

Benchmarks: ReXVal RadEvalX CT-RATE

Metrics: Kendall's tau Spearman correlation Pearson correlation F1 score

Baselines: BLEU-1 ROUGE-L METEOR BERTScore BLEURT MOVERScore RadGraphF1 RaTEScore GREEN CheXbert RadBERT

Builds on: GREEN RadGraphF1 RadCliQ AutoGen MAD

Closest papers by meaning

• ReFINE: A Reward-Based Framework for Interpretable and Nuanced Evaluation of Radiology Report Generation (AAAI 2026 · similarity 0.88)
• Image-aware Evaluation of Generated Medical Reports (NeurIPS 2024 · similarity 0.86)
• Reporting Practice Matters: The Impact of Reference Choice on Chest X-ray Report Evaluation (NeurIPS 2026 · similarity 0.85)
• LLM-RG4: Flexible and Factual Radiology Report Generation Across Diverse Input Contexts (AAAI 2025 · similarity 0.85)
• MedAgent-Pro: Towards Evidence-based Multi-modal Medical Diagnosis via Reasoning Agentic Workflow (ICLR 2026 · similarity 0.85)
• DDx-TRACE: A Benchmark for Medical Diagnostic Trajectories in VLMs (NeurIPS 2026 · similarity 0.84)