Research map / Large language models / Language, safety and interpretability

FactCheXcker: Mitigating Measurement Hallucinations in Chest X-ray Report Generation Models

CVPR 2025 · Large language models › Language, safety and interpretability · approach: Data and evaluation methodology · 8 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FactCheXcker, a modular query-code-update pipeline that uses GPT-4o mini to generate measurement queries and code over a tool API to correct endotracheal tube measurements in chest X-ray reports from 11 report generation models.

Problem

Medical vision-language report generation models hallucinate quantitative measurements (e.g., endotracheal tube-to-carina distance), which undermines clinical reliability.

Approach

A training-free, tool-based post-hoc pipeline: an LLM generates measurement queries from the report, writes Python code calling specialized vision tools (exists, find, distance, etc.), and a report updater injects verified measurements and placement judgments back into the report, without retraining the original model.

Key results

• Across 11 report generation models, FactCheXcker improves ETT measurement MAE by an average of 135.0% (average MAE 7.25 to 2.54 and composite improvement of 186%).
• ETT presence precision rises on average from 0.49 to 0.65, with LLM-CXR improving most (0.08 to 0.58).
• Average ETT placement precision increases from 0.84 to 0.94; GPT4V failure rate drops from 77.5% to 22.5%.

Measured effects reported in the paper

Surprising findings

• RadFM's measurement MAE worsened after applying FactCheXcker (-30%), showing the pipeline can occasionally make measurements worse.
• Running FactCheXcker on every image regardless of report mentions improves false negatives but significantly drops precision.

Limitations

• Evaluated only on endotracheal tube placement; other measurement types (lesions, nodules) not addressed.
• Placement uses a strict rule-based threshold (3-7 cm) because GPT-4o mini numeric reasoning is insufficient.
• Failure cases from tube misidentification and obscured carina landmarks; false positive rate depends on exists module.

Experimental setup

Datasets: MIMIC-CXR 2.0.0 CheXpert Plus private dataset (22 hospitals)

Benchmarks: MIMIC-CXR test set (2288 studies, 182 with ETT mentions)

Metrics: precision MAE composite (MAE x F1) F1 balanced accuracy failure rate (error > 1.5 cm) placement precision

Baselines: CheXagent CheXpertPlus GPT-4V LLM-CXR MAIRA-2 MedVersa RadFM RaDialog RGRG VLCI Cvt2distilgpt2

Builds on: ViperGPT HuggingGPT CarinaNet ResNet-50

Closest papers by meaning

• Reporting Practice Matters: The Impact of Reference Choice on Chest X-ray Report Evaluation (NeurIPS 2026 · similarity 0.84)
• CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rays (NeurIPS 2025 · similarity 0.84)
• LLM-RG4: Flexible and Factual Radiology Report Generation Across Diverse Input Contexts (AAAI 2025 · similarity 0.82)
• ReFINE: A Reward-Based Framework for Interpretable and Nuanced Evaluation of Radiology Report Generation (AAAI 2026 · similarity 0.82)
• CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Dataset (CVPR 2025 · similarity 0.82)
• DAMPER: A Dual-Stage Medical Report Generation Framework with Coarse-Grained MeSH Alignment and Fine-Grained Hypergraph Matching (AAAI 2025 · similarity 0.82)