Research map / Vision-language and multimodal / Multimodal LLMs

Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds MedGround-Bench, a causally filtered CXR-VQA attribution benchmark, shows 11 attribution methods fail on it, and proposes MedFocus, a concept-based causal attribution method using unbalanced optimal transport and concept intervention.

Problem

Whether visual attribution explanations of large vision language models on chest X-ray questions reflect the visual evidence actually driving predictions, given that ground-truth annotations of internal model reasoning are unavailable.

Approach

Uses counterfactual editing (RadEdit foreground/background edits) to keep only expert-annotated regions that causally drive LVLM answers as verified ground truth, and attributes predictions by intervening on UOT-localized anatomical concepts and measuring token log-probability drops.

Key results

• MedFocus reaches 54.24% IoU on ImaGenome direct mode, versus 39.24% for the best baseline (Gradient-weighted Attention).
• On MedGround-Bench-Reason, MedFocus attains 52.95% IoU on ImaGenome while GradCAM++ drops from 30.54% to 23.70%.
• Ablation: UOT + MedSAM with bounding-box zero masking gives 37.82% IoU, above end-to-end SAM3 (30.89%) and MedSAM3 (33.52%).

Surprising findings

• Reasoning mode shows substantially lower failure rates than direct-answer mode for all models, often approaching zero on the final grounded group.
• Gradient-weighted Attention achieves near-perfect recall (99.90% on ImaGenome) but very low precision, indicating overly broad maps.

Limitations

• Evaluation limited to chest X-ray and binary yes/no VQA on correctly answered samples.
• Concept vocabulary limited to 11 ImaGenome anatomical regions, so findings like small nodules are not covered.
• Benchmark relies on RadEdit counterfactuals, which may introduce editing artifacts.

Experimental setup

Datasets: ImaGenome VinDR-CXR PadChest-GR MIMIC-CXR-JPG

Benchmarks: MedGround-Bench-Direct MedGround-Bench-Reason

Metrics: IoU F1 precision recall

Baselines: Attention Head Attention Rollout LRP Gradient-weighted Attention GradCAM GradCAM++ Integrated Gradients Prompting Prompting + MedSAM Occlusion RISE

Builds on: Sinkhorn algorithm MedSAM RadEdit GradCAM Occlusion

Closest papers by meaning

• CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rays (NeurIPS 2025 · similarity 0.90)
• BenchX: A Unified Benchmark Framework for Medical Vision-Language Pretraining on Chest X-Rays (NeurIPS 2024 · similarity 0.89)
• AOR: Anatomical Ontology-Guided Reasoning for Medical Large Multimodal Model in Chest X-Ray Interpretation (NeurIPS 2025 · similarity 0.88)
• CRAFT: Causal Responsibility and Failure Tracing in Medical Vision Language Models (NeurIPS 2026 · similarity 0.88)
• Medical thinking with multiple images (ICLR 2026 · similarity 0.87)
• M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding (ICLR 2026 · similarity 0.87)