Research map / Vision-language and multimodal / Visual question answering

MedVIGIL: Evaluating Trustworthy Medical VLMs Under Broken Visual Evidence

NeurIPS 2026 · Vision-language and multimodal › Visual question answering · approach: Vision-language model VQA evaluation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MedVIGIL, a clinician-authored 300-case medical VQA suite that tests whether vision-language models silently answer under perturbed visual or textual evidence, scored with a harmonic-mean composite (MCS) across 16 models and a radiologist baseline.

Problem

Medical VLMs are evaluated on intact image-question pairs, so they are not tested on whether they recognise when visual evidence has failed (false premises, ROI masking) and refuse instead of giving fluent non-refusal answers.

Approach

An evidence-contract framework with silent-failure rate under text- and image-side perturbations, clinician-adjudicated refusal options and risk tiers, ROI-conditioned grounding contrast, and a harmonic-mean composite (MCS) combining Capability, Safety, and Grounding axes.

Key results

• Independent radiologist R4 scores MCS 83.3 at 5.8% trap silent-failure rate, 14.1 points above the strongest audited model.
• Claude Opus 4.7 leads audited models with MCS 69.2 and the lowest aggregate SFR (22.0%).
• GPT-4o falls to MCS 44.1 because Safety is its bottleneck (SFR 53.0%); Gemini 3.1 Flash-Lite has the largest ROI contrast (VGR +49.5 pp).

Surprising findings

• Even the safest audited model, Claude Opus 4.7, retains 30.9% L3 and 28.4% L5 trap silent-failure rates.
• Knowledge-only accuracy is 90.5% to 99.6% across audited models, showing high language-prior exploitation that clean VQA accuracy does not reveal.

Limitations

• Single construction-blind radiologist provides the human reference baseline.
• ROI boxes are benchmark regions for binary masks rather than segmentation labels; triplet-coherence axis and several ablations deferred.
• Possible pretraining overlap with public sources cannot be ruled out; 300 cases are small.

Experimental setup

Datasets: VQA-RAD SLAKE ROCO MIMIC-CXR CheXpert

Benchmarks: MedVIGIL

Metrics: Overall accuracy SFR (silent-failure rate) SFR_w (risk-weighted SFR) VGR PR NEG SDR LPA MCS

Baselines: LLaVA-Med-7B HuatuoGPT-Vision-7B GPT-4o Claude Sonnet 4.6 Gemini 3-Flash Qwen3.5-9B

Builds on: HEAL-MedVQA VQA-RAD SLAKE ROCO MIMIC-CXR

Closest papers by meaning

• MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models (ICLR 2025 · similarity 0.92)
• Medical thinking with multiple images (ICLR 2026 · similarity 0.88)
• OpenMedReason: Scientific Reasoning Supervision for Medical Vision–Language Models (NeurIPS 2026 · similarity 0.88)
• CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models (NeurIPS 2024 · similarity 0.88)
• How Do Medical MLLMs Fail? A Study on Visual Grounding in Medical Images (ICLR 2026 · similarity 0.87)
• MedAtlas: Evaluating LLMs for Multi-Round, Multi-Task Medical Reasoning Across Diverse Imaging Modalities and Clinical Text (AAAI 2026 · similarity 0.87)