Research map / Vision-language and multimodal / Multimodal LLMs

CRAFT: Causal Responsibility and Failure Tracing in Medical Vision Language Models

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Bias-aware multi-image robustness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Uses causal head excision with deconfounded interchange interventions to localize arbitration (textual override) and brake (over-commitment) attention heads in medical VLMs, and training-free excision reduces conflict following and restores abstention.

Problem

Medical vision-language models can be hijacked by misleading text despite correct visual evidence (arbitration failure) and commit to diagnoses under degraded visual evidence instead of abstaining (brake failure); existing mechanistic work covers only text-only LLMs and binary knowledge conflicts.

Approach

Localize failure-specific heads in two stages: a layer-wise conflict score narrows the search to a critical layer band, then heads are selected by a dual criterion (causal effect on the failure via excision vs. collateral damage on clean inputs), with clean-state interchange used to block back-door confounding; excising the selected heads is training-free.

Key results

• On VQA-RAD (Before Q) with Hulu-Med 4B, excising the selected arbitration heads reduces conflict following rate (CFR) from 44.62% to 14.36% with C2W of 0.51%.
• On SLAKE image conflict, excising brake heads raises unknown rate on Hulu-Med 4B from 37.88% to 65.15% with zero U2O.
• Clean accuracy drops by 2.2 to 3.6 points for arbitration interventions and 6.8 to 11.1 points for brake interventions across backbones, which is larger than the drop reported for the arbitration case.

Surprising findings

• Random head ablation sometimes increases conflict following (e.g., InternVL3.5 Before A) and can reduce unknown rate on SLAKE for InternVL3.5, showing indiscriminate ablation can worsen failures.
• Brake interventions incur larger clean accuracy losses than arbitration interventions, which the authors attribute to direct suppression of answer commitment.

Limitations

• Clean accuracy cost of brake intervention is substantial (about 7-11 points), as the paper acknowledges.
• Clinically valid abstention thresholds require deployment-specific human validation, per the paper.
• Evaluated on a limited set of medical VQA benchmarks and mostly 4B-14B VLMs with binary and synthetic conflict construction.

Experimental setup

Datasets: VQA-RAD SLAKE HealMed-VQA ConflictMedQA PubMedQA

Benchmarks: VQA-RAD SLAKE HealMed-VQA ConflictMedQA PubMedQA

Metrics: CFR Resist C2W W2C UR O2U U2O

Baselines: Random heads CCD MARINE ITI HALP Conflict SFT Preference Tuning

Builds on: activation patching Tuned Lens contrastive decoding

Closest papers by meaning

• The Abstraction Gap in Vision-Language Causal Reasoning (ICML 2026 · similarity 0.89)
• Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models (NeurIPS 2026 · similarity 0.88)
• Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models (ICLR 2026 · similarity 0.88)
• StemBind: When MLLMs Get Lost Between Rules and Instances in Abstract Visual Reasoning (NeurIPS 2026 · similarity 0.88)
• Diagnosing and Repairing Unsafe Channels in Vision-Language Models via Causal Discovery and Dual-Modal Safety Subspace Projection (CVPR 2026 · similarity 0.88)
• VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety (ICLR 2026 · similarity 0.87)