Research map / Vision-language and multimodal / Multimodal LLMs
Do Vision & Language Decoders use Images and Text equally? How Self-consistent are their Explanations?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends MM-SHAP and CC-SHAP to vision-language decoders to measure modality reliance and self-consistency of answers vs. post-hoc/CoT explanations, and benchmarks VL decoders on VALSE.
Problem
It is unclear how much VLM decoders use image vs. text when generating answers versus explanations, and how self-consistent their natural language explanations are; VALSE had only been evaluated on encoders.
Approach
Adapts MM-SHAP (modality contribution via Shapley values) and CC-SHAP (answer-vs-explanation input-contribution agreement) to autoregressive VLM decoders, combines them with edit-based self-consistency tests, and compares modality reliance across answer and explanation generation.
Key results
- mPLUG-Owl3 achieves the best average pairwise accuracy on VALSE (88%), followed by LLaVA-NeXT-Mistral (85%), BakLLaVA (83%), and LLaVA-NeXT-Vicuna (79%).
- Text contributions (T-SHAP) for answers are 85% or higher across all four VL decoders, while image contributions rise significantly during explanation generation, and more so in CoT.
- BakLLaVA, LLaVA-NeXT-Mistral and LLaVA-NeXT-Vicuna show negative CC-SHAP post-hoc scores on multiple-choice VALSE, while mPLUG-Owl3 reaches about 0.12 average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Explanation generation (post-hoc) instead of answer generation | VL decoders (BakLLaVA, LLaVA-NeXT variants, mPLUG-Owl3) · VQA, GQA, GQA balanced | T-SHAP (text share): text share drops by 1 to 30 percentage points | Hurt |
| Chain-of-thought explanation instead of post-hoc explanation | VL decoders · VQA, GQA, GQA balanced | T-SHAP explanation (image reliance) | Hurt |
| Switching from post-hoc to CoT explanations | LLaVA-NeXT-Mistral · VQA | CC-SHAP: -0.07 to -0.09 | Hurt |
| Use mPLUG-Owl3 (newer, video-trained) instead of LLaVA-based VLMs | VL decoders self-consistency · VQA | CC-SHAP post-hoc: 0.43 vs 0.22 (BakLLaVA) and -0.07 (LLaVA-NeXT-Mistral) | Helped |
| Adding a biasing feature to the input (Biasing Features test) | LLaVA-NeXT-Vicuna · VQA | Biasing Features self-consistency (% deemed faithful) | Hurt |
| Pairwise multiple-choice on VALSE counting adversarial vs other instruments | VL decoders · VALSE counting adversarial | acc_r | Hurt |
Surprising findings
- Edit-based self-consistency tests give extreme 0% or 100% scores on generative tasks, mirroring inconsistency found for LLMs.
- Most tested VL decoders are less self-consistent than LLMs, with negative CC-SHAP scores on multiple-choice tasks.
Limitations
- Edit-based self-consistency tests are unreliable on generative tasks because semantic evaluation of outputs is needed.
- Only 100 samples per dataset used for MM-SHAP and self-consistency experiments due to compute cost.
- Edit-based tests were not run for mPLUG-Owl3; only open-source 7B models tested.
Experimental setup
Datasets: VQA GQA GQA balanced FoilIt MSCOCO VALSE
Benchmarks: VALSE VQA GQA GQA balanced FoilIt MSCOCO
Metrics: pairwise accuracy accuracy caption precision foil precision T-SHAP CC-SHAP edit-based test pass rate
Baselines: mPLUG-Owl3 LLaVA-NeXT-Mistral LLaVA-NeXT-Vicuna BakLLaVA VL encoders (VALSE results)
Builds on: MM-SHAP CC-SHAP VALSE counterfactual edits test corrupting CoT tests
Closest papers by meaning
- Towards Self-Refinement of Vision-Language Models with Triangular Consistency (NeurIPS 2025 · similarity 0.88)
- Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs? (ICML 2025 · similarity 0.88)
- Aligning What Vision-Language Models See and Perceive with Adaptive Information Flow (CVPR 2026 · similarity 0.87)
- Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs (NeurIPS 2025 · similarity 0.87)
- Caption This, Reason That: VLMs Caught in the Middle (NeurIPS 2025 · similarity 0.87)
- Probing Visual Language Priors in VLMs (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.