Research map / Vision-language and multimodal / Visual question answering
Is Your (Reasoning) Multimodal Language Model Vulnerable Toward Distractions?
Ming Liu, Hao Chen, Jindong Wang, Liwen Wang, Jingchen Sun, Wensheng Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| insert irrelevant hint text (Add Hint) | InternVL2 (2B) · I-ScienceQA | exact match: 91.40 -> 82.35 | Hurt |
| insert irrelevant hint text | VL-Rethinker (7B) · I-ScienceQA Add Hints | exact match: 88.05 -> 82.30 | Hurt |
| add an unrelated image | GPT-4o · I-ScienceQA Add Image | exact match: 93.50 -> 93.00 | No clear effect |
| add an unrelated image | GPT-o4 mini · I-ScienceQA Add Image | exact match: 98.15 -> 98.75 | No clear effect |
| insert a distracting image alongside the original | VL-Rethinker (7B) · I-ScienceQA Insert Image | exact match: 82.75 -> 88.90 | Improved |
| insert a distracting image alongside the original | InternVL2 (1B) · I-ScienceQA Add Image | exact match: 85.60 -> 79.70 | Hurt |
| use textual rather than visual distractions | VLMs in general · I-ScienceQA | exact match drop | Hurt |
| scale InternVL2 from 8B to 26B | InternVL2 · I-ScienceQA Add Image | exact match: 94.45 -> 93.40 | No clear effect |
| scale LLaVA from 7B to 34B | LLaVA · I-ScienceQA Add Hint | exact match: 63.80 -> 82.65 | Improved |
| add prompt telling model to ignore irrelevant information (ablation) | Qwen2-VL (2B) · I-ScienceQA Add Image | exact match: 63.30 -> 73.80 | Improved |
| add prompt telling model to ignore irrelevant information (ablation) | Qwen2-VL (7B) · I-ScienceQA Add Image | exact match: 83.10 -> 81.35 | Hurt |
| swap CLIP for robust-CLIP vision encoder (ablation) | LLaVA-7B · I-ScienceQA | exact match: +0.85 on Add Image | No clear effect |
Closest papers by meaning
- Adaptive Distraction: Probing LLM Contextual Robustness with Automated Tree Search (NeurIPS 2025 · similarity 0.90)
- Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs (ICLR 2026 · similarity 0.89)
- Caption This, Reason That: VLMs Caught in the Middle (NeurIPS 2025 · similarity 0.89)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.89)
- Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectly (CVPR 2025 · similarity 0.88)
- When to Think and When to Look: Uncertainty-Guided Lookback (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
