Research map / Vision-language and multimodal / Multimodal LLMs

Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectly

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MMVU, a manually built benchmark of paired positive/negative questions showing MLLMs answer incorrectly despite understanding images, plus a GPT-4o-generated paired training set and two prompting strategies (CGR, VAR) to reduce this.

Problem

MLLMs often give wrong answers to misleading (negative) questions even when they correctly answer direct questions about the same image, so standard benchmarks conflate lack of visual understanding with lack of robustness to misleading questions.

Approach

Pair each positive question (directly about visible content) with a negative misleading question on the same image, measure Misleading Rate and Robustness Accuracy, and attribute failures to dataset bias toward positive questions and low attention to visual tokens; mitigate with paired training data and content-guided/visual-attention prompting.

Key results

• Ovis1.6-Gemma2-9B reaches 66.74% RA on MMVU, the best open-source model; GPT-4o has the best closed-source RA at 65.06% but a 19.53% MR.
• LLaVA-OneVision-7B + MMVU-Train improves RA from 60.58% to 63.49% and lowers MR from 27.77% to 21.03%.
• VILA1.5-13B + MMVU-Train improves RA from 25.76% to 37.74%, and MR drops from 62.30% to 45.02%.

Measured effects reported in the paper

Surprising findings

• Simply adding a detailed instruction to the system prompt reduced RA for LLaVA 1.6 7B (47.14% to 43.34%).
• Training with only negative samples prevents the model from producing negative answers, so balanced positive/negative pairs are needed.

Limitations

• The MMVU test set is manually constructed with only 1,786 questions, a relatively small benchmark.
• Attention and logit analyses are limited to a few open-source LLaVA-style models.
• The best prompting combination (VAR+CGR) does not improve positive-question accuracy over CGR alone, and gains from the prompting strategies are small (under 1 point RA).

Experimental setup

Datasets: MMVU-Train LLaVA 158k COCO Visual Genome OCR-VQA VQA-v2 GQA OK-VQA A-OKVQA RefCOCO WizardLM Bunny-pretrain-LAION-2M

Benchmarks: MMVU MME MMBench SEED-Bench MMMU VQA-v2 GQA POPE

Metrics: Response Accuracy (RA) Misleading Rate (MR) accuracy

Baselines: GPT-4o Qwen-VL-max Claude3.5-Sonnet Gemini-1.5-Pro Ovis1.6-Gemma2-9B Llama3.2-90B-Vision-Instruct MiniCPM-V-2.6 LLaVA-OneVision-7B InternVL2-8B VILA1.5-13B Phi3.5-Vision-Instruct LLaVA-v1.6-Vicuna-7B

Builds on: LLaVA LRV Bunny LoRA

Closest papers by meaning

• MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs (ICLR 2025 · similarity 0.90)
• Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs (ICLR 2026 · similarity 0.90)
• LOVA3: Learning to Visual Question Answering, Asking and Assessment (NeurIPS 2024 · similarity 0.89)
• Notes-guided MLLM Reasoning: Enhancing MLLM with Knowledge and Visual Notes for Visual Question Answering (CVPR 2025 · similarity 0.88)
• Right this way: Can VLMs Guide Us to See More to Answer Questions? (NeurIPS 2024 · similarity 0.88)
• FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answering (NeurIPS 2025 · similarity 0.88)