Research map / Vision-language and multimodal / Multimodal LLMs

MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Bias-aware multi-image robustness · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MVI-Bench, a paired-image VQA benchmark (624 pairs, 1,248 instances) across six misleading visual categories, plus the MVI-Sensitivity metric, and evaluates 18 LVLMs against human performance.

Problem

Existing LVLM robustness benchmarks focus on hallucination, adversarial perturbations, or misleading text, leaving robustness to naturally occurring misleading visual inputs underexplored.

Approach

Organize misleading visual cues into a three-level taxonomy (concept, attribute, relationship) with six categories, and pair each normal image with a near-identical misleading counterpart sharing the same MCQ, measuring robustness as the relative accuracy drop (MVI-Sensitivity).

Key results

• Best open-source model Qwen2-VL-72B reaches 32.38% MVI-Sensitivity, while the best closed-source GPT-5-Chat achieves 63.78% Acc_m overall and GPT-4.1 has the lowest closed-source sensitivity at 20.80%.
• Human participants achieve 98.24% misleading-image accuracy with 1.63% MVI-Sensitivity.
• Adding GPT-4.1-generated captions raises Qwen2.5-VL-7B overall misleading accuracy from 45.99% to 53.85%.

Surprising findings

• Long-thinking variants of open-source models (SAIL-VL2, Qwen2-VL, QVQ) show lower misleading-image accuracy than their non-thinking counterparts, attributed to visual forgetting.
• About 4% of cases where models answer misleading images correctly but fail normal ones are driven by spurious correlations with visual artifacts.

Limitations

• Spurious-correlation analysis relies on the paired design and does not directly generalize to other datasets.
• Human evaluation uses only four participants with a 5-second time limit.
• Benchmark size is modest (1,248 instances) and dataset is reduced by filtering with two LVLMs, which may bias discriminability.

Experimental setup

Datasets: MVI-Bench

Benchmarks: MVI-Bench

Metrics: accuracy MVI-Sensitivity

Baselines: GPT-4o GPT-4.1 Claude-3.7-Sonnet Gemini-2.5-Flash Gemini-2.5-Pro GPT-5-Chat Qwen2.5-VL-72B InternVL3-78B Qwen2-VL-72B SAIL-VL2-8B LLaVA-OneVision-7B Molmo-7B

Builds on: MAD-Bench IllusionVQA O-Bench

Closest papers by meaning

• VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models (ICML 2026 · similarity 0.92)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.92)
• Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision (ICLR 2024 · similarity 0.91)
• SynDORBench: Evaluating LVLM Perceptual Robustness Under Physically Constrained Visibility Conditions (NeurIPS 2026 · similarity 0.91)
• Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework (CVPR 2026 · similarity 0.91)
• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.90)