Research map / Vision-language and multimodal / Multimodal LLMs

VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds VLM-RobustBench, a benchmark of 133 corruption configurations (49 augmentations at graded/binary severities) used to evaluate 9–11 open VLMs from Qwen3-VL, InternVL3.5, Molmo2 and Gemma3 on MMBench and MMMU-Pro, finding spatial/resampling corruptions are most harmful.

Problem

VLM robustness to real-world image distortions is poorly characterised, and corruption benchmarks assume severity monotonicity, which may not hold for vision-language models.

Approach

A broad corruption suite (42 graded corruptions plus 7 binary transforms) applied to two complementary benchmarks, with drop-based tail-risk metrics, a visual-gain normalised relative corruption error (RCE), and a severity-monotonicity analysis.

Key results

• On MMBench, low-severity glass_blur reduces accuracy by about 8.1 pp on average across models, exceeding most high-severity corruptions.
• Upsample at high severity causes up to 34 pp drop (Molmo2-8B 33.9 pp worst case) on MMBench, the largest drops in the study.
• Vertical flip (10.3 pp) and color inversion (10.1 pp) are catastrophic on MMBench; mean RCE on MMBench rises from 1.6% (low) to 9.7% (high).

Surprising findings

• Low-severity glass blur causes larger drops than many high-severity photometric corruptions such as JPEG compression, so visual severity is a weak predictor of difficulty.
• Qwen3-VL-4B shows negative RCE (-6.8%) on MMMU-Pro, meaning corruptions on average slightly improve accuracy relative to clean images.

Limitations

• Evaluation uses stratified 20% subsamples of each benchmark (532 MMMU-Pro, 869 MMBench samples), which may limit precision.
• Only open-weights models from four families are evaluated; the paper's family-level claims rest on few checkpoints per family.
• Robustness-oriented training recommendations are not tested experimentally.

Experimental setup

Datasets: MMBench MMMU-Pro

Benchmarks: MMBench MMMU-Pro

Metrics: clean accuracy accuracy drop worst-case drop severe-failure rate worst@low benign@low visual gain relative corruption error mean corruption error flip rate

Baselines: Qwen3-VL-4B Qwen3-VL-8B Qwen3-VL-30B InternVL3.5-4B InternVL3.5-8B InternVL3.5-14B Molmo2-4B Molmo2-8B Gemma-3-12B

Builds on: ImageNet-C MMBench MMMU-Pro

Closest papers by meaning

• MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs (ICML 2026 · similarity 0.92)
• Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.91)
• Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.90)
• MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly (NeurIPS 2025 · similarity 0.90)
• Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.90)
• Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning (ICML 2024 · similarity 0.90)