Research map / Vision-language and multimodal / Multimodal LLMs

ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ZeroBench, a 100-question adversarially filtered visual reasoning benchmark that was 0% pass@1 and pass^5 for all frontier LMMs at release, and tracks a year of progress across 46 LMMs.

Problem

Existing visual benchmarks are saturating quickly, leaving little headroom, so there is a need for lightweight, hard multi-step visual reasoning benchmarks that remain informative for longer.

Approach

Manually written multi-step visual questions with exact-match numeric answers, filtered adversarially so no model answers any main question correctly at release, paired with reviewed subquestions for finer-grained differentiation and a year of longitudinal tracking with community red teaming.

Key results

• At release (Jan 2025), all 20 evaluated LMMs scored 0% pass@1 and pass^5 on the 100 main questions; top pass@5 was 5% (Gemini 2 Flash Thinking).
• Over the following year, SotA pass@5 rose from 5% to 19% (Gemini 3 Pro) and SotA pass^5 from 0% to 6% (GPT-5.2).
• Human baseline averaged 29.5±27.1% on the main questions; error analysis found Counting (171), Recognition (80) and Grounding (68) as dominant failure categories.

Surprising findings

• Reasoning models (o1, QVQ) used far more tokens but did not outperform traditional models on subquestions.
• Visual hints substantially improved performance, indicating visual extraction rather than reasoning is the main bottleneck.

Limitations

• Small benchmark of 100 questions, so scores are noisy and sensitive to individual questions.
• Many answers are numeric and partial credit is not assigned; difficulty of verifying answers for questions near human limits.
• Some evaluated models accessed via APIs and version/access drift affects reproducibility.

Experimental setup

Datasets: ZeroBench v1 (100 questions) ZeroBench v2 (100 questions, 334 subquestions)

Benchmarks: ZeroBench

Metrics: pass@1 pass@5 pass^5 subquestion accuracy standard error (CLT)

Baselines: o1 o1 pro Gemini 2 Flash Thinking QVQ GPT-4o Claude 3.5 Sonnet v2 Gemini 2 Flash Pixtral-Large Qwen2-VL-72B-Instruct NVLM-D-72B

Builds on: SEED-Bench MMBench GPQA HumanEval-V VibeEval

Closest papers by meaning

• JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images (NeurIPS 2024 · similarity 0.89)
• MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks (ICLR 2025 · similarity 0.89)
• MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models (ICML 2026 · similarity 0.89)
• Latent Implicit Visual Reasoning (CVPR 2026 · similarity 0.88)
• MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI (ICML 2024 · similarity 0.88)
• No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance (NeurIPS 2024 · similarity 0.88)