Research map / Vision-language and multimodal / Multimodal LLMs

Will Multimodal Models Be Dazzled by Multi-Image Visual Puzzles?

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MIRACLE, a 4,000-question benchmark of multi-image (2-14 images) visual reasoning puzzles and subject questions, and evaluates leading proprietary and open-source MLLMs, finding that even Gemini-2.5-Pro reaches only 55.91%.

Problem

Existing multi-image benchmarks use few images and shallow inter-image relations, so they do not adequately test complex cross-image reasoning in MLLMs.

Approach

A benchmark of cross-image reasoning tasks (puzzles, games, sorting/jigsaw, subject questions) with 2-14 images per question, a fine-grained taxonomy of 15 subtasks, and a partial-credit adjacency-based scoring rule for sorting tasks.

Key results

• Gemini-2.5-Pro achieves the top overall accuracy of 55.91% on MIRACLE, with OpenAI o3 at 51.86%.
• Best open-source model Qwen2.5-VL-72B-Instruct reaches 32.15% overall accuracy.
• On ultra multi-image questions (>7 images), Gemini-2.5-Pro gets 42.93% while most open-source models fall below 20%.

Surprising findings

• Thinking-mode models consistently outperform regular counterparts within model families.
• Models perform much better on subject-type questions than on puzzle-like jigsaw and number-constraint reasoning tasks.

Limitations

• Evaluation covers only a selected set of MLLMs and default settings, with no training or method proposed.
• Sorting tasks rely on a custom adjacency-based partial-credit rule whose relation to human judgment is not validated.

Experimental setup

Datasets: MIRACLE

Benchmarks: MIRACLE

Metrics: accuracy adjacency-based partial score for sorting tasks

Baselines: GPT-4o GPT-4.1 OpenAI o3 OpenAI o4-mini Claude-3.7V-Sonnet Claude-4-Sonnet Seed-1.6 Seed-1.6-Thinking InternVL2.5-MPO InternVL3 Ovis2 QVQ-72B-Preview

Builds on: VLMEvalKit MMMU

Closest papers by meaning

• LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.91)
• MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning (ICLR 2026 · similarity 0.90)
• JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images (NeurIPS 2024 · similarity 0.90)
• MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.89)
• Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency (NeurIPS 2026 · similarity 0.89)
• VGR: Visual Grounded Reasoning (ICLR 2026 · similarity 0.89)