Research map / Vision-language and multimodal / Multimodal LLMs
VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VOILA, a large-scale open-ended benchmark where MLLMs must generate an image completing a visual analogy across image pairs, with a multi-step least-to-most evaluation showing large gaps to humans.
Problem
Existing multimodal reasoning benchmarks are mostly multiple-choice and do not test whether MLLMs can perceive, map and transfer abstract relations across multiple images to generate a solution.
Approach
Generates dynamic visual analogy questions (A:A'::B:B') from SDXL-generated images combined by stable/change/arithmetic/distraction rules over subject number, type and action, and scores each step of a least-to-most pipeline plus the generated image.
Key results
- Best applying-relationship accuracy on VOILA-WD is 13.16% (LLaMa 3.2, image collage) versus 71.36% for human participants.
- GPT-4o reaches 29.03% applying-relationship accuracy on VOILA-ND with three separate images under L2M, the best among models there.
- Using separate sequential images instead of a collage improves GPT-4o accuracy on VOILA-ND from 19.43% to 29.03% under L2M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use L2M prompting instead of direct answering (ablation) | GPT-4o · VOILA-WD | applying relationship accuracy: 0.9 -> 6.44 | Helped |
| Use L2M prompting instead of direct answering (ablation) | GPT-4o · VOILA-ND | applying relationship accuracy: 16.94 -> 29.03 | Helped |
| Use three separate images instead of an image collage (ablation) | GPT-4o with L2M · VOILA-WD | applying relationship accuracy: 3.94 -> 6.44 | Helped |
| Use three separate images instead of an image collage (ablation) | GPT-4o with L2M · VOILA-ND | applying relationship accuracy: 19.43 -> 29.03 | Helped |
| Provide ground truth image descriptions at phase 2 (ablation) | GPT-4o · VOILA-WD | identifying relationships accuracy | Helped |
| Provide ground truth relationships at step 3 (ablation) | GPT-4o · VOILA-WD | applying relationships accuracy | Hurt |
| Use textual descriptions instead of visual input (ablation) | GPT-4o with direct answering · VOILA-ND | accuracy: 22% -> 49% | Helped |
| Use three sub-prompts instead of a single prompt (ablation) | GPT-4o · VOILA-WD | identifying relationships accuracy: 42.8% -> 42.3% | No clear effect |
| Use L2M instructions instead of CoT with two examples and rationales (ablation) | GPT-4o · VOILA-WD | applying relationship accuracy: 5.96% -> 6.44% | Helped |
| Provide two examples and a property list for direct answering (ablation) | GPT-4o · VOILA-WD | applying relationship accuracy: 6.44% -> 6.8% | Helped |
| Use three sequential images instead of an image collage (ablation) | LLaVA-OneVision with AnyRes · VOILA-ND | image description accuracy: 53% -> 57.5% | Helped |
| Add the Distraction rule to the benchmark (ablation) | MLLMs · VOILA-WD versus VOILA-ND | applying relationship accuracy | Hurt |
Surprising findings
- Providing ground-truth relationships to GPT-4o only raises applying-relationship accuracy to 17%, suggesting difficulty applying known relations to new visuals.
- Image collage input degrades performance by roughly 40% compared to sequential images, attributed to resolution limits.
Limitations
- Relies on SDXL-generated images that required manual filtering, limiting visual diversity.
- Evaluation of steps relies on GPT-4o as judge with up to ~10% per-step error rate.
- Only 14 subjects, 4 numbers and 13 actions, limiting coverage of abstract relations.
Experimental setup
Datasets: VOILA-WD VOILA-ND
Benchmarks: VOILA-WD VOILA-ND
Metrics: step-wise accuracy (describing images, identifying relations
Baselines: GPT-4o Emu2-37B Qwen2-VL-7B-Instruct CogVLM2-19B SEED-LLaMA-8B Llama 3.2-11B MolmoE-7B LLaVA-OneVision
Builds on: least-to-most prompting Raven's Progressive Matrices SDXL
Closest papers by meaning
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.92)
- VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning (ICLR 2026 · similarity 0.92)
- Reasoning Limitations of Multimodal Large Language Models. A case study of Bongard Problems (ICML 2025 · similarity 0.91)
- Look-Back: Implicit Visual Re-focusing in MLLM Reasoning (AAAI 2026 · similarity 0.91)
- VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs (CVPR 2026 · similarity 0.91)
- VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models (ICLR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.