Research map / Vision-language and multimodal / Multimodal LLMs
MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MARVEL, a 770-puzzle multidimensional abstract visual reasoning benchmark with a hierarchical perception-plus-AVR evaluation, showing that nine MLLMs perform near random and about 40 points below humans.
Problem
Whether multimodal LLMs have abstract visual reasoning ability is unclear, since existing AVR benchmarks cover few patterns, input shapes, and task configurations and do not test perception underlying reasoning.
Approach
A multidimensional AVR benchmark spanning six core-knowledge-based patterns, geometric and abstract shapes, and five task configurations, paired with a hierarchical evaluation that adds perception questions (panel counting, fine-grained binary attribute questions) and group-based consistency accuracy to diagnose whether AVR accuracy is grounded in perception.
Key results
- All nine MLLMs perform near random on the AVR question (best Claude3 Opus at 28.83% vs. random 25%), against a human average of 68.86%.
- Few-shot chain-of-thought gives marginal gains for GPT-4V and does not help Claude3 Opus; GPT-4V rises from 22.34% zero-shot to 28.31% two-shot.
- Adding accurate text descriptions of panels raises AVR accuracy from 11.57% to 44.21% on average, with GPT-4V reaching 65.26%, close to humans.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add accurate text descriptions of puzzle panels | MLLMs on MARVEL subset · MARVEL subset (95 puzzles) | accuracy: 11.57% to 44.21% | Helped |
| Add accurate text descriptions of puzzle panels | GPT-4V · MARVEL subset (95 puzzles) | accuracy: 21.05 -> 65.26 | Helped |
| Add two same-pattern CoT demonstrations | GPT-4V · MARVEL | accuracy: 22.34 -> 28.31 | Helped |
| Add two same-pattern CoT demonstrations | Claude3 (Opus) · MARVEL | accuracy: 28.83 -> 23.64 | Hurt |
| Use a randomly selected different-pattern CoT demonstration instead of the same-pattern demonstration (ablation) | GPT-4V · MARVEL | accuracy: 27.31 -> 23.85 | Hurt |
| Use a randomly selected different-pattern CoT demonstration instead of the same-pattern demonstration (ablation) | Claude3 (Opus) · MARVEL | accuracy: 29.62 -> 25.00 | Hurt |
| Mix demonstrations from sequence and two-row task formats instead of using sequence-format demonstrations (ablation) | Gemini · MARVEL | accuracy: 30.48 -> 24.29 | Hurt |
| Mix demonstrations from sequence and two-row task formats instead of using sequence-format demonstrations (ablation) | Claude3 (Sonnet) · MARVEL | accuracy: 28.57 -> 23.81 | Hurt |
| Use few-shot prompting with CoT demonstrations | Closed-source MLLMs · MARVEL | accuracy | Mixed |
| Try prompt engineering, including different-pattern/task-format demonstrations and panel-by-panel prompting | Closed-source MLLMs · MARVEL | accuracy | Hurt |
Surprising findings
- Adding accurate text descriptions of puzzle panels lifts closed-source model accuracy sharply, implying perception rather than reasoning is the main bottleneck.
- Several open-source models output a single choice for almost all puzzles, so their accuracy reflects inductive bias rather than reasoning.
Limitations
- Open-source models limited to under 13B parameters due to compute, so larger open models were not evaluated.
- Few-shot demonstrations use only the sequence configuration and manually written CoT, limiting generalization of findings.
- Dataset mixes crawled puzzles with manually created ones, and abstract shapes are hard to describe in text.
Experimental setup
Datasets: MARVEL
Benchmarks: MARVEL
Metrics: accuracy group-based accuracy
Baselines: GPT-4V Gemini-pro-vision Claude3 (Sonnet) Claude3 (Opus) Qwen-VL Fuyu BLIP-2 InstructBLIP LLaVA-1.5
Builds on: RAVEN Bongard-LOGO SVRT chain-of-thought
Closest papers by meaning
- Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad? (ICML 2025 · similarity 0.91)
- VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning (ICLR 2026 · similarity 0.90)
- MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.90)
- Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency (NeurIPS 2026 · similarity 0.90)
- Zebra-CoT: A Dataset for Interleaved Vision-Language Reasoning (ICLR 2026 · similarity 0.90)
- PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error Detection (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.