Research map / Large language models / Retrieval and knowledge
VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledge
Yueqi Song, Tianyue Ou, Yibo Kong, Zecheng Li, Graham Neubig, Xiang Yue
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| enable extended thinking mode (ablation) | Claude-3.7-Sonnet · VisualPuzzles | accuracy (%): 48.3 -> 48.2 | No clear effect |
| enable extended thinking mode (ablation) | Claude-4.5-Sonnet · VisualPuzzles | accuracy (%): 48.4 -> 49.4 | No clear effect |
| use thinking variant of Gemini-2.0-flash (ablation) | Gemini-2.0-flash · VisualPuzzles | accuracy (%): 45.0 -> 42.2 | Hurt |
| use reasoning model QvQ-72B instead of Qwen2.5-VL-72B (ablation) | Qwen2.5-VL-72B · VisualPuzzles | accuracy (%): 42.3 -> 37.8 | Hurt |
| use reasoning model o1 instead of GPT-4o (ablation) | GPT-4o · VisualPuzzles | accuracy (%): 41.3 -> 51.8 | Improved |
| scale up model size (ablation) | open MLLMs · VisualPuzzles | accuracy (%): LLaVA-OV 7B 29.4 vs 72B 30.8 | No clear effect |
| scale up model size (ablation) | open MLLMs · MMMU | knowledge and reasoning accuracy | Improved |
| higher knowledge accuracy (correlation with reasoning accuracy) | open MLLMs · MMMU vs VisualPuzzles | Pearson correlation: 0.8 (MMMU) vs 0.4 (VisualPuzzles) | Mixed |
| branching and re-validation reasoning patterns | Claude-3.7-Sonnet-Thinking · VisualPuzzles | correlation with accuracy: 0.01 and 0.12 vs 0.07 and 0.19 on MMMU | No clear effect |
| enforce Answer-First versus Option-First prompting (ablation) | Kimi-K2.5 · VisualPuzzles vs MMMU | accuracy | Mixed |
| best model versus human 5th percentile | all evaluated MLLMs · VisualPuzzles | accuracy (%): 57.0 (o4-mini) vs 57.5 (human 5th pct) | Hurt |
| perform VisualPuzzles instead of MMMU for knowledge need | benchmark design · knowledge concept checklist | average knowledge questions: 3.9 -> 1.1 | Improved |
Closest papers by meaning
- MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.91)
- AStar: Boosting Multimodal Reasoning with Automated Structured Thinking (AAAI 2026 · similarity 0.91)
- VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning (ICLR 2026 · similarity 0.91)
- ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.91)
- Look on Demand: A Cognitive Scheduling Framework for Visual Evidence Acquisition in Multimodal Reasoning (ICML 2026 · similarity 0.91)
- VGR: Visual Grounded Reasoning (ICLR 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
