Research map / Vision-language and multimodal / Multimodal LLMs
Seeing Without Understanding: Disentangling Perception, Reasoning, and Simulation in VLM Gameplay
Dingyang Jin, Jiawei He, Calvin Lo, Steven Hu, Ryan Rad
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a two-stage diagnostic framework that separates VLM game-playing failures into perception, rule comprehension, and reasoning components.
Problem
VLMs underperform on game-based reasoning, but existing evaluations conflate failures in visual perception, rule comprehension, and reasoning.
Approach
A two-stage diagnostic: controlled perception tests isolate visual encoding, and a diagnostic matrix with a six-level rule-comprehension scale separates rule understanding from reasoning and simulation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| align pieces with patch boundaries vs centers (ablation) | VLMs (6 models) · Chess and Gomoku perception | piece accuracy: within 3-5 points | No clear effect |
| use non-divisible instead of divisible image resolution (ablation) | VLMs (6 models) · Chess and Gomoku perception | piece accuracy: within 1-2 points | No clear effect |
| lower image resolution (ablation) | GPT-5.2 · Gomoku perception | piece accuracy: 75% -> 32% | Hurt |
| use 3D realistic rendering instead of 2D (ablation) | VLMs (6 models) · Chess and Gomoku perception | piece accuracy: 1-5 points, inconsistent | No clear effect |
| increase piece density (ablation) | VLMs (6 models) · Gomoku perception | piece accuracy: e.g. GPT-5.2 54.83 -> 79.75 | Improved |
| scale Qwen3-VL from 30B to 235B (ablation) | Qwen3-VL · Rule complexity ladder Explicit mode | accuracy | No clear effect |
| increase rule complexity from L1 to L4 (ablation) | Qwen3-VL 235B · Rule complexity ladder Predictive | reasoning accuracy given verified perception: 99% -> 56% | Hurt |
| use predictive instead of explicit mode (simulation) (ablation) | VLMs · Rule complexity ladder Level 6 | accuracy: 75.9 -> 49.2 | Hurt |
| replace image with text-only FEN input (ablation) | Qwen3-VL 235B · Predictive L6 | accuracy: 41.7 -> 79.0 | Improved |
| add chain-of-thought scratchpad (ablation) | VLMs · Rule ladder L3-L6 | accuracy: e.g. GPT-5.2 Explicit L3 96.5 -> 81.3 | Hurt |
| add chain-of-thought scratchpad (ablation) | GPT-5.2 · Predictive L5 | accuracy: 68.4 -> 82.1 | Improved |
| add chain-of-thought scratchpad (ablation) | Qwen3-VL 235B · Predictive L6 | accuracy: 41.7 -> 23.1 | Hurt |
Closest papers by meaning
- VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environments (CVPR 2026 · similarity 0.91)
- From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.90)
- Are Large Vision Language Models Good Game Players? (ICLR 2025 · similarity 0.90)
- GameVerse: Can Vision-Language Models Learn from Video-based Reflection? (ICML 2026 · similarity 0.90)
- Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic View (ICML 2024 · similarity 0.90)
- Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoning (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
