Research map / Vision-language and multimodal / Multimodal LLMs
Evaluating Spatiotemporal Reasoning of Vision-Language Models in Atari Gameplay
Mingjia Huo, Yao Fu, Bo Chang, Yaqing Wang, Dawei Zhu, Dylan Zhang, Xinyang Yi, Lichan Hong, Ed Chi, Hao Zhang, Jiaxi Tang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AtariBench, a controlled benchmark evaluating vision-language models on sequential decision-making across 20 Atari 2600 games from high-temporal-resolution gameplay streams.
Problem
VLMs perform well on static visual understanding benchmarks, but their ability to act in dynamic, fast-changing environments is poorly understood.
Approach
A controlled Atari 2600 benchmark that decouples fast in-game dynamics from wall-clock time so VLMs are evaluated on sequential decisions from high-temporal-resolution gameplay streams.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use append-only full history instead of structured history (ablation) | Gemini 2.5 Flash (off) · Breakout | noop-only share of turns: 0.7% -> 48.8% | Hurt |
| use append-only history (ablation) | Gemini 2.5 Flash/Pro, Gemini 3 Flash · Air Raid, Pacman, Phoenix | reward | No clear effect |
| add textual reward and life-loss feedback (ablation) | Gemini 2.5 Flash · Breakout | reward: 1.5 -> 2.3 | Improved |
| add textual feedback (ablation) | Gemini 2.5 Flash · Fishing Derby | reward: -11.8 -> -9.0 | Improved |
| add textual feedback (ablation) | Gemini 3.1 Pro · four games | reward | No clear effect |
| allow 1-10 actions per query instead of 1 action (ablation) | Gemini 3 Flash · Breakout | reward: 4.6 -> 3.7 | Hurt |
| vary action-chunk length (ablation) | Gemini 3 Flash and GPT-5.4 Mini · three other game pairs and models | reward | No clear effect |
| raise reasoning effort | VLMs · AtariBench | score and diagnostic accuracy | No clear effect |
| re-query with current frame only instead of complete history (ablation) | VLM on spatial-relation failure cases · Freeway etc. | spatial error persistence: 39.3% -> 10.7% | Improved |
| use fixed 30-second horizon instead of game-over termination (ablation) | Gemini 2.5 Flash · DemonAttack | per-run reward SD: 36.3 -> 13.7 | Improved |
| add survival instruction to prompt under game-over termination (ablation) | Gemini 2.5 Flash · Phoenix | average reward: 177.9 -> 195.4 | Improved |
Experimental setup
Datasets: Atari 2600 gameplay (20 games)
Benchmarks: AtariBench
Closest papers by meaning
- GameVerse: Can Vision-Language Models Learn from Video-based Reflection? (ICML 2026 · similarity 0.89)
- VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environments (CVPR 2026 · similarity 0.88)
- Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World (CVPR 2026 · similarity 0.87)
- PhysVLM-AVR: Active Visual Reasoning for Multimodal Large Language Models in Physical Environments (NeurIPS 2025 · similarity 0.87)
- Are Large Vision Language Models Good Game Players? (ICLR 2025 · similarity 0.87)
- ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Prompting (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
