Research map / Vision-language and multimodal / Multimodal LLMs
Are Large Vision Language Models Good Game Players?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LVLM-Playground, a game-based benchmark that evaluates LVLMs on perceiving, Q&A, rule following, and end-to-end play across six board games, finding weak dense perception, rule comprehension, and sustained play.
Problem
Existing LVLM benchmarks (VQA, captioning) poorly capture detailed perception, multi-turn reasoning, and are prone to data contamination and prompt inconsistency.
Approach
Use structured, turn-based board games with rule-based outcomes and search-based opponents as a controlled evaluation environment, decomposed into four tasks, plus a game-based ability rating scheme (perception, reasoning, decision, adversary) derived from state space, branching factor, and game length.
Key results
- Gemini-1.5-Pro and Claude-3.5-Sonnet reach 0.994 and 0.985 perception accuracy on TicTacToe, but Claude scores only 0.554 on Chess perception.
- GPT-4o scores 0.013 on Gomoku perception versus 0.742 for Claude-3.5-Sonnet, attributed to vision-language misalignment.
- On Reversi rule following, GPT-4o scores 0.070, below the random baseline of 0.127.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Gemini1.5-pro instead of GPT-4o for Gomoku visual perception | LVLMs evaluated on LVLM-Playground · Gomoku | accuracy: 0.013 vs. 0.583 | Helped |
| Use Claude-3.5-sonnet instead of GPT-4o for Reversi visual perception | LVLMs evaluated on LVLM-Playground · Reversi | accuracy: 0.195 vs. 0.992 | Helped |
| Use Gemini1.5-pro instead of GPT-4o for TicTacToe rule following | LVLMs evaluated on LVLM-Playground · TicTacToe | accuracy: 0.895 -> 1.000 | Helped |
| Use Claude-3.5-sonnet instead of GPT-4o for Reversi rule following | LVLMs evaluated on LVLM-Playground · Reversi | accuracy: 0.070 -> 0.255 | Helped |
| Use Gemini1.5-pro instead of open-source models for Reversi end-to-end play | LVLMs evaluated on LVLM-Playground · Reversi | game-weighted performance: 40.0 -> 100.7 | Helped |
| Use Claude-3.5-sonnet instead of LLaVA-1.6-7b for Sudoku perception | LVLMs evaluated on LVLM-Playground · Sudoku | accuracy: 0 -> 0.912 | Helped |
Surprising findings
- GPT-4o often claims it cannot parse images, scoring far below commercial peers on perception.
- Open-source LVLMs stagnate at the initial Reversi score in end-to-end play, suggesting 'stochastic parrot' behavior where plausible text does not yield valid moves.
Limitations
- E2E evaluation uses only 100 gameplays per model, and Q&A/perception/rule tasks use 2,000 samples each.
- Ability ratings rely on estimated and derived parameters, some involving infeasible game states, and the human study only partially matches the computed ratings.
- Evaluation is limited to six board games and a small set of models.
Experimental setup
Benchmarks: LVLM-Playground
Metrics: accuracy (perceiving) multiple-choice accuracy (Q&A) valid move rate (rule following) game progress score (E2E)
Baselines: Gemini-1.5-Pro Claude-3.5-Sonnet Qwen2-VL-7B DeepSeek-VL-7B Phi-3-VL LLaVA-1.6-7B InternVL2-8B
Builds on: VLMEvalKit SmartPlay GAMA-Bench GTBench
Closest papers by meaning
- Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoning (ICLR 2026 · similarity 0.89)
- Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs (NeurIPS 2024 · similarity 0.89)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.88)
- Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model (NeurIPS 2025 · similarity 0.88)
- Caption This, Reason That: VLMs Caught in the Middle (NeurIPS 2025 · similarity 0.88)
- GameVerse: Can Vision-Language Models Learn from Video-based Reflection? (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.