Research map / Vision-language and multimodal / Multimodal LLMs
LLMs as Rules Oracles: Exploring Real-World Multimodal Reasoning in Tabletop Strategy Game Environments
Joseph J Peper, Sai Krishna Gandra, Yunxiang Zhang, Vaibhav Chennareddy, Shloki Jha, Ali Payani, Lu Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LudoBench, a multimodal benchmark testing whether vision-enabled LLMs can acquire and reason over tabletop game rules from heterogeneous sources to correctly understand new strategy games.
Problem
Uninitiated gamers struggle to correctly comprehend the rules of a new tabletop strategy game, which requires integrating heterogeneous multimodal game knowledge; existing benchmarks emphasize deep strategic mastery rather than this initial rules comprehension.
Approach
A benchmark framing LLMs as rules oracles for analog tabletop games, targeting initial rule acquisition and integration across multimodal game materials rather than strategic play.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| require rule integration on top of perception (Tier 1 to Tier 2) | frontier multimodal LMs · LudoBench | accuracy: 63% -> 36% | Hurt |
| require short-horizon optimization (Tier 2 to Tier 3) | frontier multimodal LMs · LudoBench | accuracy: 36% -> 8% | Hurt |
| add text rulebook (ablation) | GPT-4o · LudoBench Tier 2, Pax Renaissance | rule application accuracy: 17% -> 64% | Improved |
| add image rulebook instead of text rulebook (ablation) | GPT-4o · LudoBench Tier 2, Pax Renaissance | rule application accuracy: 64% -> 56% | Hurt |
| add text rulebook (ablation) | all evaluated models · LudoBench | accuracy: Gemini 2.5 Pro and o3 gain nearly 7% | Improved |
| add image rulebook (ablation) | GPT models versus Gemini models (except Pro 3) · LudoBench | accuracy | Mixed |
| longer reasoning outputs on harder tiers | frontier LMs · LudoBench Tier 3 | accuracy | No clear effect |
| plurality voting across all model and modality outputs | 27 outputs per example · LudoBench Tier 3 | accuracy: best plurality 11% on Pax Ren. | No clear effect |
Experimental setup
Benchmarks: LudoBench
Closest papers by meaning
- Play to Generalize: Learning to Reason Through Game Play (ICLR 2026 · similarity 0.90)
- GTBench: Uncovering the Strategic Reasoning Capabilities of LLMs via Game-Theoretic Evaluations (NeurIPS 2024 · similarity 0.89)
- Are Large Vision Language Models Good Game Players? (ICLR 2025 · similarity 0.88)
- BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (ICLR 2025 · similarity 0.88)
- Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoning (ICLR 2026 · similarity 0.88)
- GameArena: Evaluating LLM Reasoning through Live Computer Games (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
