AI Paper Atlas

Research map / Vision-language and multimodal / Multimodal LLMs

LLMs as Rules Oracles: Exploring Real-World Multimodal Reasoning in Tabletop Strategy Game Environments

Joseph J Peper, Sai Krishna Gandra, Yunxiang Zhang, Vaibhav Chennareddy, Shloki Jha, Ali Payani, Lu Wang

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces LudoBench, a multimodal benchmark testing whether vision-enabled LLMs can acquire and reason over tabletop game rules from heterogeneous sources to correctly understand new strategy games.

Problem

Uninitiated gamers struggle to correctly comprehend the rules of a new tabletop strategy game, which requires integrating heterogeneous multimodal game knowledge; existing benchmarks emphasize deep strategic mastery rather than this initial rules comprehension.

Approach

A benchmark framing LLMs as rules oracles for analog tabletop games, targeting initial rule acquisition and integration across multimodal game materials rather than strategic play.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
require rule integration on top of perception (Tier 1 to Tier 2)frontier multimodal LMs · LudoBenchaccuracy: 63% -> 36%Hurt
require short-horizon optimization (Tier 2 to Tier 3)frontier multimodal LMs · LudoBenchaccuracy: 36% -> 8%Hurt
add text rulebook (ablation)GPT-4o · LudoBench Tier 2, Pax Renaissancerule application accuracy: 17% -> 64%Improved
add image rulebook instead of text rulebook (ablation)GPT-4o · LudoBench Tier 2, Pax Renaissancerule application accuracy: 64% -> 56%Hurt
add text rulebook (ablation)all evaluated models · LudoBenchaccuracy: Gemini 2.5 Pro and o3 gain nearly 7%Improved
add image rulebook (ablation)GPT models versus Gemini models (except Pro 3) · LudoBenchaccuracyMixed
longer reasoning outputs on harder tiersfrontier LMs · LudoBench Tier 3accuracyNo clear effect
plurality voting across all model and modality outputs27 outputs per example · LudoBench Tier 3accuracy: best plurality 11% on Pax Ren.No clear effect

Experimental setup

Benchmarks: LudoBench

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy