Research map / Large language models / Agents, code and math
KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces KORGym, a dynamic benchmark platform of over 50 text and multimodal games across six reasoning dimensions, built on Gymnasium for RL integration, and evaluates 19 LLMs and 8 VLMs.
Problem
Existing LLM reasoning benchmarks are domain-specific or confounded by pretraining knowledge and opponent dynamics, so they fail to measure general, knowledge-orthogonal reasoning in multi-turn settings.
Approach
A modular game platform (inference, game interaction, evaluation, communication modules) with over 50 knowledge-orthogonal games, controllable difficulty, multi-turn play and RL support, plus a dimension-aware normalized aggregate score that handles heterogeneous game scales.
Key results
- O3-mini achieves the highest overall average (82%) across reasoning dimensions, followed by Gemini-2.5-pro (79%) and O1 (73%).
- Thinking models outperform size-matched non-thinking variants; DeepSeek-R1-Distill-Qwen-32B exceeds Qwen2.5-72B-Instruct (39% vs 18% average).
- Textual versions of games score consistently higher than visual versions for open-source VLMs, and visual versions of some games score higher for closed-source VLMs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Disable individual reasoning paradigms using prompt constraints (ablation) | Evaluated LLMs · Selected KORGym games | average performance | Helped |
| Disable the mathematical reasoning paradigm (ablation) | Most evaluated models · Selected KORGym games | performance | Mixed |
| Use visual rather than textual game versions | Evaluated VLMs · Six representative games | score | Mixed |
| Apply reinforcement-learning fine-tuning, including training on classic games | Doubao-1.5-thinking-pro · KORGym | reasoning performance | Helped |
Surprising findings
- Disabling explicit reasoning paradigms (via prompt constraints) increased average performance across all models.
- Open-source VLMs perform better on text-based than visual versions, while some closed-source VLMs score higher on visual versions.
Limitations
- Leaderboard for closed-source models relies on hosted APIs and evaluation uses only zero-shot prompting with default sampling parameters.
- Evaluation uses 50 seeds for single-epoch games and 20 environments for multi-epoch games, a limited number of instances per game.
- RL experiments are described only qualitatively for Doubao-1.5-thinking-pro, with no controlled RL training results.
Experimental setup
Benchmarks: KORGym
Metrics: normalized score capability dimension aggregated mean binary score proportional score cumulative score
Baselines: LogicGame AgentBench GameArena SPIN-Bench TextArena ReasoningGym
Builds on: KORBench Gymnasium Minigrid
Closest papers by meaning
- GTBench: Uncovering the Strategic Reasoning Capabilities of LLMs via Game-Theoretic Evaluations (NeurIPS 2024 · similarity 0.91)
- GameArena: Evaluating LLM Reasoning through Live Computer Games (ICLR 2025 · similarity 0.91)
- Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective (NeurIPS 2025 · similarity 0.90)
- BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (ICLR 2025 · similarity 0.89)
- Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search (ICML 2025 · similarity 0.89)
- Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.