Research map / Large language models / Instruction and fine-tuning
Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games
Dongmin Park, Minkyu Kim, Beongjun Choi, Junhyuck Kim, Keon Lee, Jonghyun Lee, Inkyu Park, Byeong-Uk Lee, Jaeyoung Hwang, Jaewoo Ahn, Ameya Sunil Mahabaleshwarkar, Bilal Kartal and 4 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add reflection and planning modules to zero-shot agent (ablation) | GPT-4o · Orak avg across 12 games | average rank: 3.3 -> 2.2 | Improved |
| add reflection and planning modules to zero-shot agent (ablation) | Llama-3.2-3B · Orak avg across 12 games | average rank: 5.6 -> 6.1 | Hurt |
| add planning to reflection agent (ablation) | Llama-3.2-3B · Orak avg across 12 games | average rank: 5.6 -> 6.1 | Hurt |
| add image input alongside text (ablation) | GPT-4o · Orak Group 1 games | average rank: 1.7 -> 3.0 | Hurt |
| use image-only input instead of text (ablation) | GPT-4o · Orak Group 1 games | average rank: 1.7 -> 5.3 | Hurt |
| add image input alongside text (ablation) | GPT-4o · Ace Attorney | normalized game score: 85.3 -> 53.5 | Hurt |
| add image input alongside text (ablation) | Claude-3.7 · Street Fighter III | normalized game score: 16.7 -> 33.3 | Improved |
| fine-tune on Orak expert trajectories | Llama-3.2-3B · Street Fighter III | normalized game score: 12.0 -> 40.0 | Improved |
| fine-tune on Orak expert trajectories | Llama-3.2-3B · WebShop-H | score: 0.0 -> 12.6 | Improved |
| fine-tune on Orak expert trajectories | Llama-3.2-1B · WebShop-H | score: 1.9 -> 0.0 | No clear effect |
| fine-tune on Orak trajectories (OOD game) | Llama-3.2-3B · 2048 | normalized game score: 0.1 -> 3.1 | Improved |
| fine-tune on low-score instead of high-score data (ablation) | Llama-3.2-3B · Darkest Dungeon | normalized game score: 92.0 -> 40.0 | Hurt |
Closest papers by meaning
- BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games (ICLR 2025 · similarity 0.89)
- TowerMind: A Tower Defence Game Learning Environment and Benchmark for LLM as Agents (AAAI 2026 · similarity 0.88)
- CollabBench: Benchmarking and Unleashing Collaborative Ability of LLMs with Diverse Players via Proactive Engagement (ICML 2026 · similarity 0.88)
- LLM-PySC2: Starcraft II learning environment for Large Language Models (NeurIPS 2025 · similarity 0.88)
- TALES: Text-Adventure Learning Environment Suite (NeurIPS 2026 · similarity 0.88)
- SmartPlay : A Benchmark for LLMs as Intelligent Agents (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
