Research map / Multi-agent systems and games / LLM agents

GameDevBench: Evaluating Agentic Capabilities Through Game Development

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces GameDevBench, a 333-task benchmark of Godot game development tasks derived from tutorials, testing coding agents on multimodal game-engine work, and shows simple screenshot/video feedback improves agent success.

Problem

Coding agents have advanced rapidly but multimodal agents lag; there is no benchmark testing agents on complex, multimodal game development inside a game engine with deterministic verification.

Approach

Build tasks automatically from Godot YouTube and web tutorials with LLM-based task construction, refine them with checklists and human annotation, verify them with Godot's in-engine unit tests, and add two visual feedback tools (editor screenshot MCP server and runtime video) to agents.

Key results

• Best agent, Gemini 3 Pro with screenshot and video feedback, solves 53.8% of 333 tasks pass@1.
• GPT-5.4 improves from 41.1% to 52.0% pass@1 with screenshot MCP and runtime video feedback.
• Average success drops from 51.4% on gameplay logic tasks to 33.0% on 2D graphics and animation tasks.

Surprising findings

• Runtime video often gives more improvement than enabling both screenshot and video feedback together.
• Kimi K2.5 solves 91.3% of Design2Code but only 18.9% of GameDevBench tasks without multimodal support, suggesting web UI multimodal skill does not transfer to game development.

Limitations

• Evaluates only code-generation solutions, not editor-based computer use.
• Tasks are restricted to the Godot engine and derived from tutorials, limiting scope.
• Some tasks required human refinement; automated task construction had noticeable error rates.

Experimental setup

Datasets: Godot YouTube tutorials Godot Recipes (KidsCanCode) web tutorials

Benchmarks: GameDevBench

Metrics: pass@1

Baselines: Claude Haiku 4.5 Claude Sonnet 4.5 Gemini 3 Flash Gemini 3 Pro GPT-5.4 Mini GPT-5.4 Kimi K2.5 Qwen3.5-397B

Builds on: SWE-Bench OpenHands

Closest papers by meaning

• AgentStudio: A Toolkit for Building General Virtual Agents (ICLR 2025 · similarity 0.87)
• MCU: An Evaluation Framework for Open-Ended Game Agents (ICML 2025 · similarity 0.86)
• AgentVista: Evaluating Multimodal Agent in Ultra-Challenging Realistic Visual Scenarios (NeurIPS 2026 · similarity 0.85)
• Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale (ICML 2025 · similarity 0.85)
• SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? (ICLR 2025 · similarity 0.85)
• 3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code (NeurIPS 2026 · similarity 0.85)