Research map / Multi-agent systems and games / Human-AI and goal-conditioned
MCU: An Evaluation Framework for Open-Ended Game Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MCU, a Minecraft evaluation framework with 3,452 composable atomic tasks, LLM-based task configuration, and a VLM-based multi-dimensional AutoEval that agrees with human ratings at 91.5%, then benchmarks four Minecraft agents.
Problem
Evaluating open-ended game agents is hard because existing Minecraft benchmarks have low task quality, limited diversity, and lack automatic evaluation, so each agent is tested on its own task set.
Approach
Treat atomic tasks as unit tests, sample LLM-generated initial configurations for intra-task diversity, compose tasks with logical operators for inter-task diversity, and score trajectories with a GPT-4o VLM judge using task-specific criteria across six dimensions.
Key results
- AutoEval with GPT-4o reaches 91.5% average agreement with human judgments across six dimensions, versus an average F1 of 34.6 for MineCLIP on comparative prediction (AutoEval GPT-4o average F1 84.0).
- Task progress correlates with human ratings at Pearson 0.78 (Kendall tau 0.71).
- Evaluated agents (VPT BC/RL, STEVE-I, GROOT) show nonzero but partial task progress; GROOT is best overall, and VPT (RL) performs well only on diamond-related crafting tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AutoEval (GPT-4o) instead of MineCLIP for trajectory comparisons | Comparative trajectory evaluation · Comparative evaluation (average) | F1: 34.6 -> 84.0 | Helped |
| Use AutoEval (GPT-4o) instead of AutoEval (MiniCPM) | Comparative trajectory evaluation · Comparative evaluation (average) | F1: 60.0 -> 84.0 | Helped |
| Evaluate GROOT in hard rather than simple task mode | GROOT · Six selected tasks | task progress: 0.50 -> 0.36 | Hurt |
| Evaluate GROOT in hard rather than simple task mode | GROOT · Six selected tasks | action control: 0.38 -> 0.33 | Hurt |
| Increase task difficulty from simple to hard | GROOT · Mine Iron Ore | task progress: 0.56 -> 0.60 | Helped |
| Increase task difficulty from simple to hard | GROOT · Build Nether Portal | task progress: 0.45 -> 0.30 | Hurt |
| Increase task difficulty from simple to hard | GROOT · Sleep in bed | task progress: 0.85 -> 0.50 | Hurt |
| Increase task difficulty from simple to hard | GROOT · Combat Skeletons | task progress: 0.56 -> 0.30 | Hurt |
| Increase task difficulty from simple to hard | GROOT · Six selected tasks | task progress: 0.50 -> 0.36 | Mixed |
| Evaluate VPT (RL) on tasks unrelated to its diamond-reward objective | VPT (RL) · Tasks unrelated to obtaining diamonds | task progress | Hurt |
| Evaluate agents on compositional rather than atomic tasks | Existing Minecraft agents · Compositional tasks | task progress | Hurt |
Surprising findings
- GROOT's task progress drops on average from 0.50 to 0.36 when moving from simple to hard configurations of the same task (e.g., sleep in bed drops from 0.85 to 0.50).
- VPT (RL), tuned for diamond collection, underperforms on unrelated tasks, showing poor inter-task generalization.
Limitations
- AutoEval performs worse on intricate craft tasks (F1 62.0) than on other categories.
- Human validation uses only 500 trajectories across 60 tasks and 20 annotators.
- The full 3,452-task dataset is described as continuously expanding and the release status is only partially stated (code and scripts available).
Experimental setup
Datasets: MineMDojo tasks SkillForge tasks Minecraft Wiki advancements VPT contractor data snapshots 500 annotated trajectories
Benchmarks: MCU
Metrics: task progress F1 for pairwise preference Pearson correlation Kendall's tau agreement rate with humans success rate
Baselines: MineCLIP VPT (BC) VPT (RL) STEVE-I GROOT
Builds on: MineStudio MineDojo Reflexion MineCLIP VPT
Closest papers by meaning
- MineAnyBuild: Benchmarking Spatial Planning for Open-world AI Agents (NeurIPS 2025 · similarity 0.86)
- Craftium: Bridging Flexibility and Efficiency for Rich 3D Single- and Multi-Agent Environments (ICML 2025 · similarity 0.86)
- GameDevBench: Evaluating Agentic Capabilities Through Game Development (ICML 2026 · similarity 0.86)
- CrafterDojo: A Suite of Foundation Models for Building Open-Ended Embodied Agents in Crafter (NeurIPS 2026 · similarity 0.85)
- ADAM: An Embodied Causal Agent in Open-World Environments (ICLR 2025 · similarity 0.84)
- MCP-AgentBench: Evaluating Real-World Language Agent Performance with MCP-Mediated Tools (AAAI 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.