Research map / Vision-language and multimodal / Multimodal LLMs
MineAnyBuild: Benchmarking Spatial Planning for Open-world AI Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MineAnyBuild, a Minecraft benchmark (~4,000 tasks plus ~2,000 VQA pairs) evaluating MLLM agents' spatial planning via executable building blueprints, spatial understanding, reasoning, creativity, and commonsense; 13 MLLMs are evaluated zero-shot.
Problem
Existing spatial intelligence benchmarks for MLLMs rely mainly on VQA about geometric attributes and do not test whether agents can turn spatial understanding into executable 3D task execution.
Approach
Evaluate spatial planning by having agents output executable 3D blueprint matrices for Minecraft architectures, scored by a critic MLLM, human review, and VQA tasks, with an expandable pipeline that converts player-generated builds into tasks.
Key results
- GPT-4o achieves the highest overall score of 41.02 (out of 100), with Claude-3.7-Sonnet at 40.70 and Gemini-1.5-Pro at 40.54.
- On spatial reasoning (mental rotation VQA), the best accuracy is 26.7% by GPT-4o-mini; open-source models score 15.56 to 23.30 overall.
- Open-source models score at most 1.34 out of 10 on executable plan generation, and most proprietary models have a much higher output success rate than open-source ones.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT-4o instead of GPT-4o-mini | GPT-4o / GPT-4o-mini · MineAnyBuild Spatial Reasoning | accuracy: 26.7 -> 24.4 | Hurt |
| Use proprietary models rather than open-source models | MLLM-based agents · MineAnyBuild | Overall score | Mixed |
| Increase model size from 3B to 7B | Qwen2.5VL · MineAnyBuild Executable Spatial Plan Generation | score: 0.63 to 1.29 | Helped |
| Increase model size from 3B to 7B | Qwen2.5VL · MineAnyBuild Spatial Reasoning | accuracy: 17.0 -> 16.0 | Hurt |
Surprising findings
- GPT-4o performs worse than GPT-4o-mini on spatial reasoning accuracy (19.8 vs 26.7 for some configs; GPT-4o-mini tops the column).
- Most MLLM agents perform poorly on mental-rotation tasks, with the best accuracy at only 26.7%.
Limitations
- Evaluation tools are slow and require manual operations.
- Scoring-based critic evaluation is less scientific, reliable and stable than fixed criteria.
- Blueprint matrix output format is limited by output token limits and action sequence length.
Experimental setup
Datasets: GrabCraft architectures Minecraft Official Wiki architectures Minecraft creator indoor decoration assets
Benchmarks: MineAnyBuild
Metrics: critic score (out of 10) accuracy output success rate
Baselines: Claude-3.5-Sonnet Claude-3.7-Sonnet Gemini-1.5-Flash Gemini-1.5-Pro Gemini-2.0-Flash GPT-4o GPT-4o-mini InternVL2.5-2B InternVL2.5-4B InternVL2.5-8B Qwen2.5VL-3B Qwen2.5VL-7B
Builds on: Voyager MineDojo Lego-Puzzles
Closest papers by meaning
- SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence (CVPR 2026 · similarity 0.89)
- Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmark (AAAI 2024 · similarity 0.87)
- ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Prompting (CVPR 2025 · similarity 0.87)
- SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning (NeurIPS 2026 · similarity 0.87)
- From Where Things Are to What They Are For: Benchmarking Spatial-Functional Intelligence in Multimodal LLMs (CVPR 2026 · similarity 0.87)
- Magma: A Foundation Model for Multimodal AI Agents (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.