Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Optimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Optimus-2, a Minecraft agent pairing an MLLM planner with a GOAP policy (action-guided behavior encoder plus MLLM backbone), and releases the 25k-video MGOA dataset built by automated data generation.
Problem
Goal-conditioned policies in Minecraft struggle to model observation-action relationships and to understand open-ended natural language sub-goals, limiting long-horizon and open-ended task performance.
Approach
A Goal-Observation-Action Conditioned Policy uses a Causal Perceiver to inject action embeddings into visual features, a History Aggregator with memory bank to compress history into fixed-length behavior tokens, and an MLLM (DeepSeek-VL-1.3B initialized, LoRA) that aligns sub-goals with these tokens to predict actions via a VPT action head, trained with behavior cloning plus KL distillation from V
Key results
- GOAP improves over prior SOTA on atomic tasks by 5%, 4%, 31%, and 35% on logs, seeds, dirt, and stone.
- Optimus-2 achieves highest success rates on all seven long-horizon task groups, e.g., 13% on Diamond and 28% on Redstone groups.
- On open-ended instruction tasks, GOAP with GPT-4V planner reaches 0.75 success on Torch, while baselines fail on Golden Shovel, Diamond Pickaxe, and Compass.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GOAP [VPT] instead of the strongest reported baseline | Minecraft atomic-task policies · Logs, Seeds, Dirt, and Stone | average reward: Logs: 14.3 -> 15.0; Seeds: 8.2 -> 8.5; Dirt: 20.3 -> 26.7; Stone: 19.0 -> 25.7 | Helped |
| Use Optimus-2 instead of Optimus-1 | Minecraft long-horizon tasks · Wood, Stone, Iron, Gold, Diamond, RedStone, and Armor | success rate: Wood: 0.98 -> 0.99; Stone: 0.92 -> 0.93; Iron: 0.46 -> 0.53; Gold: 0.08 -> 0.09; Diamond: 0.11 -> 0.13; RedStone: 0.25 - | Helped |
| Use GOAP rather than STEVE-1 | GPT-4V-planned open-ended tasks · Torch and Rail | success rate: Torch: 0.66 -> 0.75; Rail: 0.10 -> 0.47 | Helped |
| Add the Action-guided Behavior Encoder's Causal Perceiver (ablation) | GOAP · Atomic Tasks (average) | average reward: 10.0 (↓47.4%) -> 19.0 | Helped |
| Add the History Aggregator (ablation) | GOAP · Atomic Tasks (average) | average reward: 12.2 (↓35.8%) -> 19.0 | Helped |
| Add the Memory Bank (ablation) | GOAP · Atomic Tasks (average) | average reward: 14.6 (↓23.2%) -> 19.0 | Helped |
| Replace the LLM backbone with Transformer-XL (ablation) | GOAP · Open-ended instructions: Golden Shovel, Diamond Pickaxe, Compass | success rate: 17.1; 13.1; 16.2 -> 0.5; 0.5; 0.0 | Hurt |
| Replace the pretrained VPT action head with a 2-layer MLP projector (ablation) | GOAP · Logs, Seeds, Dirt, and Stone | average reward: Logs: 15.0 -> 7.2; Seeds: 8.5 -> 4.3; Dirt: 26.7 -> 14.4; Stone: 25.7 -> 15.5 | Hurt |
| Train with MGOA instead of only the OpenAI Contractor Dataset (ablation) | GOAP · Minecraft atomic tasks | average reward | Helped |
| Train with only the OpenAI Contractor Dataset rather than the mixed dataset (ablation) | GOAP · Stone | average reward: 89% | Hurt |
Surprising findings
- Replacing the pretrained VPT action head with a 2-layer MLP projector degrades performance, attributed to VPT's pretraining knowledge.
- Training only on the OpenAI Contractor Dataset yields poor atomic-task performance compared to MGOA, despite its task variety.
Limitations
- Long-horizon success rates remain low on harder groups such as Iron, Gold, and Diamond.
- Relies on GPT-4V planner and existing STEVE-1 agent for data generation.
Experimental setup
Datasets: MGOA OpenAI Contractor Dataset
Benchmarks: MineRL Minecraft atomic tasks Minecraft long-horizon tasks (67 tasks, 7 groups) Minecraft open-ended instruction tasks
Metrics: average rewards success rate
Baselines: VPT (text) STEVE-1 GROOT FSQ GROOT GPT-4V DEPS Jarvis-1 Optimus-1
Builds on: VPT STEVE-1 GROOT Optimus-1 MineCLIP
Closest papers by meaning
- MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception (CVPR 2024 · similarity 0.87)
- Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks (NeurIPS 2024 · similarity 0.85)
- CrafterDojo: A Suite of Foundation Models for Building Open-Ended Embodied Agents in Crafter (NeurIPS 2026 · similarity 0.83)
- ADAM: An Embodied Causal Agent in Open-World Environments (ICLR 2025 · similarity 0.82)
- OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents (NeurIPS 2024 · similarity 0.82)
- GROOT: Learning to Follow Instructions by Watching Gameplay Videos (ICLR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.