Research map / Vision-language and multimodal / Robot action and navigation
MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MP5, a five-module MLLM-based Minecraft agent with goal-conditioned active perception that handles context- and process-dependent open-ended tasks.
Problem
Long-horizon open-world embodied tasks in Minecraft are hard due to process dependency (ordering of sub-objectives) and context dependency (execution depends on perceiving environment context), which prior LLM agents handle by assuming all-seeing state knowledge.
Approach
A goal-conditioned active perception scheme in which a LoRA-tuned multimodal LLM (MineLLM, built on MineCLIP + Vicuna-13B) answers objective-specific questions round by round, mediated by a Patroller, so that GPT-4-based Parser/Planner and an action Performer can plan and execute with situation awareness.
Key results
- On Context-Dependent Tasks with active perception, MP5 reaches 91.0% success on Complex tasks vs 87.5% for GPT-4V and 0.0% for LLaVA-1.5.
- On Process-Dependent Tasks, MP5 achieves 22.00% success on Diamond level, versus 14.00% without situation-aware planning and 0.00% without embodied execution.
- Replacing GPT-4 with GPT-3.5-turbo as Planner drops Diamond-level success from 22.00% to 0.00%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Active Perception instead of Fine-Grained Global Perception | MP5 · Context-Dependent Tasks: Complex | success rate: 67.5 -> 91.0 | Helped |
| use Active Perception instead of Fine-Grained Global Perception | MP5 · Context-Dependent Tasks: Hard | success rate: 77.5 -> 93.0 | Helped |
| use CLIP instead of MineCLIP as MineLLM's visual encoder (ablation) | MineLLM · Context-Dependent Tasks | success rate: 95.00 vs 98.50; 90.00 vs 94.50; 87.00 vs 93.00; 80.00 vs 91.00 | Hurt |
| use LLaVA-1.5 instead of MineLLM for perception (ablation) | MP5 · Context-Dependent Tasks: Mid | success rate: 50.00 vs 90.00 | Hurt |
| use Vicuna-13B-v1.5 instead of GPT-4 as zero-shot Planner (ablation) | MP5 · Process-Dependent Tasks: Basic | success rate: 96.00 vs 1.33 | Hurt |
| use GPT-3.5-turbo instead of GPT-4 as zero-shot Planner (ablation) | MP5 · Process-Dependent Tasks: Diamond | success rate: 22.00 vs 0.00 | Hurt |
| remove situation-aware planning (ablation) | MP5 · Process-Dependent Tasks: Diamond | success rate: 22.00 -> 14.00 | Hurt |
| remove embodied action execution (ablation) | MP5 · Process-Dependent Tasks | success rate: 0.00% | Hurt |
| remove Performer Memory (ablation) | MP5 · Process-Dependent Tasks: Diamond | success rate: 22.00% -> 16.67% | Hurt |
| remove Patroller's error-checking feedback (ablation) | MP5 with Random Drop · Process-Dependent Tasks: Wooden | success rate: 76.67% to 7.33% | Hurt |
| enable Random Drop (ablation) | MP5 · Process-Dependent Tasks: Wooden | success rate: 88.67 -> 76.67 | Hurt |
| enable Random Drop (ablation) | MP5 · Process-Dependent Tasks: Diamond | success rate: 22.00 -> 1.33 | Hurt |
Surprising findings
- Without embodied action execution (w/o E.) the system still reaches 14-92% on process tasks, but without situation-aware planning (w/o P.) success falls to 0% at all levels.
- A frozen MineCLIP vision encoder outperforms a frozen CLIP encoder for the Minecraft-tuned MLLM (Complex 91.0 vs 80.0).
Limitations
- Reliance on GPT-3.5-turbo or GPT-4 APIs limits usability and accessibility.
- Evaluated only in Minecraft; not extended to other simulation platforms.
Experimental setup
Datasets: MineDojo snapshots (50K images) 500K generated image-text instruction data
Benchmarks: Context-Dependent Tasks (16 tasks, custom) Process-Dependent Tasks (25 tasks, custom) Open-Ended Tasks
Metrics: success rate (%)
Baselines: LLaVA-1.5 GPT-4V GPT-3.5-turbo Vicuna-13B-v1.5 DEPS GITM Voyager DreamerV3 VPT
Builds on: LLaVA-1.5 MineCLIP Voyager MineDojo GITM
Closest papers by meaning
- ADAM: An Embodied Causal Agent in Open-World Environments (ICLR 2025 · similarity 0.88)
- CrafterDojo: A Suite of Foundation Models for Building Open-Ended Embodied Agents in Crafter (NeurIPS 2026 · similarity 0.88)
- Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds (ICLR 2024 · similarity 0.88)
- OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents (NeurIPS 2024 · similarity 0.87)
- Optimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policy (CVPR 2025 · similarity 0.87)
- Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.