Research map / Multi-agent systems and games / LLM agents
MrSteve: Instruction-Following Agents in Minecraft with What-Where-When Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MrSteve, a low-level Minecraft instruction-following controller with Place Event Memory (what-where-when episodic memory), a count-based hierarchical exploration strategy, and a goal-conditioned VPT-Nav navigator, improving exploration and sparse sequential task solving over Steve-1.
Problem
Low-level instruction-following controllers in Minecraft (e.g., Steve-1) have only short-term Transformer-XL memory (128 steps), so they forget previously seen resources and re-explore randomly, becoming bottlenecks in LLM-augmented hierarchical agents.
Approach
Augment the low-level controller with Place Event Memory, a hierarchical store that clusters MineCLIP frame embeddings first by place (2D position and yaw) and then by distinct events (DP-Means on embeddings), enabling task-relevant recall via MineCLIP similarity; combine with count-based goal selection and a goal-conditioned VPT-Nav navigator trained with PPO and LoRA.
Key results
- On the 100x100 exploration task, the count-based plus VPT-Nav method reaches 84.42% map coverage with 0.38 revisit count, versus 50.77% coverage and 2.68 revisits for Steve-1.
- In Memory Tasks with 2K memory capacity, MrSteve with Place Event Memory shows high success rates while MrSteve-FM reaches about 60% on Find Water and MrSteve-EM about 50% on Find First-Visited House.
- In Long-Instruction (500K steps), MrSteve and MrSteve-PM solve over 80 tasks, while in Long-Navigation MrSteve solves around 70 tasks versus under 10 for the weakest baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Count-based high-level goal selection and VPT-Nav instead of Steve-1 exploration | MrSteve exploration · 100×100 map, 6K-step exploration | Map Coverage: 50.77 \pm 0.13 -> 84.42 \pm 0.06 | Helped |
| Use Count-based high-level goal selection and VPT-Nav instead of Steve-1 exploration | MrSteve exploration · 100×100 map, 6K-step exploration | Revisit Count: 2.68 \pm 1.36 -> 0.38 \pm 0.6 | Helped |
| Replace VPT-Nav with DQN as the low-level goal achiever (ablation) | Count-based hierarchical exploration · 100×100 map, 6K-step exploration | Map Coverage: 84.42 \pm 0.06 -> 31.83 \pm 0.11 | Hurt |
| Replace the Count-based goal selector with an RNN-based selector (ablation) | Hierarchical exploration with VPT-Nav · 100×100 map, 6K-step exploration | Map Coverage: 84.42 \pm 0.06 -> 29.82 \pm 0.07 | Hurt |
| Use the Plan4MC RNN-based selector and DQN goal achiever instead of Count-based and VPT-Nav (ablation) | Hierarchical exploration · 100×100 map, 6K-step exploration | Revisit Count: 0.38 \pm 0.6 -> 6.2 \pm 1.73 | Hurt |
| Add Place Event Memory and memory-augmented exploration/task solving to Steve-1 | Steve-1 · ABA-Sparse | success rate | Helped |
| Use Plan4MC exploration with FIFO Memory instead of Count-based exploration, VPT-Nav, and Place Event Memory (ablation) | MrSteve · ABA-Sparse | success rate | Hurt |
| Use FIFO Memory instead of Place Event Memory (ablation) | MrSteve · Find Water | success rate: about 60% | Hurt |
| Use Place Memory instead of Place Event Memory (ablation) | MrSteve · Find Zombies' Death Spot | success rate: about 60% | Hurt |
| Use Event Memory without place clusters instead of Place Event Memory (ablation) | MrSteve · Find First-Visited House | success rate: about 50% | Hurt |
| Use Event Memory without place clusters instead of Place Event Memory (ablation) | MrSteve · Long-Instruction | tasks completed: over $80$ tasks; around $50$ tasks | Hurt |
| Use Place Memory instead of Place Event Memory (ablation) | MrSteve · Long-Navigation | tasks completed: around $70$ tasks; less than $20$ tasks | Hurt |
Surprising findings
- Event-only memory (MrSteve-EM) performs worse than place-based memory in long-instruction tasks because visually similar events in different places merge into one cluster.
Limitations
- Experiments are limited to surface-level navigation, omitting underground mining.
- VPT-Nav is shown only on plains biomes; cliffs and bridge-building terrain are not addressed.
- Uses exact position data, which may not transfer to noisy-position robotics settings; PEM is not yet accessible to the high-level planner.
Experimental setup
Datasets: MineDojo environment interaction
Benchmarks: MineDojo custom exploration map (100x100) ABA-Sparse tasks Memory Tasks (Find Water, Find Zombies' Death Spot, Find First-Visited House) Long-Instruction Long-Navigation
Metrics: map coverage revisit count success rate task duration number of tasks solved
Baselines: Steve-1 Plan4MC exploration (RNN high-level + DQN low-level) FIFO Memory variant (MrSteve-FM) Event Memory variant (MrSteve-EM) Place Memory variant (MrSteve-PM)
Builds on: Steve-1 VPT MineCLIP Place Memory DP-Means PPO
Closest papers by meaning
- ADAM: An Embodied Causal Agent in Open-World Environments (ICLR 2025 · similarity 0.88)
- Experience Transfer for Multimodal LLM Agents in Minecraft Game (CVPR 2026 · similarity 0.88)
- Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks (NeurIPS 2024 · similarity 0.87)
- AUTOMEM: Automated Learning of Memory as a Cognitive Skill (NeurIPS 2026 · similarity 0.86)
- MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agents (NeurIPS 2026 · similarity 0.86)
- MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.