Research map / Multi-agent systems and games / Human-AI and goal-conditioned
GROOT: Learning to Follow Instructions by Watching Gameplay Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
GROOT learns a video-instruction-following Minecraft controller from unlabeled gameplay videos via a variational future-state prediction objective (behavior cloning plus KL goal-space regularization), evaluated on the new Minecraft SkillForge benchmark.
Problem
Low-level controllers in open-world environments such as Minecraft cannot follow open-ended instructions, since existing goal specifications (task indicators, future outcomes, language) are either not expressive enough or require costly trajectory-text annotations.
Approach
Specify goals as reference gameplay video clips and learn a goal space and goal-conditioned policy jointly from unlabeled gameplay videos by maximizing a variational lower bound on future-state likelihood, which reduces to behavior cloning plus a KL regularizer; the video encoder is a non-causal transformer and the policy is a causal transformer decoder with gated cross-attention to the goal embed
Key results
- In a human-judged Elo tournament over 1500 comparisons, GROOT outperforms the best baseline STEVE-1 by 150 points (about 70% winning rate), though human players remain far stronger.
- On 9 programmatic tasks, GROOT reaches human-level (100%) success on tasks such as dye and shear sheep, enchant sword, and use bow, but only 60% on build snow golems and 50% on build obsidian.
- On the long-horizon diamond-finding setting, GROOT finds diamonds with 16% success over 25 trials versus 0% for STEVE-1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use video instructions and the GROOT controller instead of STEVE-1 | Minecraft SkillForge overall · Minecraft SkillForge | Elo rating: 150 points | Helped |
| Use GROOT rather than STEVE-1 (visual) for diamond mining with sequential video instructions | Minecraft agents · Obtain diamond | success rate: 16% vs. 0% | Helped |
| Add KL loss during training (ablation) | GROOT · Collect seagrass | amount collected: 2× | Helped |
| Add KL loss during training (ablation) | GROOT · Collect wood | amount collected: 1.5× | Helped |
| Add KL loss during training (ablation) | GROOT · Use bow | performance | No clear effect |
| Increase condition slots from N=1 to N=3 (ablation) | GROOT · Minecraft SkillForge tasks | task performance: explore run distance: 54.0 -> 59.0; build pillar height: 37.6 -> 13.3; collect grass: 23.8 -> 5.6; collect seagrass: 3.3 | Mixed |
| Apply inference-time bias subtraction with λ=1.5 (ablation) | GROOT · Build obsidian, build snow golem, enchant sword, and dig three down and fill one up | success rate | Helped |
| Condition on a concatenated reference video instead of a raw single-task video | GROOT · Chop trees and hunt animals | single-task performance: 60% of the performance of raw video | Hurt |
Surprising findings
- Concatenated reference videos reach about 60% of the single-video performance while still driving the agent toward multiple tasks.
- KL regularization gave roughly 2x gains on collect seagrass and 1.5x on collect wood but no significant effect on use bow.
Limitations
- Highly sensitive to the choice of reference videos because the unsupervised goal space may not align with human intent.
- Struggles to identify items in the hotbar, leading to low success on build snow golems and build obsidian.
- Large Elo gap to human players remains.
Experimental setup
Datasets: OpenAI contractor data (VPT) Minecraft SkillForge (evaluation)
Benchmarks: Minecraft SkillForge programmatic task success rates (9 tasks) diamond mining long-horizon task
Metrics: Elo rating winning rate success rate distance collected wood killed mobs
Baselines: VPT (fd) VPT (bc) VPT (rl) STEVE-1 (visual) STEVE-1 (text)
Builds on: VPT STEVE-1 Flamingo Transformer-XL EfficientNet
Closest papers by meaning
- ADAM: An Embodied Causal Agent in Open-World Environments (ICLR 2025 · similarity 0.88)
- OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents (NeurIPS 2024 · similarity 0.87)
- MrSteve: Instruction-Following Agents in Minecraft with What-Where-When Memory (ICLR 2025 · similarity 0.86)
- GROOT-2: Weakly Supervised Multimodal Instruction Following Agents (ICLR 2025 · similarity 0.85)
- CrafterDojo: A Suite of Foundation Models for Building Open-Ended Embodied Agents in Crafter (NeurIPS 2026 · similarity 0.85)
- Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.