Research map / Multi-agent systems and games / LLM agents
GROOT-2: Weakly Supervised Multimodal Instruction Following Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GROOT-2, a weakly supervised latent-variable instruction-following policy trained on mostly unlabeled demonstrations plus a small labeled subset, evaluated on Atari, Minecraft, Language Table and Simpler Env.
Problem
Instruction-following policies need costly multimodal labels at scale, while label-free latent-variable models (VAE-style) suffer from latent space ambiguity (mechanical imitation vs posterior collapse) that misaligns the latent with human intent.
Approach
Train a shared latent space with constrained self-imitation (KL-regularized ELBO-style objective) on unlabeled demonstrations, and align it with human intention using a small set of multimodal labels (text or episode returns) via a closed-form MLE term that makes label-conditioned latents likely under the video-conditioned posterior.
Key results
- On Minecraft (100 tasks), GROOT-2 visual reaches 37% on Combat vs 18% for GROOT-1 and 48% on Hunt vs 28%.
- On Language Table, GROOT-2 (100% labels) achieves 84% overall success with language vs 78% for LAVA and 74% for RT-1.
- On Simpler Env, GROOT-2 (100%) language reaches 42% average on Pick Coke Can and 31% on Open/Close Drawer, comparable to RT-1-X (49%, 29%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add human intention alignment loss L_lab (ablation) | GROOT-2 · Language Table video instructions | success rate: 10±2 -> 76±7 | Helped |
| add constrained self-imitating loss L_dem (ablation) | GROOT-2 · Language Table language instructions | success rate: 12±3 -> 82±8 | Helped |
| increase labeled trajectory proportion from 50% to 100% (ablation) | GROOT-2 · Language Table overall | success rate: 82±8 -> 84±6 | Helped |
| increase labeled trajectory proportion from 0%-25% to 50%-80% (ablation) | GROOT-2 · Language Table | success rate: 10% to 65%; 82% to 83% | Helped |
| increase unlabeled trajectory data from 0% to 100% (ablation) | GROOT-2 · Minecraft Climb Mountain | human-normalized task score: 5× | Helped |
| initialize vision and language backbones with CLIP instead of random weights (ablation) | GROOT-2 · Language Table | success rate: 76±10 -> 82±8 | Helped |
| use ImageNet rather than random initialization for ViT (ablation) | GROOT-2 · Language Table | success rate: 76±10 -> 80±11 | Helped |
| use BERT rather than random initialization for BERT (ablation) | GROOT-2 · Language Table | success rate: 79±12 -> 81±8 | Helped |
| use language rather than visual instructions | GROOT-2 (50% labels) · Language Table overall | success rate: 76±7 -> 82±8 | Helped |
| use language rather than visual instructions | GROOT-2 (100% labels) · Language Table overall | success rate: 78±8 -> 84±6 | Helped |
| use GROOT-2 visual instructions instead of GROOT-1 visual instructions | Minecraft · all task categories | success rate | Helped |
| add labeled return trajectories to self-supervised training | GROOT · Atari video-instruction following | instruction-following performance | Helped |
Surprising findings
- Removing the label alignment loss yields 10% success with visual prompts, with the agent mechanically mimicking arm trajectories and ignoring object colors and shapes.
- GROOT-1 on Language Table reaches only 8% overall, showing that self-supervised reference-video following fails to capture human intent.
Limitations
- Relies on trajectory data with action labels, so it cannot use action-free video data.
- Video instructions underperform text instructions by about 6% on Language Table, partly due to reference video ambiguity.
Experimental setup
Datasets: MineRL contractor dataset Language Table oracle sim datasets OpenX d4rl-atari (DQN replay) Atari 2600 (ALE)
Benchmarks: Minecraft SkillForge (100 tasks, 8 families) Language Table Simpler Env Atari (4 games and 35 games) Minecraft 5-task unlabeled-scaling set
Metrics: success rate human-normalized task score success rate within 200 steps
Baselines: VPT STEVE-1 GROOT-1 BC-Zero LAVA RT-1 RT-1-X Octo-base GROOT (ssl)
Builds on: GROOT-1 VPT STEVE-1 Transformer-XL CLIP
Closest papers by meaning
- Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning (ICML 2024 · similarity 0.87)
- From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons (CVPR 2025 · similarity 0.87)
- Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents (NeurIPS 2026 · similarity 0.86)
- VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought (NeurIPS 2024 · similarity 0.86)
- DeepEyesV2: Toward Agentic Multimodal Model (ICLR 2026 · similarity 0.86)
- Multimodal Policy Internalization for Conversational Agents (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.