Research map / Vision-language and multimodal / Robot action and navigation
ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Prompting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes visual-temporal context prompting, where a VLM passes object segmentation masks and interaction types to ROCKET-1, a segmentation-conditioned Minecraft policy with SAM-2 tracking, and reports large gains on open-world interaction and long-horizon tasks.
Problem
Hierarchical embodied agents built on VLMs communicate with low-level policies via language, which cannot convey detailed spatial information about which object to interact with in open-world environments like Minecraft.
Approach
Replace language or future-image task specifications with object segmentation masks drawn on past observations plus interaction-type cues; a causal transformer policy (ROCKET-1) takes concatenated observation+mask channels, trained with mask dropout to encourage use of visual-temporal context, and training labels are produced automatically by backward trajectory relabeling with SAM-2.
Key results
- ROCKET-1 with Molmo reasoner reaches 0.82 overall success on the Minecraft Interaction Benchmark, versus 0.19 for STEVE-1 with human prompts and 0.09 for GROOT-1.
- With human prompts ROCKET-1 reaches 0.95 overall success.
- On long-horizon tasks ROCKET-1 reaches 0.70 on the obsidian task and 0.50 on pink wool, where all baselines fail.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use visual-temporal context prompts with ROCKET-1 instead of language, future-image, or latent-code communication | Hierarchical Minecraft agent · Long-horizon Minecraft tasks | success rate | Helped |
| Use ROCKET-1 with Molmo prompts instead of STEVE-1 | Minecraft Interaction Benchmark · Minecraft Interaction Benchmark | overall success rate: 0.19 -> 0.82 | Helped |
| Fuse interaction types in TransformerXL rather than in the visual backbone (ablation) | ROCKET-1 · Hunt | success rate: 0.72 -> 0.91 | Helped |
| Fuse interaction types in TransformerXL rather than in the visual backbone (ablation) | ROCKET-1 · Mine | success rate: 0.69 -> 0.78 | Helped |
| Add SAM-2 tracking when Molmo prompts are low-frequency (ablation) | ROCKET-1 with Molmo · Minecraft object interaction | success rate | Helped |
| Increase SAM-2 model size (ablation) | ROCKET-1 with Molmo · Minecraft object interaction | success rate | Mixed |
Surprising findings
- Fusing interaction type inside the visual backbone performed much worse (Hunt 0.72) than fusing in the transformer layer (Hunt 0.91).
Limitations
- Requires a high-level reasoner to supply prompts; baseline results used human prompts.
- Larger SAM-2 models reduce inference speed.
- Evaluated only in Minecraft.
Experimental setup
Datasets: OpenAI contractor data (VPT)
Benchmarks: Minecraft Interaction Benchmark long-horizon Minecraft tasks
Metrics: success rate FPS
Baselines: VPT-bc STEVE-1 GROOT-1 DEPS MineDreamer OmniJarvis
Builds on: VPT STEVE-1 SAM-2 TransformerXL hindsight relabeling
Closest papers by meaning
- VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environments (CVPR 2026 · similarity 0.89)
- SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RL (CVPR 2026 · similarity 0.88)
- Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds (ICLR 2024 · similarity 0.87)
- PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understanding (ICLR 2025 · similarity 0.87)
- OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents (NeurIPS 2024 · similarity 0.87)
- MineAnyBuild: Benchmarking Spatial Planning for Open-world AI Agents (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.