Research map / Vision-language and multimodal / Robot action and navigation
Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Hi Robot builds a hierarchical VLA system where a PaliGemma-3B high-level VLM reasons over complex user prompts and feedback to emit atomic language commands for a pi0 low-level flow-matching VLA, trained with synthetic VLM-generated interaction data.
Problem
Robots need to follow complex, open-ended instructions and situated mid-task user feedback, which flat instruction-following VLA policies handle poorly because they are trained mainly on simple atomic commands.
Approach
A two-level VLM-based hierarchy: a high-level VLM interprets open-ended prompts and images to output atomic skill commands and verbal responses for a low-level pi0 VLA; the high-level model is trained on teleoperated skill labels augmented with synthetic user prompts and interjections generated by a large VLM conditioned on images and skill labels.
Key results
- Hi Robot averages over 40% higher instruction accuracy than GPT-4o across table bussing, sandwich making, and grocery shopping.
- With expert human high-level commands the low-level policy executes nearly flawlessly, indicating failures stem more from reasoning than actuation.
- Removing synthetic data from high-level training substantially reduces handling of clarifications and constraints.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use hierarchical VLM reasoning with synthetic interaction data instead of GPT-4o as the high-level policy | Hi Robot · Table Bussing, Sandwich Making, and Grocery Shopping | Instruction Accuracy: over 40% higher | Helped |
| Use hierarchical VLM reasoning with synthetic interaction data instead of GPT-4o as the high-level policy | Hi Robot · Table Bussing, Sandwich Making, and Grocery Shopping | Task Progress | Helped |
| Use hierarchical reasoning rather than a flat VLA | Hi Robot · Table Bussing, Sandwich Making, and Grocery Shopping | Instruction Accuracy | Helped |
| Use hierarchical reasoning rather than a flat VLA | Hi Robot · Table Bussing, Sandwich Making, and Grocery Shopping | Task Progress | Helped |
| Add synthetic interaction data to high-level policy training (ablation) | Hi Robot · Open-ended instruction following | Instruction Accuracy | Helped |
| Add synthetic interaction data to high-level policy training (ablation) | Hi Robot · Open-ended instruction following | Task Progress | Helped |
| Use a separate high-level reasoning policy instead of a flat policy trained on the same synthetic data (ablation) | Hi Robot · Open-ended instruction following | Instruction Accuracy | Helped |
| Use a separate high-level reasoning policy instead of a flat policy trained on the same synthetic data (ablation) | Hi Robot · Open-ended instruction following | Task Progress | Helped |
Surprising findings
- Expert human high-level commands show the low-level policy is nearly flawless, so failures are mainly from high-level reasoning.
- GPT-4o frequently loses context once physical interaction begins and issues nonsensical commands.
Limitations
- High-level policy lacks memory and struggles with long-context reasoning.
- High-level and low-level models are decoupled and unaware of each other's capabilities.
- Synthetic data generation relies on prompt engineering.
Experimental setup
Datasets: teleoperated robot demonstrations segmented into skills synthetic user prompts and robot responses generated by a large VLM
Benchmarks: table bussing sandwich making grocery shopping
Metrics: instruction accuracy task progress
Baselines: GPT-4o high-level policy Flat VLA (pi0) Flat VLA with synthetic data Expert human high-level Hi Robot without synthetic data
Builds on: pi0 PaliGemma SayCan YAY Robot Mobile ALOHA
Closest papers by meaning
- ThinkBot: Embodied Instruction Following with Thought Chain Reasoning (ICLR 2025 · similarity 0.86)
- Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models (ICLR 2024 · similarity 0.86)
- HAMSTER: Hierarchical Action Models for Open-World Robot Manipulation (ICLR 2025 · similarity 0.85)
- RoboDreamer: Learning Compositional World Models for Robot Imagination (ICML 2024 · similarity 0.85)
- Robotic Visual Instruction (CVPR 2025 · similarity 0.85)
- RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.