Research map / Vision-language and multimodal / Robot action and navigation
HAMSTER: Hierarchical Action Models for Open-World Robot Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HAMSTER, a hierarchical VLA where a VLM fine-tuned on off-domain data (RoboPoint, RLBench sim, Bridge/DROID) predicts 2D end-effector paths that condition a low-level 3D policy (RVT-2 or 3D Diffuser Actor).
Problem
Monolithic vision-language-action models need expensive on-robot data and generalize poorly across visual and semantic shifts; the paper asks how cheap off-domain data (sim, videos, other embodiments) can be used for robot manipulation generalization.
Approach
Fine-tune a VLM to output coarse 2D end-effector paths (with gripper open/close markers) from off-domain data, then draw these paths onto images to condition a small 3D low-level policy, so the VLM never needs in-domain action labels and the policy handles precise control.
Key results
- Averaged over seven generalization axes, HAMSTER improves success by about 20 points over OpenVLA (about 50% relative gain) in real-world tests.
- On novel camera views, HAMSTER+RVT2 (Concat) achieves 0.98 success vs 0.23 for OpenVLA; HAMSTER+RVT2 gets 0.73.
- In Colosseum simulation, HAMSTER+3D-DA outperforms vanilla 3D-DA by 31% on average over 5 seeds (avg 0.46 vs 0.35 across variations).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add HAMSTER 2D path guidance to 3D-DA | 3D-DA · Colosseum visual variations | Success: $0.35 \pm 0.04$ -> $\textbf{0.46} \pm 0.04 $ | Helped |
| Add HAMSTER path guidance to 3D-DA with 50% of demonstrations (ablation) | 3D-DA · Colosseum | Success: 0.18 $\pm$ 0.10 -> 0.36 $\pm$ 0.04 | Helped |
| Add HAMSTER path guidance to 3D-DA | 3D-DA · Colosseum | Success: 0.18 $\pm$ 0.10 -> \textbf{0.43 $\pm$ 0.05} | Helped |
| Concatenate the path in a separate RGB input instead of overlaying it (ablation) | HAMSTER+RVT2 · Real-world novel camera | Success: 0.73 -> \textbf{0.98} | Helped |
| Concatenate the path in a separate RGB input instead of overlaying it (ablation) | HAMSTER+RVT2 · Real-world novel camera | Complete: 0.40 -> \textbf{0.90} | Helped |
| Fine-tune the VLM with RLBench simulation path sketches (ablation) | HAMSTER VILA path predictor · Real-world VLM path ranking, all data | Average rank: 2.13 -> \textbf{1.40} | Helped |
| Fine-tune VILA on cross-domain data rather than use zero-shot GPT-4o path generation | HAMSTER VILA path predictor · Real-world VLM path ranking, all data | Average rank: 3.47 -> \textbf{1.40} | Helped |
| Fine-tune VILA on cross-domain data rather than use GPT-4o with Code-as-Policies | HAMSTER VILA path predictor · Real-world VLM path ranking, all data | Average rank: 3.41 -> \textbf{1.40} | Helped |
| Fine-tune VILA for HAMSTER on the reported training mixture | VILA1.5-13B · VQA-v2, GQA, VizWiz, MME, MMBench, MMBench-Chinese, LLaVA-Bench, MM-Vet | Score: 82.8 -> \textbf{82.9}; 64.3 -> \textbf{64.9}; 62.6 -> \textbf{63.4}; 1569.6 -> \textbf{1588.4}; 74.9 -> \textbf{75.3}; 6 | Helped |
| Fine-tune VILA for HAMSTER on the reported training mixture | VILA1.5-13B · ScienceQA-IMG, TextVQA, POPE, SEED-Bench, SEED-Bench (Image), MMMU-val | Score: 80.1 -> 78.0; 65.0 -> 61.4; 86.3 -> 85.8; 65.1 -> 64.2; 72.6 -> 71.9; 37.9 -> 37.8 | Hurt |
| Fine-tune OpenVLA on RLBench before fine-tuning on real-world tasks (ablation) | OpenVLA · Real-world pick-and-place, 6 Basic tasks | Success score: 0.58 -> 0.54 | No clear effect |
| Add HAMSTER hierarchical path guidance | RVT-2 · Real-world tabletop manipulation across generalization axes | Success rate: 20% improvement; 50% relative gain | Helped |
Surprising findings
- Fine-tuning OpenVLA on the RLBench simulation data gave little benefit in the real world (0.54 vs 0.58 success), suggesting monolithic VLAs cannot easily exploit off-domain sim data.
- Concatenating the path as a separate image channel worked better than overlaying it on RGB for RVT-2, which the authors attribute to reprojection fragmenting the drawn path.
Limitations
- Paths are 2D only, so the VLM has no native 3D spatial understanding.
- The 2D path interface is bandwidth-limited and cannot express force or rotation.
- Real-world evaluation uses a single Franka setup with a limited number of tasks and human-assigned partial-credit scores.
Experimental setup
Datasets: RoboPoint RLBench Bridge DROID VQA dataset (LLaVA-style, 660k) Colosseum
Benchmarks: Real-world Franka tabletop tasks (222 evaluations, 74 tasks) Colosseum RLBench VQA-v2 GQA VizWiz ScienceQA-IMG TextVQA POPE MME MMBench MMBench-Chinese
Metrics: success score (partial points per sub-action) success rate complete executions human ranking
Baselines: OpenVLA RVT-2 3D Diffuser Actor RT-Trajectory GPT-4o
Builds on: RT-Trajectory VILA-1.5 RVT-2 3D Diffuser Actor OpenVLA
Closest papers by meaning
- VLA-REPLICA: A Low-Cost, Reproducible Benchmark for Real-World Evaluation of Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
- CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (CVPR 2025 · similarity 0.90)
- BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Models (NeurIPS 2025 · similarity 0.90)
- DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge (NeurIPS 2025 · similarity 0.90)
- ActionUNet: Improving Robustness of VLA Models with Efficient Multi-scale Fine-tuning (NeurIPS 2026 · similarity 0.90)
- Beyond World-Frame Action Heads: Motion-Centric Action Frames for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.