Research map / Vision-language and multimodal / Robot action and navigation
Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Heterogeneous Pre-trained Transformers (HPT), which pre-trains a shared transformer trunk with embodiment-specific stems and heads on up to 52 heterogeneous robot, simulation, and human-video datasets, and transfers it to new embodiments and tasks.
Problem
Training generalist robot policies is hard because robot data is heterogeneous across embodiments (proprioception, camera views, action spaces), tasks, and environments, and per-robot policies overfit and require expensive data.
Approach
Modularize the policy into embodiment-specific stems (cross-attention tokenizers mapping proprioception and vision to a fixed 16 tokens), a shared transformer trunk pre-trained with behavior cloning across all datasets, and task-specific heads; transfer by reinitializing stem and head and freezing or finetuning the trunk.
Key results
- HPT improves fine-tuned policy performance by over 20% on unseen tasks across multiple simulator benchmarks and real-world settings (per abstract).
- Pre-training validation loss scales with dataset size, number of datasets (up to 52), and model size (1M to 1B parameters).
- Real-world tool-use tasks (Sweep Leftover, Fill Water, Scoop Food, Switch Insertion) on Franka Panda show pre-trained HPT beating No-Trunk and From-Scratch baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pre-train the shared trunk on heterogeneous robot datasets | HPT transfer policies · simulation benchmarks | task success rate | Helped |
| pre-train the shared trunk on heterogeneous robot datasets | HPT transfer policies · real-world tasks | task success rate | Helped |
| pre-train on additional simulation and human-video datasets | HPT · pre-training validation datasets | validation loss | Mixed |
| increase the number of trajectories while also increasing model size and batch size | HPT pre-training · 27 datasets | validation loss | Helped |
| increase dataset size while keeping HPT-S and iterations fixed | HPT pre-training · 27 datasets | validation loss | No clear effect |
| increase model size and compute to convergence | HPT pre-training · 27 datasets | validation loss | Helped |
| scale model depth rather than width | HPT pre-training · 27 datasets | validation loss | No clear effect |
| increase batch size and training tokens | HPT pre-training · 27 datasets | validation loss | Helped |
| add previous actions as inputs (ablation) | HPT-Base pre-training · default setting | validation loss | Helped |
| add multiview inputs (ablation) | HPT-Base pre-training · default setting | validation loss | Helped |
| add language input (ablation) | HPT-Base pre-training · default setting | validation loss | Helped |
| remove proprioceptive tokens (ablation) | HPT transfer policies · simulation benchmarks | task success rate | Hurt |
Surprising findings
- Proprioception and vision inputs are both critical; removing either increases pre-training validation loss.
- Scaling depth vs. width made no significant difference in the model-scaling experiments.
Limitations
- Pre-training uses supervised behavior cloning and converges slowly; moderate data and compute scale relative to LLM training.
- Evaluated tasks are short-horizon manipulation with a fixed embodiment; learned policies are below 90% reliability on tested tasks.
- Embodiment splits in the dataset mixture are simple and data filtering/quality curation is under-explored.
Experimental setup
Datasets: Open X-Embodiment subset (41-42 datasets) Droid Simulation datasets (Drake, MuJoCo, Isaac Sim, PyBullet, Sapien, Flex) EPIC-Kitchen PoCo FrodoBot-2k Meta-world RoboMimic Fleet-Tools Simpler
Benchmarks: Meta-world RoboMimic Fleet-Tools Simpler
Metrics: validation loss task success rate
Baselines: From Scratch No Trunk R3M Voltron VC-1 Octo RT-1-X RT-2-X Diffusion Policy ACT
Builds on: Open X-Embodiment ResNet Diffusion Policy T5
Closest papers by meaning
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.91)
- Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.89)
- Beyond Data Scaling: Representation-Centric Pre-training for Vision-Language-Action Models (NeurIPS 2026 · similarity 0.89)
- Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations (ICML 2025 · similarity 0.89)
- Data Scaling Laws in Imitation Learning for Robotic Manipulation (ICLR 2025 · similarity 0.88)
- Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.