Research map / 3D vision / 3D generation
Pre-training Auto-regressive Robotic Models with 4D Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ARM4R, an autoregressive robotic model pre-trained by predicting 3D point tracks (4D representations lifted from monocular depth) on Epic-Kitchens human videos, then fine-tuned on robot point tracks and proprioceptive control.
Problem
Robot learning lacks large-scale pre-training data, and existing VLA pre-training uses high-level vision-language objectives that do not model low-level physical dynamics needed for control.
Approach
Pre-train an autoregressive transformer to predict 3D point tracks (4D: 3D plus time) pseudo-labeled by SpatialTracker on human egocentric videos, exploiting that robot point motion is a linear function of joint states so the representation transfers to proprioceptive control with minimal robot data.
Key results
- On RLBench 12 tasks, ARM4R reaches 59.47% average success vs. 55.33% for PerAct and 48.33% for LLARVA.
- On 13 real Kinova tasks, ARM4R averages 83.1% vs. 37.2% for OpenVLA, 21.2% for pi0-FAST, and 6.4% for ATM.
- Kinova-to-Franka transfer with Epic pre-training and Kinova video fine-tuning improves average Franka performance by 19.6%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ARM4R with human-video 4D point-track pre-training and robot fine-tuning instead of PerAct | Robot policies on RLBench Multi-Task · RLBench Multi-Task | average success rate (%): 55.33 -> 59.47 | Helped |
| Use ARM4R instead of OpenVLA | Real Kinova robot policies · Real Kinova Multi-Task | average success rate (%): 37.2 -> 83.1 | Helped |
| Use ARM4R instead of LLARVA | Kinova cube tasks · Kinova cube pick | success rate (%): 93.75 -> 96.0 ± 2.3 | Helped |
| Add human-video pre-training (Stage 1) (ablation) | ARM4R with Stage 3 control fine-tuning · Kinova real tasks | success rate | Helped |
| Add robot-video point-tracking fine-tuning (Stage 2) (ablation) | ARM4R with Stage 3 control fine-tuning · Kinova real tasks | success rate | Helped |
| Add human-video pre-training and Kinova point-tracking fine-tuning before Franka control fine-tuning (ablation) | ARM4R · Franka cube tasks | success rate (%): 73.3 ± 2.7 -> 93.3 ± 1.3 | Helped |
| Move the cube during the robot trajectory at 1/3 or 2/3 descent | ARM4R · Kinova cube tasks | success rate: No Disturbance: 96.0±2.3, 61.3±1.3, 94.7±1.3; At 1/3 Descent: 94.7±3.5, 60.0±0.0, 93.3±1.3; At 2/3 Descent: 92.0±2.3, 57 | Hurt |
| Reduce ambient lighting to 50% | ARM4R · Kinova cube tasks | success rate: Standard: 96.0±2.3, 61.3±1.3, 94.7±1.3; Dim Light: 86.7±1.3, 52.0±2.3, 81.3±2.7 | Hurt |
| Add dynamic background distractors | ARM4R · Kinova cube tasks | success rate: Standard: 96.0±2.3, 61.3±1.3, 94.7±1.3; Background Distractor: 94.7±1.3, 57.3±1.3, 90.7±1.3 | Hurt |
| Add two tabletop distractor objects near the target | ARM4R · Kinova cube tasks | success rate: Standard: 96.0±2.3, 61.3±1.3, 94.7±1.3; Table-top Distractors: 81.3±1.3, 48.0±2.3, 74.7±1.3 | Hurt |
Surprising findings
- Pre-training solely on human video (Epic-Kitchens) gives better real-robot performance than OpenVLA and pi0-FAST pre-trained on OpenX robot data.
- Ablation indicates human-video 4D pre-training (Stage 1) contributes more to performance than robot-video point-tracking fine-tuning (Stage 2).
Limitations
- Tracks 3D points in camera coordinates, so learned representations entangle object and camera motion and are sensitive to viewpoint.
- Fixed uniform grid tracking may waste capacity on irrelevant points; struggles on precision tasks like screw bulb and on unnatural rotations like put knife.
- Performance drops under tabletop distractors (e.g., 81.3% pick vs 96.0% standard).
Experimental setup
Datasets: Epic-Kitchens100 ImageNet OpenX RLBench Kinova robot demonstrations Franka demonstrations
Benchmarks: RLBench (12 tasks) real Kinova Gen3 tasks (13 tasks) real Franka cube tasks
Metrics: success rate
Baselines: ATM LLARVA OpenVLA pi0-FAST PerAct C2FARM-BC ManiGaussian Image-BC (ViT) MVP RPT Octo
Builds on: SpatialTracker PerAct LLARVA CrossMAE ATM
Closest papers by meaning
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.89)
- Object-centric 3D Motion Field for Robot Learning from Human Videos (NeurIPS 2025 · similarity 0.87)
- PointZero: 3D Point Track Completion for Learning Transferable 3D Dynamics (NeurIPS 2026 · similarity 0.87)
- DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control (NeurIPS 2026 · similarity 0.87)
- Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training (NeurIPS 2024 · similarity 0.87)
- Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.