Research map / Vision-language and multimodal / Robot action and navigation
Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GR-1, a GPT-style transformer for language-conditioned visual robot manipulation that is pre-trained on Ego4D language-conditioned video prediction and then finetuned end-to-end on robot data to predict actions and future frames.
Problem
Multi-task language-conditioned visual robot manipulation suffers from sparse, expensive robot data; the paper asks whether large-scale video generative pre-training can improve manipulation policies, including generalization to unseen scenes, objects, and instructions.
Approach
A unified GPT-style causal transformer takes language, image, and robot-state tokens, is pre-trained on language-conditioned video prediction over Ego4D clips, and is then finetuned with joint behavior cloning (arm, gripper) and future-image prediction losses using the same architecture.
Key results
- On CALVIN ABCD->D, GR-1 reaches average length 4.21 versus 3.06 for HULC (success rate 94.9% vs 88.9%).
- On CALVIN ABC->D zero-shot unseen scenes, GR-1 reaches 85.4% success on the first task versus 53.3% for the best baseline (average length 3.06 vs 0.93 for MT-R3M).
- With 10% of the ABCD->D data, GR-1 achieves 77.8% success on the first task versus 66.8% for HULC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add large-scale video pre-training and video prediction (ablation) | GR-1 · CALVIN ABCD→D | average length: 3.33 -> 4.21 | Helped |
| Add large-scale video pre-training (ablation) | GR-1 with video prediction · CALVIN ABC→D | average length: 2.65 -> 3.06 | Helped |
| Add large-scale video pre-training and video prediction (ablation) | GR-1 · CALVIN 10% data | average length: 1.04 -> 2.00 | Helped |
| Predict the future frame 3 steps ahead instead of 1 step ahead (ablation) | GR-1 without pre-training · CALVIN | average length: 3.61 -> 3.82 | Helped |
| Predict the future frame 5 steps ahead instead of 3 steps ahead (ablation) | GR-1 without pre-training · CALVIN | average length: 3.82 -> 3.67 | Hurt |
| Use GR-1 with video generative pre-training instead of HULC | CALVIN ABCD→D · CALVIN ABCD→D | average length: 3.06 -> 4.21 | Helped |
| Use GR-1 instead of RT-1 | CALVIN ABC→D unseen scene · CALVIN ABC→D (unseen scene) | success rate: 53.3% to 85.4% | Helped |
| Use GR-1 instead of HULC | CALVIN 10% data · CALVIN 10% data | success rate: 66.8% to 77.8% | Helped |
| Use GR-1 instead of RT-1 | Real robot object transportation · Unseen instances | success rate: 0.13 -> 0.73 | Helped |
| Use GR-1 instead of RT-1 | Real robot object transportation · Unseen categories | success rate: 0.00 -> 0.30 | Helped |
| Use GR-1 instead of RT-1 | Real robot articulated object manipulation · Drawer manipulation | success rate: 0.35 -> 0.75 | Helped |
| Use GR-1 instead of GR-1 without video prediction or pre-training (ablation) | CALVIN task rotate blue block right · CALVIN | task success rate: 71.2 -> 94.9 | Helped |
Surprising findings
- Pre-training on out-of-domain human egocentric video (Ego4D) transfers to robot manipulation without any robot video in pre-training.
- Predicting future frames 3 steps ahead performed better than predicting the next frame (1 step), though gains saturated at 5 steps.
Limitations
- Pre-training uses only language-annotated video, so unlabeled videos are not used.
- Unseen-category real-robot generalization is weak (30% success), with confusion between similarly colored objects.
- Real-robot evaluation is limited to two tasks with few settings.
Experimental setup
Datasets: Ego4D CALVIN real-robot demonstrations (HTC Vive teleoperation)
Benchmarks: CALVIN real robot object transportation real robot articulated object manipulation
Metrics: task success rate per chain position average length (tasks completed in a row of 5) success rate
Baselines: MCIL RT-1 HULC MT-R3M RoboCat
Builds on: GPT MAE CLIP R3M Perceiver resampler
Closest papers by meaning
- VideoVLA: Video Generators Can Be Generalizable Robot Manipulators (NeurIPS 2025 · similarity 0.91)
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.91)
- Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models (ICLR 2024 · similarity 0.90)
- Object-Centric Prompt-Driven Vision-Language-Action Model for Robotic Manipulation (CVPR 2025 · similarity 0.90)
- Exploring the Limits of Vision-Language-Action Manipulation in Cross-task Generalization (NeurIPS 2025 · similarity 0.90)
- Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.