Research map / Video / Video segmentation and tracking
Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Video Prediction Policy (VPP), which fine-tunes Stable Video Diffusion into a text-guided manipulation video predictor and uses one-step internal diffusion features (via a Video Former and diffusion policy head) as predictive visual representations for a closed-loop generalist robot policy.
Problem
Existing visual encoders for robot policies are pre-trained on single images or image pairs and capture static information, neglecting future dynamics needed for embodied control; diffusion-based future-prediction policies are slow and low-frequency.
Approach
Treat intermediate features of a video diffusion model fine-tuned for manipulation as "predictive visual representations" obtained from a single forward denoising step, aggregate multi-layer up-sampling features, and learn an implicit inverse dynamics policy on top, enabling high-frequency control.
Key results
- On CALVIN ABC-D, VPP reaches an average task completion length of 4.33 versus 3.35 for the prior best (3D Diffuser Actor / RoboUniview-level baselines), with 10% of ABC data achieving 3.25.
- On real-world dexterous hand tasks, VPP achieves 0.75 seen and 0.60 unseen success versus 0.45 and 0.28 for SuSIE (the strongest baseline).
- On Bridge prediction, VPP attains FVD 41.4 versus 246.3 for Seer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use predictive video-diffusion representations instead of Stable-VAE, VC-1, or Voltron features (ablation) | VPP · CALVIN ABC→D | Avg. Len: 2.58 -> 4.33; 1.23 -> 4.33; 1.54 -> 4.33 | Helped |
| add internet manipulation data to video-model training (ablation) | VPP · CALVIN ABC→D | Avg. Len: 3.97 -> 4.33 | Helped |
| include Calvin video in video-model training (ablation) | VPP · CALVIN ABC→D | Avg. Len: 3.31 -> 4.33 | Helped |
| use SVD pretraining together with internet data (ablation) | VPP · CALVIN ABC→D | Avg. Len: 1.63 -> 4.33 | Helped |
| add Video Former to aggregate TVP features (ablation) | VPP · CALVIN ABC→D | Avg. Len: 3.86 -> 4.33 | Helped |
| add Video Former to aggregate TVP features (ablation) | VPP · CALVIN ABC→D | Latency: ~450ms -> $~140ms | Helped |
| add multi-layer feature aggregation instead of using final-layer features (ablation) | VPP · CALVIN ABC→D | Avg. Len: 3.60 -> 4.33 | Helped |
| add Temporal-attn to Video Former (ablation) | VPP · CALVIN ABC→D | Avg. Len: 4.18 -> 4.33 | Helped |
| use one-step rather than 2-step TVP denoising for representations (ablation) | VPP · CALVIN ABC→D | Avg. Len: 4.19 -> 4.33 | No clear effect |
| use multiple camera views rather than only the static view (ablation) | VPP · CALVIN ABC→D | Avg. Len: 3.58 -> 4.33 | Helped |
| use Layer-9 rather than Layer-3, Layer-6, or Layer-12 features (ablation) | VPP · CALVIN ABC→D | Avg. Len: 3.72; 3.88; 4.29; 4.05; 4.33 | Mixed |
| extract features at diffusion time-step 20 rather than time-step 10 or 30 (ablation) | VPP · CALVIN ABC→D | Avg. Len: 4.21; 4.33; 4.25 | No clear effect |
Surprising findings
- A single denoising step of the video model already yields representations useful for control, while a 2-step denoising variant did not improve performance.
- Removing the internet human manipulation data lowers CALVIN average length from 4.33 to 3.97, and training without SVD pretraining drops it to 1.63.
Limitations
- Single-step representations lack precise texture and detail, so only coarse future dynamics are captured.
- Real-world evaluation uses only three baselines and modest rollout counts per task category.
Experimental setup
Datasets: Something-Something-v2 RT-1 Bridge BC-Z Taco-Play Jaco-Play CALVIN ABC Metaworld Self-collected Panda Arm Self-collected Dexterous Hand
Benchmarks: CALVIN ABC-D MetaWorld (50 tasks) Bridge FVD real-world Franka Panda tasks real-world Xhand dexterous tasks
Metrics: average task completion length success rate FVD
Baselines: RT-1 Diffusion Policy Robo-Flamingo UniPi MDT SuSIE GR-1 RoboUniview Vidman VC-1 Voltron Stable-VAE
Builds on: Stable Video Diffusion diffusion policy CLIP Octo
Closest papers by meaning
- Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training (NeurIPS 2024 · similarity 0.91)
- Prediction with Action: Visual Policy Learning via Joint Denoising Process (NeurIPS 2024 · similarity 0.91)
- VideoVLA: Video Generators Can Be Generalizable Robot Manipulators (NeurIPS 2025 · similarity 0.90)
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.90)
- Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.90)
- VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.