Research map / Video / Video generation
Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pre-trains a unified discrete (mask-and-replace) diffusion model on VQ-VAE video tokens from actionless Ego4D human videos and robot videos, then fine-tunes a separate action denoising network on few robot demonstrations for Meta-World and RLBench manipulation.
Problem
Robot policy learning is limited by scarce action-labeled robot data, and the human-robot domain gap and noisy multimodal human videos make it hard to transfer knowledge from actionless human videos.
Approach
Compress human and robot videos into a shared VQ-VAE token space, pre-train a unified discrete diffusion model to predict future video tokens, and use the predicted-video hidden representations to guide action denoising during few-shot fine-tuning.
Key results
- On RLBench 16 tasks with 10 demos, VPDD outperforms RVT and PerAct with an average improvement of about 1.24x while using only RGB inputs.
- VPDD beats R3M-Diffusion by 16.4% and VC-1-Diffusion by 26.5% in success rate on Meta-World MT50-rand; pre-training on human videos adds 6.9% relative improvement over the robot-only variant.
- Generated future videos reach FVD 235.81 versus 264.66 for UniPi on Meta-World samples.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train on human videos in addition to robot videos (ablation) | VPDD · Meta-World MT50-rand | success rate: 6.9% relative improvement | Helped |
| Pre-train on increasing amounts of human videos (ablation) | VPDD · Meta-World policy fine-tuning | success rate | Helped |
| Use VPDD rather than UniPi for video prediction | Meta-World video prediction · Meta-World | FVD: 264.66 -> 235.81 | Helped |
| Use VPDD rather than RVT | RLBench multi-task policy · RLBench 16 tasks | success rate: 1.24× average improvement | Mixed |
Surprising findings
- VPDD maintains comparable success with only 1 fine-tuning demonstration on Meta-World.
Limitations
- Predicted videos are blurry, with errors in gripper pose and cross-view correspondence.
- Video quality is not the primary focus; resolution and semantic detail are limited.
- Evaluated on simulated benchmarks (Meta-World, RLBench) with a limited set of tasks.
Experimental setup
Datasets: Ego4D Meta-World RLBench RoboSet
Benchmarks: MT50-rand RLBench (16 tasks)
Metrics: success rate FVD
Baselines: PerAct RVT R3M-Diffusion VC-1-Diffusion MTDiff-p Video-MTDT SODA UniPi GR-1
Builds on: VQ-VAE VideoGPT D3PM discrete diffusion UniD3 Perceiver Transformer
Closest papers by meaning
- Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations (ICML 2025 · similarity 0.91)
- Prediction with Action: Visual Policy Learning via Joint Denoising Process (NeurIPS 2024 · similarity 0.90)
- Learning to Act from Actionless Videos through Dense Correspondences (ICLR 2024 · similarity 0.90)
- DiT4DiT: Jointly Modeling Video Dynamics and Actions for Generalizable Robot Control (NeurIPS 2026 · similarity 0.90)
- Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations (NeurIPS 2025 · similarity 0.90)
- Action Images: End-to-End Policy Learning via Multiview Video Generation (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.