Research map / Image generation / Autoregressive and latent generation
Prediction with Action: Visual Policy Learning via Joint Denoising Process
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PAD, a DiT-based visual policy that jointly denoises future images and robot actions (and optionally depth), co-trained on BridgeData-v2 videos and robot demos, and evaluated on 50-task Metaworld and real-world Panda manipulation.
Problem
Visual robot policy learning from limited demonstrations; existing approaches either use image generation as a separate two-stage step (e.g., goal-image synthesis) or decouple prediction from action, missing shared physical dynamics and internet video pretraining benefits.
Approach
A single diffusion transformer jointly denoises future RGB images, depth images, and robot actions via token concatenation with attention masks, allowing co-training on action-free video data by masking missing modalities, initialized from an ImageNet-pretrained DiT.
Key results
- PAD achieves 72.5% average success rate across 50 Metaworld tasks, versus 57.4% for the strongest baseline GR-1 (26.3% relative improvement).
- In real-world in-distribution tasks, PAD averages 72.5% success vs 69.0% for RT-2* in the reported table; PAD-Depth reaches 78%.
- Ablations show removing image prediction drops Metaworld average to 43.6%, and removing video co-training drops it to 59.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use joint future-image prediction and action generation with PAD | Single text-conditioned visual policy · MetaWorld (50 tasks) | success rate: 57.4% -> 72.5% | Helped |
| Use PAD instead of the strongest baseline | Real-world manipulation policy · Real-world manipulation (in-distribution) | success rate: 0.69 -> 0.72 | Helped |
| Add depth modality (ablation) | PAD · Real-world manipulation (in-distribution) | success rate: 0.72 -> 0.78 | Helped |
| Add future-image prediction (ablation) | PAD · MetaWorld (50 tasks) | success rate: 43.6% -> 72.5% | Helped |
| Add co-training on internet video data (ablation) | PAD · MetaWorld (50 tasks) | success rate: 59.2% -> 72.5% | Helped |
| Increase model size and computational allocation from XL/8 to XL/2 (ablation) | PAD · MetaWorld (50 tasks) | success rate: 48.2% -> 72.5% | Helped |
| Vary model size and patchify configuration (ablation) | PAD · MetaWorld (50 tasks) | success rate: 62.4%, 68.4%, 64.5%, 72.5% | Mixed |
Surprising findings
- Removing image prediction causes a large drop (72.5% to 43.6%), indicating that predicting only low-dimensional robot poses overfits in this data-efficient setting.
- Scaling analysis shows a model with fewer tokens (XL/8, 48.2%) performs worse than a smaller-compute but finer-patch model (B/2, 62.4%), indicating success correlates with transformer Gflops.
Limitations
- Control frequency is low because joint denoising of images and actions is expensive.
- Only three modalities (RGB, robot pose, depth) were tested; extension to tactile or other signals is left to future work.
- Real-world evaluation uses a limited set of tasks and the Metaworld setup uses a fixed camera viewpoint.
Experimental setup
Datasets: BridgeData-v2 Metaworld demonstrations real-world Panda teleoperation demonstrations
Benchmarks: Metaworld (50 tasks) real-world Panda manipulation (in-distribution and unseen-object generalization)
Metrics: success rate average success rate relative improvement
Baselines: Diffusion Policy SuSIE RT-1 RT-2* GR-1
Builds on: Diffusion Policy DiT DDPM latent diffusion SuSIE
Closest papers by meaning
- Two-Steps Diffusion Policy for Robotic Manipulation via Genetic Denoising (NeurIPS 2025 · similarity 0.92)
- Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations (ICML 2025 · similarity 0.91)
- VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation (NeurIPS 2024 · similarity 0.90)
- Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training (NeurIPS 2024 · similarity 0.90)
- One-Step Diffusion Policy: Fast Visuomotor Policies via Diffusion Distillation (ICML 2025 · similarity 0.90)
- Conditioning Matters: Training Diffusion Policies is Faster Than You Think (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.