Research map / Image generation / Fast diffusion sampling
Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SuSIE, which finetunes InstructPix2Pix on robot and human video to generate language-conditioned subgoal images, and pairs them with a goal-conditioned diffusion policy for zero-shot robotic manipulation.
Problem
Language-guided robotic manipulation that generalizes zero-shot to novel objects, scenes, and commands, where low-level policies often lack precision and semantic grounding.
Approach
Use a pretrained image-editing diffusion model (InstructPix2Pix) finetuned on robot and human videos as a high-level planner that generates nearby future subgoal images from the current observation and a language command; a language-agnostic goal-conditioned diffusion policy then reaches each subgoal.
Key results
- On CALVIN zero-shot (train A,B,C to test D), SuSIE reaches 0.87 success on 1 instruction and 0.26 on a 5-instruction chain, above the best prior baseline AugLC (0.69 and 0.05).
- In real-world tasks SuSIE averages 0.87 in Scene A, 0.50 in Scene B, and 0.88 in Scene C, beating RT-2-X (0.43, 0.00, 0.75) which uses a 55B model and over 1.1M trajectories.
- Oracle GCBC with privileged ground-truth goal images averages 0.66 on CALVIN versus 0.95 for SuSIE; SuSIE also beats it on real-world averages (0.87 vs 0.83 in Scene A, 0.50 vs 0.05 in Scene B).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use generated image subgoals to guide a goal-conditioned policy instead of a language-conditioned policy | robot manipulation · CALVIN test D, 5 instructions | success rate: 0.06 -> 0.26 | Helped |
| use generated image subgoals to guide a goal-conditioned policy instead of a language-conditioned policy | robot manipulation · Scene B average | success rate: 0.20 -> 0.50 | Helped |
| use generated image subgoals to guide a goal-conditioned policy instead of a language-conditioned policy | robot manipulation · Scene C average | success rate: 0.75 -> 0.88 | Helped |
| use generated subgoal guidance instead of privileged ground-truth final goals | goal-conditioned robot policy · Scene B average | success rate: 0.05 -> 0.50 | Helped |
| use generated subgoal guidance instead of privileged ground-truth final goals | goal-conditioned robot policy · CALVIN, selected tasks average | success rate: 0.66 -> 0.95 | Helped |
| add Something-Something video cotraining to BridgeData training (ablation) | SuSIE subgoal model · Scene B average | success rate: 0.30 -> 0.50 | Helped |
| add Something-Something video cotraining to BridgeData training (ablation) | SuSIE subgoal model · Scene C average | success rate: 0.80 -> 0.88 | Helped |
| remove InstructPix2Pix initialization and train the subgoal model from scratch (ablation) | SuSIE subgoal model · real-world subgoal generation | subgoal quality | Hurt |
| remove Something-Something video cotraining (ablation) | SuSIE subgoal model · BridgeData V2 validation set | subgoal quality | No clear effect |
Surprising findings
- Executing generated subgoals outperforms a policy given privileged ground-truth final goal images (Oracle GCBC) on CALVIN and in real-world scenes.
- Open-source video-generation (UniPi) reimplementations perform poorly due to temporally inconsistent frames, whereas single-subgoal image editing avoids this issue.
Limitations
- Diffusion subgoal model and low-level policy are trained separately, so the planner is unaware of the policy's capabilities.
- Performance is often bottlenecked by the low-level goal-conditioned policy, especially on precise grasps.
- Real-world evaluation uses only 9 tasks across 3 scenes with 10 trials per task, a small sample.
Experimental setup
Datasets: BridgeData V2 Something-Something CALVIN Ego4D
Benchmarks: CALVIN real-world WidowX tasks (Scenes A, B, C)
Metrics: success rate per instruction in chain task success rate
Baselines: HULC MCIL MdetrLC AugLC LCBC UniPi (HiP) UniPi (Ours) RT-2-X MOO Oracle GCBC
Builds on: InstructPix2Pix diffusion policy UniPi HiP
Closest papers by meaning
- AffordDP: Generalizable Diffusion Policy with Transferable Affordance (CVPR 2025 · similarity 0.90)
- Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation (ICLR 2024 · similarity 0.90)
- Generate Subgoal Images before Act: Unlocking the Chain-of-Thought Reasoning in Diffusion Model for Robot Manipulation with Multimodal Prompts (CVPR 2024 · similarity 0.90)
- GravMAD: Grounded Spatial Value Maps Guided Action Diffusion for Generalized 3D Manipulation (ICLR 2025 · similarity 0.90)
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.90)
- Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.