Research map / Vision-language and multimodal / Robot action and navigation
Generate Subgoal Images before Act: Unlocking the Chain-of-Thought Reasoning in Diffusion Model for Robot Manipulation with Multimodal Prompts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoTDiffusion, a hierarchical framework where a conditional diffusion model generates chained subgoal images from multimodal prompts as a high-level planner, guiding a low-level goal-conditioned MLP policy, evaluated on VIMA-BENCH.
Problem
Robot agents struggle to follow complex multimodal prompts in long-horizon manipulation, where text-only subgoal decomposition is redundant and deviations accumulate without intermediate guidance.
Approach
Use a diffusion model as a high-level visual planner that generates step-by-step subgoal images, with a triple-alignment semantic module (coarse mask-patch pretraining) tracking progress along the chain, plus bi-directional generation and frame concatenation, feeding a simple goal-conditioned policy.
Key results
- CoTDiffusion achieves 64.6 ± 3.0 overall success rate on VIMA-BENCH long-horizon tasks, versus 49.5 for VIMA and 10.0 for SuSIE.
- On visual constraints, CoTDiffusion reaches 83.1 ± 4.7 success vs 67.2 for VIMA.
- FID of generated subgoals is 11.3/10.8/8.6 (rearrange/reasoning/constraints) vs 19.7/18.3/23.2 for SuSIE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| generate chained subgoal images from multimodal prompts before action execution | CoTDiffusion · Overall VIMA-BENCH long-horizon tasks | success rate: 49.5 ± 4.3 -> 64.6 ± 3.0 | Helped |
| generate chained subgoal images from multimodal prompts before action execution | CoTDiffusion · Visual rearrange | success rate: 43.1 ± 3.3 -> 59.0 ± 1.7 | Helped |
| generate chained subgoal images from multimodal prompts before action execution | CoTDiffusion · Visual reasoning | success rate: 38.2 ± 4.4 -> 51.7 ± 2.6 | Helped |
| generate chained subgoal images from multimodal prompts before action execution | CoTDiffusion · Visual constraints | success rate: 67.2 ± 5.2 -> 83.1 ± 4.7 | Helped |
| add manually decomposed step-wise sub-prompts | SuSIE · Overall VIMA-BENCH long-horizon tasks | success rate: 10.0 ± 2.3 -> 43.0 ± 5.9 | Helped |
| add coarse semantic alignment pre-training (ablation) | CoTDiffusion · Visual rearrange | FID: 23.1 -> 11.3 | Helped |
| add coarse semantic alignment pre-training (ablation) | CoTDiffusion · Visual reasoning | FID: 16.2 -> 10.8 | Helped |
| add coarse semantic alignment pre-training (ablation) | CoTDiffusion · Visual constraints | FID: 18.9 -> 8.6 | Helped |
| add bi-directional generation (ablation) | CoTDiffusion · Visual rearrange | FID: 18.7 -> 11.3 | Helped |
| add bi-directional generation (ablation) | CoTDiffusion · Visual reasoning | FID: 15.0 -> 10.8 | Helped |
| add frame concatenation (ablation) | CoTDiffusion · Visual rearrange | FID: 15.4 -> 11.3 | Helped |
| add frame concatenation (ablation) | CoTDiffusion · Visual reasoning | FID: 13.7 -> 10.8 | Helped |
Surprising findings
- SuSIE with manually decomposed sub-prompts (privileged information) improves substantially but still trails CoTDiffusion, which uses raw multimodal prompts.
- Bi-directional generation without coarse semantic pretraining degrades performance, since alignment tokens become too similar across steps.
Limitations
- Evaluated only on VIMA-BENCH simulated tasks in Ravens simulator.
- Low-level policy is a simple MLP; the paper does not improve the policy model.
- Relies on privileged expert data and ground-truth keyframes from scripted oracles.
Experimental setup
Datasets: VIMA-BENCH scripted oracle trajectories
Benchmarks: VIMA-BENCH
Metrics: success rate FID normalized CLIP score single-view vs multi-view performance drop
Baselines: Gato Flamingo VIMA SuSIE
Builds on: SuSIE VIMA VDT
Closest papers by meaning
- PERIA: Perceive, Reason, Imagine, Act via Holistic Language and Vision Planning for Manipulation (NeurIPS 2024 · similarity 0.91)
- Zero-Shot Robotic Manipulation with Pre-Trained Image-Editing Diffusion Models (ICLR 2024 · similarity 0.90)
- AffordDP: Generalizable Diffusion Policy with Transferable Affordance (CVPR 2025 · similarity 0.89)
- GravMAD: Grounded Spatial Value Maps Guided Action Diffusion for Generalized 3D Manipulation (ICLR 2025 · similarity 0.89)
- RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation (ICLR 2025 · similarity 0.89)
- DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.