Research map / Image generation / Autoregressive and latent generation
DynScene: Scalable Generation of Dynamic Robotic Manipulation Scenes for Embodied AI
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DynScene, a diffusion-based framework that jointly generates static robot-object scenes and residual end-effector action trajectories from text instructions, filters them in Isaac Sim, and uses them to train PerAct agents on ARNOLD.
Problem
Collecting large-scale, high-quality robotic manipulation data with realistic object interactions is slow, expensive and manual; existing generative methods produce either static scenes or robot actions in isolation without physical consistency.
Approach
Decomposes dynamic scene generation into a static scene diffusion model (with layout sampling and quaternion quantization for stability) and a separate action diffusion model conditioned on the static scene, using residual end-effector coordinates so that actions can be reused across scenes for augmentation.
Key results
- DynScene generates dynamic scenes in 2.52 s with 69.50% success vs. human experts' 67.50 s and 37.50% success (26.8× faster, 1.84× higher success).
- PerAct-PSA trained on ARNOLD plus DynScene data reaches 45.62% average success vs. 41.46% with ARNOLD alone.
- Action diversity improves over ARNOLD: FD 30.32 vs 28.39, DTW 55.25 vs 47.58, SC 3.62 vs 2.83.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Generate dynamic scenes with DynScene instead of collecting human demonstrations | robotic manipulation scene generation · ARNOLD, 8 tasks average | generation time: 67.50 -> 2.52 ± 0.02 sec (26.8× faster) | Helped |
| Generate dynamic scenes with DynScene instead of human demonstrations | robotic manipulation scene generation · ARNOLD, 8 tasks average | generation success rate: 37.50 -> 69.50 | Helped |
| Jointly generate static scenes and robot actions rather than pairing DiffuScene scenes with ARNOLD actions | scene and action generation · Static scene generation | success rate: 20.04 -> 69.50 | Helped |
| Train PerAct with ARNOLD plus DynScene scenes instead of ARNOLD alone | PerAct · ARNOLD test dataset, 8 tasks average | success rate: 29.14 -> 34.79 | Helped |
| Train PerAct-PSA with ARNOLD plus DynScene scenes instead of ARNOLD alone | PerAct-PSA · ARNOLD test dataset, 8 tasks average | success rate: 41.46 -> 45.62 | Helped |
| Train with ARNOLD plus DynScene scenes instead of ARNOLD alone | PerAct · Unseen objects | success rate: 14.53 -> 17.29 | Helped |
| Train with ARNOLD plus DynScene scenes instead of ARNOLD alone | PerAct · Unseen scenes | success rate: 19.00 -> 22.92 | Helped |
| Train with ARNOLD plus DynScene scenes instead of ARNOLD alone | PerAct · Novel states | success rate: 4.58 -> 3.30 | Hurt |
| Train with ARNOLD plus DynScene scenes instead of ARNOLD alone | PerAct · Any state | success rate: 11.01 -> 13.54 | Helped |
| Generate actions with DynScene rather than using ARNOLD actions | action generation · ARNOLD, 8 tasks average | Fréchet Distance: 28.39 -> 30.32 | Helped |
| Generate actions with DynScene rather than using ARNOLD actions | action generation · ARNOLD, 8 tasks average | Dynamic Time Warping: 47.58 -> 55.25 | Helped |
| Generate actions with DynScene rather than using ARNOLD actions | action generation · ARNOLD, 8 tasks average | Spatial Coverage: 2.83 -> 3.62 | Helped |
Surprising findings
- DiffuScene achieves slightly higher entropy than DynScene in unconditional object and scene generation, so DynScene is not better on every diversity metric.
- Augmenting training with DynScene data does not help novel-state generalization (3.30 vs 4.58 for PerAct).
Limitations
- Evaluated only on the single ARNOLD benchmark.
- Generalization to novel, out-of-distribution states remains weak.
- Experiments use only 8 tasks with success judged by Isaac Sim thresholds.
Experimental setup
Datasets: ARNOLD
Benchmarks: ARNOLD
Metrics: success rate Fréchet Distance Dynamic Time Warping Spatial Coverage generation time entropy
Baselines: ARNOLD human-collected data DiffuScene PerAct BC-Lang-CNN BC-Lang-ViT PerAct-PSA
Builds on: DiffuScene PerAct ARNOLD
Closest papers by meaning
- ManiTaskGen: A Comprehensive Task Generator for Benchmarking and Improving Vision-Language Agents on Embodied Decision-Making (NeurIPS 2026 · similarity 0.88)
- AffordDP: Generalizable Diffusion Policy with Transferable Affordance (CVPR 2025 · similarity 0.88)
- RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation (ICLR 2025 · similarity 0.88)
- DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation (NeurIPS 2025 · similarity 0.88)
- Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.88)
- Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.