Research map / Image generation / Autoregressive and latent generation
EC-Diffuser: Multi-Object Manipulation via Entity-Centric Behavior Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EC-Diffuser, a diffusion-based behavioral cloning method that jointly generates DLP object-centric particle states and actions with an entity-centric Transformer, achieving compositional zero-shot generalization in multi-object manipulation.
Problem
Learning multi-object manipulation policies from high-dimensional image observations under combinatorial state and behavior complexity, with compositional generalization to unseen object counts and goal configurations from limited offline demonstrations.
Approach
Combine an unsupervised object-centric image representation (DLPv2 particles) with a permutation-equivariant entity-centric Transformer denoiser that co-generates future particle states and actions via DDPM diffusion, with AdaLN conditioning, used in MPC-style goal-conditioned BC.
Key results
- On PushCube with 2 and 3 objects, EC-Diffuser achieves 0.917 and 0.894 success rate versus 0.388 and 0.668 for EC Diffusion Policy, the best baseline.
- Trained on 3 cubes, EC-Diffuser generalizes zero-shot to 6 cubes with 0.711 success fraction versus 0.122 for EC Diffusion Policy.
- On FrankaKitchen, EC-Diffuser reaches 3.031 goals versus 2.384 for VQ-BeT (reproduced) and 2.360 for EIT+BC; with Slot Attention it reaches 3.340.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EC-Diffuser instead of the strongest listed baseline | PushCube · PushCube, 3 objects | success rate: 0.668 $\pm$ 0.169 -> 0.894 $\pm$ 0.025 | Helped |
| Use EC-Diffuser instead of the strongest listed baseline | PushT · PushT, 3 objects | average radian difference: 1.538 $\pm$ 0.040 -> 0.805 $\pm$ 0.256 | Helped |
| Use EC-Diffuser instead of the strongest listed baseline | FrankaKitchen · FrankaKitchen | goals reached: 3.046 $\pm$ 0.156 -> 3.031 $\pm$ 0.087 | No clear effect |
| Train with DLP rather than an unstructured representation (ablation) | EC-Diffuser · PushCube, 3 objects | success rate: 0.008 $\pm$ 0.004 -> 0.894 $\pm$ 0.025 | Helped |
| Use diffusion rather than autoregressive generation (ablation) | EC-Diffuser · PushCube, 3 objects | success rate: 0.000 $\pm$ 0.000 -> 0.894 $\pm$ 0.025 | Helped |
| Generate latent states alongside actions (ablation) | EC-Diffuser · PushCube, 3 objects | success rate: 0.529 $\pm$ 0.311 -> 0.894 $\pm$ 0.025 | Helped |
| Use DLP rather than Slot Attention (ablation) | EC-Diffuser · PushCube, 3 objects | success rate: 0.255 $\pm$ 0.055 -> 0.894 $\pm$ 0.025 | Helped |
| Use multiple views rather than only the front view (ablation) | EC-Diffuser · PushCube, 3 objects | success rate: 0.685 $\pm$ 0.021 -> 0.894 $\pm$ 0.025 | Helped |
| Use ground-truth entity states rather than DLP (ablation) | EC-Diffuser · PushCube, 3 objects | success rate: 0.894 $\pm$ 0.025 -> 0.916 $\pm$ 0.025 | Helped |
| Replace DLP with Slot Attention (ablation) | EC-Diffuser · FrankaKitchen | goals reached: 3.031 $\pm$ 0.087 -> 3.340 $\pm$ 0.097 | Helped |
Surprising findings
- Without diffusion (auto-regressive with Chamfer loss), the model fails to learn even single-object tasks.
- Removing state generation (action-only) causes performance to deteriorate rapidly as object count increases.
Limitations
- Relies on quality of object-centric representation; DLP captures different entity notions in FrankaKitchen.
- Long inference time due to iterative diffusion sampling.
- Unsupervised object-centric factorization of real-world natural images remains hard; real-world results are preliminary.
Experimental setup
Datasets: PushCube demonstrations (IsaacGym) PushT demonstrations (IsaacGym) FrankaKitchen demonstrations Language-Table
Benchmarks: PushCube PushT FrankaKitchen Language-Table
Metrics: success rate success fraction average radian difference goals reached
Baselines: VQ-BeT Diffuser EIT+BC EC Diffusion Policy Slot Attention
Builds on: Diffuser Deep Latent Particles DDPM ECRL DDLP
Closest papers by meaning
- AffordDP: Generalizable Diffusion Policy with Transferable Affordance (CVPR 2025 · similarity 0.88)
- Entity-Centric Reinforcement Learning for Object Manipulation from Pixels (ICLR 2024 · similarity 0.87)
- SOLD: Slot Object-Centric Latent Dynamics Models for Relational Manipulation Learning from Pixels (ICML 2025 · similarity 0.87)
- RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation (ICLR 2025 · similarity 0.87)
- PDFactor: Learning Tri-Perspective View Policy Diffusion Field for Multi-Task Robotic Manipulation (CVPR 2025 · similarity 0.86)
- VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.