Research map / Video / Video segmentation and tracking
PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PlaySlot, an object-centric video prediction model that infers slots and hybrid discrete-continuous latent actions from unlabeled video via an inverse dynamics module, enabling controllable multi-future prediction and sample-efficient behavior learning.
Problem
Object-centric video prediction models are either deterministic or require precise action labels, which limits use of unlabeled video for controllable world models in robotics.
Approach
Infers object slots (SAVi) and latent actions from consecutive slot pairs via an inverse dynamics module with a hybrid parameterization (VQ action prototype plus continuous variability), and conditions a transformer object-centric predictor (cOCVP) on these latent actions for controllable prediction and latent-policy learning.
Key results
- On BlockPush, PlaySlot reaches 21.41 PSNR / 0.066 LPIPS vs. 21.18 PSNR for CADDY and 17.26 for OCVP.
- On GridShapes with two objects, PlaySlot achieves 54.09 PSNR and 0.996 SSIM, versus 39.16 PSNR for CADDY.
- With as few as 200 expert demonstrations on ButtonPress, PlaySlot matches the CADDY Oracle performance, and it outperforms LAPO on BlockPush across data regimes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PlaySlot instead of CADDY | BlockPush video prediction · BlockPush | PSNR: 21.18 -> 21.41 | Helped |
| use PlaySlot instead of SVG | ButtonPress video prediction · ButtonPress | PSNR: 32.23 -> 26.03 | Hurt |
| use PlaySlot instead of CADDY | GridShapes video prediction (2 objects) · GridShapes (2 objects) | PSNR: 39.16 -> 54.09 | Helped |
| use PlaySlot instead of SVG | Sketchy video prediction · Sketchy | PSNR: 24.24 -> 24.43 | Helped |
| use continuous rather than hybrid latent actions (ablation) | PlaySlot on BlockPush · BlockPush random exploration | PSNR: 26.64 -> 26.24 | Hurt |
| use discrete rather than hybrid latent actions (ablation) | PlaySlot on BlockPush · BlockPush random exploration | PSNR: 26.64 -> 20.60 | Hurt |
| condition on ground-truth actions rather than inferred hybrid actions (ablation) | PlaySlot on BlockPush · BlockPush random exploration | PSNR: 26.64 -> 27.77 | Helped |
| use continuous rather than hybrid latent actions (ablation) | PlaySlot on BlockPush · BlockPush expert demonstrations | PSNR: 21.41 -> 22.00 | Helped |
| use discrete rather than hybrid latent actions (ablation) | PlaySlot on BlockPush · BlockPush expert demonstrations | PSNR: 21.41 -> 18.00 | Hurt |
| condition on ground-truth actions rather than inferred hybrid actions (ablation) | PlaySlot on BlockPush · BlockPush expert demonstrations | PSNR: 21.41 -> 22.30 | Helped |
| use 5 rather than 8 action prototypes (ablation) | PlaySlot · BlockPush | PSNR: 21.41 -> 21.26 | Hurt |
| use 10 rather than 8 action prototypes (ablation) | PlaySlot · BlockPush | PSNR: 21.41 -> 21.36 | Hurt |
Surprising findings
- Deterministic object-centric models (SlotFormer, OCVP) perform poorly on BlockPush because they cannot infer agent actions and average over futures.
- A variant with unconstrained continuous latent actions outperforms PlaySlot on expert demonstrations, attributed to action distribution shift.
Limitations
- SAVi decomposition fails on complex real-world scenes, limiting scope to tabletop simulations and simple real-world settings.
- A single latent action jointly encodes simultaneous robot actions, limiting control precision.
- Performance drops on expert demonstrations due to action distribution shift from random exploration training data.
Experimental setup
Datasets: BlockPush ButtonPress GridShapes Sketchy MetaWorld
Benchmarks: BlockPush ButtonPress GridShapes Sketchy
Metrics: PSNR SSIM LPIPS success rate
Baselines: SVG CADDY SlotFormer OCVP LAPO DreamerV3 SOLD
Builds on: SAVi SlotFormer OCVP CADDY Slot Attention
Closest papers by meaning
- Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation (ICLR 2025 · similarity 0.89)
- Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training (NeurIPS 2024 · similarity 0.88)
- Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations (ICML 2025 · similarity 0.88)
- Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations (NeurIPS 2025 · similarity 0.88)
- VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation (NeurIPS 2024 · similarity 0.88)
- EA-WM: Event-Aware Generative World Model with Structured Kinematic-to-Visual Action Fields (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.