Research map / Video / Video generation
InterDyn: Controllable Interactive Dynamics with Video Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends frozen Stable Video Diffusion with a ControlNet-style driving-motion branch so it generates interactive object dynamics from an image plus hand/object mask sequences.
Problem
Predicting continuous interactive object dynamics (not just a single static future state) from an image and a driving motion, in complex real-world scenes as well as simple synthetic ones.
Approach
Treat a large pretrained video diffusion model as an implicit physics simulator and renderer: keep SVD frozen and train only a ControlNet-like control branch that conditions generation on pixel-aligned binary masks of the driving entity, so uncontrolled objects' dynamics are generated implicitly.
Key results
- On SSV2, InterDyn 256x384 reaches FVD 22.22 and KVD 2.09 versus 90.30 and 13.68 for CosHand-Autoregressive.
- InterDyn 256x384 achieves LPIPS 0.252 and motion fidelity 0.641, versus 0.408 LPIPS and 0.570 motion fidelity for the best CosHand variant.
- Qualitative CLEVRER experiments show force propagation to uncontrolled objects and counterfactual futures from different control signals.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use InterDyn instead of CosHand-Autoregressive | SSV2 video generation · Something-Something-v2 validation | FVD: 90.30 -> 22.22 | Helped |
| Use InterDyn instead of CosHand-Autoregressive | SSV2 video generation · Something-Something-v2 validation | LPIPS: 0.408 -> 0.252 | Helped |
| Use InterDyn instead of CosHand-Independent | SSV2 video generation · Something-Something-v2 validation | Motion Fidelity: 0.432 -> 0.641 | Helped |
| Increase InterDyn output resolution from 256×256 to 256×384 | InterDyn · Something-Something-v2 validation | SSIM: 0.664 -> 0.680 | Helped |
| Increase InterDyn output resolution from 256×256 to 256×384 | InterDyn · Something-Something-v2 validation | LPIPS: 0.260 -> 0.252 | Helped |
| Increase InterDyn output resolution from 256×256 to 256×384 | InterDyn · Something-Something-v2 validation | FVD: 19.27 -> 22.22 | Hurt |
Surprising findings
- InterDyn's frame-wise image quality is slightly worse than CosHand on some metrics (FID and KID), which the authors attribute to the frozen SVD spatial prior.
- CosHand-Independent beats InterDyn on FID and KID despite much worse video-level metrics.
Limitations
- Object dynamics are not directly controllable; only driving-entity masks are given, and post-interaction dynamics are sometimes inconsistent.
- Evaluated mainly on CLEVRER and SSV2; real-world physical plausibility is assessed qualitatively.
Experimental setup
Datasets: CLEVRER Something-Something-v2 Something-Else
Benchmarks: Something-Something-v2 CLEVRER
Metrics: SSIM PSNR LPIPS FID KID FVD KVD Motion Fidelity
Baselines: CosHand-Independent CosHand-Autoregressive Seer DynamiCrafter
Builds on: Stable Video Diffusion ControlNet EDM CosHand
Closest papers by meaning
- InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction (NeurIPS 2024 · similarity 0.88)
- LongScape: Advancing Long-Horizon Embodied World Models with Context-Aware MoE (NeurIPS 2026 · similarity 0.87)
- HOI-Dyn: Learning Interaction Dynamics for Human-Object Motion Diffusion (NeurIPS 2025 · similarity 0.87)
- PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation (NeurIPS 2025 · similarity 0.87)
- PlaySlot: Learning Inverse Latent Dynamics for Controllable Object-Centric Video Prediction and Planning (ICML 2025 · similarity 0.87)
- VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.