Research map / Video / Video generation
PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PhysCtrl, a diffusion-based generative physics network that predicts 3D point trajectories from physics parameters and forces, then uses them to drive image-to-video models.
Problem
Image-to-video generators lack physical plausibility and explicit control over physics parameters and external forces, while physics simulators are costly, hyperparameter-sensitive, and material-specific.
Approach
A diffusion transformer with alternating spatial and temporal attention over point-cloud trajectories, conditioned on force, drag point, Young's modulus/Poisson's ratio, floor height and material token, with an MPM-derived deformation-gradient physics loss; its trajectories are projected to pixel tracks to control a pretrained video model (DaS).
Key results
- Trajectory generation reaches 77.59% vIoU, 0.0028 CD and 0.0015 Corr on the elastic test set, versus 53.78% vIoU for MDM and 24.92% for M2V.
- GPT-4o rating for image-to-video gives Ours 4.5 SA / 4.5 PC / 4.3 VQ versus 3.8 / 3.7 / 3.6 for Wan2.1 and 3.2 / 3.2 / 3.1 for CogVideoX.
- In a 35-person user study, 81.0% of physics-plausibility votes went to PhysCtrl, versus 10.2% for Wan.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PhysCtrl-generated trajectories to condition image-to-video generation instead of text-only or coarse 2D trajectory controls | Image-to-video generation · Image-to-video generation evaluation | SA: 3.8 -> 4.5 | Helped |
| Use PhysCtrl-generated trajectories to condition image-to-video generation instead of text-only or coarse 2D trajectory controls | Image-to-video generation · Image-to-video generation evaluation | PC: 3.7 -> 4.5 | Helped |
| Use PhysCtrl-generated trajectories to condition image-to-video generation instead of text-only or coarse 2D trajectory controls | Image-to-video generation · Image-to-video generation evaluation | VQ: 3.6 -> 4.3 | Helped |
| Use PhysCtrl trajectory diffusion instead of MDM | Elastic trajectory generation · Elastic trajectory test set | vIoU: 53.78% -> 77.59% | Helped |
| Use PhysCtrl trajectory diffusion instead of MDM | Elastic trajectory generation · Elastic trajectory test set | CD: 0.0159 -> 0.0028 | Helped |
| Use PhysCtrl trajectory diffusion instead of MDM | Elastic trajectory generation · Elastic trajectory test set | Corr: 0.0240 -> 0.0015 | Helped |
| Add spatial attention (ablation) | PhysCtrl trajectory generator · Elastic trajectory test set | vIoU: 33.76% -> 77.59% | Helped |
| Add temporal attention (ablation) | PhysCtrl trajectory generator · Elastic trajectory test set | vIoU: 53.63% -> 77.59% | Helped |
| Add physics loss (ablation) | PhysCtrl trajectory generator · Elastic trajectory test set | vIoU: 76.30% -> 77.59% | Helped |
| Use 8192 rather than 2048 points with traditional simulator trajectories (ablation) | Traditional-simulator-conditioned video generation · Video generation using trajectories | VQ: 4.4 -> 4.2 | Hurt |
| Use 2048 PhysCtrl-generated trajectory points rather than 2048 traditional-simulator points (ablation) | Video generation · Video generation using trajectories | SA: 4.3 -> 4.5 | Helped |
| Use 2048 PhysCtrl-generated trajectory points rather than 2048 traditional-simulator points (ablation) | Video generation · Video generation using trajectories | PC: 4.3 -> 4.5 | Helped |
Surprising findings
- Poisson's ratio has negligible influence on generated trajectories, similar to PhysDreamer.
- Using 8192 points did not improve video generation over 2048 points.
Limitations
- Focuses on single-object dynamics for four material types; multi-object interaction only preliminarily studied.
- Struggles with thin structures and complex internal deformations.
- Video model may not follow trajectories that conflict with its visual prior (e.g., stiff material appearing soft).
Experimental setup
Datasets: ObjaverseXL MPM simulation data rigid-body simulation data
Benchmarks: in-house PhysCtrl test set of image-to-video cases
Metrics: Semantic Adherence Physical Commonsense Video Quality vIoU Chamfer Distance Correspondence L2 error user preference rate MAE of log10(E)
Baselines: Wan2.1-I2V-14B CogVideoX-5B DragAnything ObjCtrl-2.5D Motion2VecSets MDM M2V
Builds on: MPM MDM CogVideoX DaS LGM
Closest papers by meaning
- Force Prompting: Video Generation Models Can Learn And Generalize Physics-based Control Signals (NeurIPS 2025 · similarity 0.91)
- How Far Is Video Generation from World Model: A Physical Law Perspective (ICML 2025 · similarity 0.89)
- PhysGen3D: Crafting a Miniature Interactive World from a Single Image (CVPR 2025 · similarity 0.89)
- DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priors (AAAI 2025 · similarity 0.89)
- VideoPhy: Evaluating Physical Commonsense for Video Generation (ICLR 2025 · similarity 0.89)
- PhyMotion: Structured 3D Motion Reward for Physics-Grounded Human Video Generation (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.