Research map / Video / Video generation
Towards Physical Understanding in Video Generation: A 3D Point Regularization Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes latent video diffusion models (I2VGen-XL and CogVideoX 1.5) on a 3D point-augmented video dataset (PointVid) with point-trajectory regularization to reduce shape morphing in generated RGB videos.
Problem
Video diffusion models learn 2D pixel motion without 3D shape awareness, causing object morphing and non-physical deformation, especially in contact-rich hand-object interactions.
Approach
Augment 2D videos with pixel-aligned 3D point trajectories (from SpaTracker on Grounded-SAM2 masks), jointly diffuse video and point channels with cross-attention, and regularize the denoised point trajectories with reconstruction (fidelity, velocity, acceleration) and local rigidity losses so RGB generation inherits shape and motion consistency.
Key results
- Ours (DiT) reaches physical commonsense 0.41384 on VideoPhy vs 0.36549 for CogVideoX 1.5; Ours (UNet) reaches 0.37805 vs 0.32876 for I2VGen-XL.
- Point MSE falls from 0.2125 (untrained) to 0.0031 (no regularization) and 0.0020 (with regularization).
- User study (10 labelers): ours preferred 86.2% vs 13.8% for I2VGen-XL on physical realism questions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 3D point augmentation and regularization | I2VGen-XL · VBench | subject consistency: 0.83704 -> 0.96003 | Helped |
| add 3D point augmentation and regularization | I2VGen-XL · VBench | aesthetic quality: 0.43140 -> 0.42246 | Hurt |
| add 3D point augmentation and regularization | CogVideoX 1.5 · VBench | motion smoothness: 0.98798 -> 0.99201 | Helped |
| add 3D point augmentation and regularization | CogVideoX 1.5 · VideoPhy | physical commonsense: 0.36549 -> 0.41384 | Helped |
| add point regularization (ablation) | point-augmented model · 3D point trajectories | MSE: 0.0031 -> 0.0020 | Helped |
| add point augmentation (ablation) | full UNet model · VBench | subject consistency: 0.9434 -> 0.9681 | Helped |
| add point augmentation (ablation) | full UNet model · VBench | background consistency: 0.9492 -> 0.9639 | Helped |
| add point regularization (ablation) | full UNet model · VBench | subject consistency: 0.9544 -> 0.9681 | Helped |
| add channel-wise cross-attention (ablation) | full UNet model · VBench | subject consistency: 0.9332 -> 0.9681 | Helped |
| add diffusion loss (ablation) | regularized UNet model · VBench | subject consistency: 0.7166 -> 0.9681 | Helped |
| add diffusion loss (ablation) | regularized UNet model · VBench | background consistency: 0.9254 -> 0.9639 | Helped |
Surprising findings
- Dropping the diffusion loss while training with regularization completely degrades results, as 3D information dominates semantics.
- Ours (DiT) has essentially unchanged subject and background consistency vs CogVideoX 1.5 (0.91768 vs 0.91754 SC).
Limitations
- Degree of 3D awareness is constrained by the resolution of sparse 3D point tracks (grid size 80).
- Generation quality limited by the base diffusion backbone and its scale.
- Regularization loss trades off motion magnitude against smoothness.
Experimental setup
Datasets: PointVid (100K videos from a public video dataset and internal data, paired with SpaTracker 3D point tracks) Kubric
Benchmarks: VBench VideoPhy
Metrics: subject consistency background consistency motion smoothness aesthetic quality imaging quality physical commonsense MSE of 3D points
Baselines: I2VGen-XL CogVideoX 1.5 SVD DynamiCrafter LTX Video
Builds on: I2VGen-XL CogVideoX SpaTracker DDIM
Closest papers by meaning
- GeoVideo: Introducing Geometric Regularization into Video Generation Model (NeurIPS 2025 · similarity 0.90)
- Diffusion4D: Fast Spatial-temporal Consistent 4D generation via Video Diffusion Models (NeurIPS 2024 · similarity 0.88)
- Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models (CVPR 2024 · similarity 0.87)
- World-consistent Video Diffusion with Explicit 3D Modeling (CVPR 2025 · similarity 0.87)
- 4Diffusion: Multi-view Video Diffusion Model for 4D Generation (NeurIPS 2024 · similarity 0.87)
- Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.