Research map / Video / Video generation
PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Drop
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Post-trains Open-Sora v1.2 on about 5k simulated Kubric freefall videos (PSFT) and adds reward-based fine-tuning (ORO) with segmentation, optical flow, and depth rewards, evaluated on a new real-world freefall benchmark (PisaBench).
Problem
Video generation models are visually impressive but not physically accurate, failing even at simple object freefall dynamics in image-to-video generation; post-training for physical accuracy is under-studied.
Approach
Supervised fine-tuning on a small set of simulated dropping videos induces correct falling behavior in a pretrained video diffusion model, and a reward-based stage (VADER-style, gradients through all denoising steps) using SAM 2 mask IoU, RAFT flow, or Depth-Anything-V2 depth against simulator ground truth further improves physical accuracy.
Key results
- Open-Sora + PSFT (5k samples) outperforms all evaluated proprietary and open models on real and simulated test sets; e.g., real-set L2 0.076 vs 0.138 for best open baseline CogVideoX-5B-I2V and 0.155 for Kling-V1.5.
- Flow and depth rewards (ORO) further lower L2 and CD (e.g., real L2 0.067, CD 0.159 for depth reward); segmentation reward gives the best real IoU (0.142).
- Models generalize poorly to out-of-distribution depths and heights (ID vs OOD time error 0.091 vs 0.187), and the KS test rejects correct p(t|y) alignment in all 50/50 cases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Open-Sora on 5,000 simulated dropping videos (PSFT) | Open-Sora v1.2 · PisaBench Real, Sim (Seen), and Sim (Unseen) | L2 | Helped |
| Add optical-flow reward optimization to PSFT | Open-Sora + PSFT · PisaBench Real, Sim (Seen), and Sim (Unseen) | L2: 0.076 -> 0.067; 0.036 -> 0.026; 0.028 -> 0.022 | Helped |
| Add optical-flow reward optimization to PSFT | Open-Sora + PSFT · PisaBench Real, Sim (Seen), and Sim (Unseen) | IoU: 0.139 -> 0.136; 0.165 -> 0.122; 0.129 -> 0.071 | Hurt |
| Add depth reward optimization to PSFT | Open-Sora + PSFT · PisaBench Real, Sim (Seen), and Sim (Unseen) | L2: 0.076 -> 0.067; 0.036 -> 0.031; 0.028 -> 0.022 | Helped |
| Add segmentation reward optimization to PSFT | Open-Sora + PSFT · PisaBench Real, Sim (Seen), and Sim (Unseen) | IoU: 0.139 -> 0.142; 0.165 -> 0.170; 0.129 -> 0.145 | Helped |
| Add depth reward optimization to PSFT | Open-Sora + PSFT · PisaBench Real, Sim (Seen), and Sim (Unseen) | IoU: 0.139 -> 0.129; 0.165 -> 0.124; 0.129 -> 0.096 | Hurt |
| Train with out-of-distribution depth-height combinations rather than in-distribution combinations | PSFT model · Simulated depth-height generalization | L2: 0.036 -> 0.044 | Hurt |
| Train with out-of-distribution depth-height combinations rather than in-distribution combinations | PSFT model · Simulated depth-height generalization | Chamfer: 0.088 -> 0.143 | Hurt |
| Train with out-of-distribution depth-height combinations rather than in-distribution combinations | PSFT model · Simulated depth-height generalization | IOU: 0.155 -> 0.049 | Hurt |
| Train with out-of-distribution depth-height combinations rather than in-distribution combinations | PSFT model · Simulated depth-height generalization | Time Error: 0.091 -> 0.187 | Hurt |
| Use Open-Sora pretraining rather than random initialization of its denoising network (ablation) | Open-Sora adapted on 5k samples · PisaBench | physical accuracy | Helped |
| Vary PSFT training dataset size from 500 to 20,000 samples; 5,000 samples reached optimal results (ablation) | Open-Sora + PSFT · PisaBench | physical accuracy | Helped |
Surprising findings
- Random initialization of the denoiser (keeping only pretrained compressor) fails to match pretrained post-training results, showing pretraining is critical.
- Post-trained model's generated dropping-time distribution is misaligned with ground truth in all 50 tested images despite faithful 3D trajectories.
Limitations
- Single-image conditioning creates depth/scale ambiguity.
- Metrics only capture spatial properties, not appearance; objects can become blurry or disappear after collision.
- Sim-to-real gap leads to smaller gains on real data than simulation.
Experimental setup
Datasets: Kubric simulated dropping videos built from Google Scanned Objects PisaBench real-world videos
Benchmarks: PisaBench real PisaBench simulated seen PisaBench simulated unseen
Metrics: Trajectory L2 Chamfer Distance IoU time error KS test p-value
Baselines: Sora Kling-V1 Kling-V1.5 Runway Gen3 CogVideoX-5B-I2V DynamiCrafter Pyramid-Flow Open-Sora
Builds on: VADER Kubric Open-Sora v1.2 SAM 2 RAFT
Closest papers by meaning
- How Far Is Video Generation from World Model: A Physical Law Perspective (ICML 2025 · similarity 0.90)
- VideoPhy: Evaluating Physical Commonsense for Video Generation (ICLR 2025 · similarity 0.88)
- PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation (NeurIPS 2025 · similarity 0.87)
- MotionCraft: Physics-Based Zero-Shot Video Generation (NeurIPS 2024 · similarity 0.87)
- LongScape: Advancing Long-Horizon Embodied World Models with Context-Aware MoE (NeurIPS 2026 · similarity 0.86)
- Empowering World Models with Reflection for Embodied Video Prediction (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.