Research map / Video / Video generation
Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PhysHPO, a four-level (instance, state, motion, semantic) hierarchical cross-modal DPO plus an automated data selection pipeline over OpenVidHD-0.4M, to improve physical plausibility of CogVideoX video generators.
Problem
Text-to-video models often generate physically implausible videos, and existing DPO methods for video only align coarse instance-level preferences while SFT approaches require costly dataset construction.
Approach
Combines hierarchical preference losses at instance, state (boundary-frame swapping), motion (optical-flow-style structural conditions), and semantic (VLM-edited captions) granularities, trained on physics-relevant real-world videos selected by a VLM-based reality filter and LLM-scored evolving-prompt physics scoring with diversity selection.
Key results
- On CogVideoX-5B, PhysHPO raises VideoPhy overall accuracy from 39.6 to 45.9 and PhyGenBench overall from 0.45 to 0.61.
- On CogVideoX-2B, PhysHPO improves VideoPhy overall from 18.6 to 25.9 and VBench Total from 81.6 to 82.5.
- On HunyuanVideo, PhysHPO raises VideoPhy overall from 33.4 to 41.6 and PhyGenBench overall from 45.6 to 54.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PhysHPO instead of Vanilla DPO | CogVideoX-2B · VideoPhy | Over.: 19.2 -> 25.9 | Helped |
| use PhysHPO instead of Vanilla DPO | CogVideoX-2B · PhyGenBench | Overall: 0.46 -> 0.53 | Helped |
| use PhysHPO instead of Vanilla DPO | CogVideoX-2B · VBench | Total: 82.2 -> 82.5 | Helped |
| use PhysHPO instead of Vanilla DPO | CogVideoX-5B · VideoPhy | Over.: 41.3 -> 45.9 | Helped |
| use PhysHPO instead of Vanilla DPO | CogVideoX-5B · PhyGenBench | Overall: 0.54 -> 0.61 | Helped |
| use PhysHPO instead of Vanilla DPO | CogVideoX-5B · VBench | Total: 82.4 -> 82.8 | Helped |
| use PhysHPO instead of PhyT2V | HunyuanVideo · VideoPhy | Over.: 36.3 -> 41.6 | Helped |
| use PhysHPO instead of PhyT2V | HunyuanVideo · PhyGenBench | Overall: 46.3 -> 54.4 | Helped |
| use PhysHPO instead of PhyT2V | HunyuanVideo · VBench | Total: 80.5 -> 81.9 | Helped |
| add semantic-level loss (ablation) | PhysHPO · VideoPhy | Over.: 45.1 -> 45.9 | Helped |
| add semantic-level and motion-level losses (ablation) | PhysHPO · VideoPhy | Over.: 43.6 -> 45.9 | Helped |
| add hierarchical losses beyond instance-level loss alone (ablation) | PhysHPO · VideoPhy | Over.: 41.9 -> 45.9 | Helped |
Surprising findings
- Vanilla DPO with the selected data gives minimal or negative gains on some metrics (e.g., CogVideoX-5B VideoPhy FF drops from 43.6 to 51.8 only partly and SF drops from 53.1 to 50.0 for vanilla DPO is inconsistent), indicating plain DPO is insufficient.
- Caption-based selection reduces GPU memory by 18.1% and GPU hours by 27.6% compared with video-based selection.
Limitations
- High computational cost of training large video generation models limits accessibility.
- Data selection pipeline is intentionally simple and may be suboptimal for specific tasks.
- Evaluated only on CogVideoX and HunyuanVideo base models and a limited set of physics benchmarks.
Experimental setup
Datasets: OpenVidHD-0.4M WISA-32K
Benchmarks: VideoPhy PhyGenBench VBench
Metrics: SS SF FF Overall accuracy PhyGenBench PCA by category VBench Total/Quality/Semantic user study MOS
Baselines: PhyT2V Vanilla DPO SFT WISA SynVideo CogVideoX-2B CogVideoX-5B HunyuanVideo
Builds on: Diffusion-DPO VideoDPO Deita WISA
Closest papers by meaning
- PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation (NeurIPS 2025 · similarity 0.88)
- VideoDPO: Omni-Preference Alignment for Video Diffusion Generation (CVPR 2025 · similarity 0.88)
- DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models (NeurIPS 2025 · similarity 0.87)
- Physics-Informed Video Generation via Mixture-of-Experts Latent Alignment (NeurIPS 2026 · similarity 0.87)
- WISA: World simulator assistant for physics-aware text-to-video generation (NeurIPS 2025 · similarity 0.87)
- $\mathcal{V}ista\mathcal{DPO}$: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Models (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.