Research map / Video / Video generation
PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PhyT2V, a training-free iterative prompt self-refinement loop where an LLM uses chain-of-thought and step-back reasoning, plus video captioning and a physics evaluator, to make text-to-video outputs follow physical rules.
Problem
Text-to-video diffusion models often violate real-world physical rules, especially for out-of-distribution prompts, and existing fixes need retraining or extra conditioning inputs.
Approach
Iteratively refine the T2V text prompt with an LLM: in parallel, extract physical rules from the prompt and identify mismatches between a caption of the generated video and the prompt (local CoT), then use step-back prompting with these outputs and an evaluator score to write a refined prompt, without retraining or extra inputs.
Key results
- PhyT2V raises CogVideoX-2B PC from 0.13 to 0.29 and SA from 0.22 to 0.42 on VideoPhy over four rounds (up to 2.3x SA improvement).
- On VideoPhy with CogVideoX-5B, PhyT2V reaches PC 0.42 and SA 0.59 versus 0.33/0.41 for the ChatGPT-4 enhancer and 0.25/0.39 for Promptist.
- Most gains occur in the first two refinement rounds, so the authors suggest 3-4 rounds suffice.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply iterative PhyT2V prompt refinement | CogVideoX-5B · VideoPhy | PC: 0.26 -> 0.42 | Helped |
| Apply iterative PhyT2V prompt refinement | CogVideoX-2B · VideoPhy | SA: 0.22 -> 0.42 | Helped |
| Apply iterative PhyT2V prompt refinement | OpenSora · PhyGenBench | PC: 0.21 -> 0.26 | Helped |
| Apply iterative PhyT2V prompt refinement | VideoCrafter · PhyGenBench | SA: 0.27 -> 0.42 | Helped |
| Use PhyT2V instead of ChatGPT 4 prompt enhancement | CogVideoX-5B · VideoPhy | PC: 0.33 -> 0.42 | Helped |
| Remove physical rule reasoning (Step 1) (ablation) | PhyT2V · VideoPhy | PC | Hurt |
| Add physical rule reasoning (Step 1) (ablation) | PhyT2V · VideoPhy | PC | Helped |
| Remove mismatch reasoning (Step 2) (ablation) | PhyT2V · VideoPhy | SA | Hurt |
| Add mismatch reasoning (Step 2) (ablation) | PhyT2V · VideoPhy | SA | Helped |
Surprising findings
- Direct CoT reasoning over the video is judged hard because of cross-modal alignment, so the method reasons only in text by captioning the video first.
- Promptist yields a higher PC than ChatGPT-4 enhancement on PhyGenBench with CogVideoX-5B (0.32 vs 0.27).
Limitations
- Relies on a VideoCon-Physics evaluator as feedback signal, which is itself a model-based judgment.
- Performance gains plateau and some complex physical scenes remain unresolved after multiple rounds.
Experimental setup
Datasets: VideoPhy PhyGenBench
Benchmarks: VideoPhy PhyGenBench VBench
Metrics: physical common sense (PC) semantic adherence (SA) VBench dimensions
Baselines: ChatGPT-4 prompt enhancer Promptist vanilla T2V model (round 1)
Builds on: chain-of-thought prompting step-back prompting VideoPhy Promptist
Closest papers by meaning
- VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models (NeurIPS 2025 · similarity 0.89)
- Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation (ICML 2025 · similarity 0.88)
- WISA: World simulator assistant for physics-aware text-to-video generation (NeurIPS 2025 · similarity 0.88)
- T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedback (NeurIPS 2024 · similarity 0.87)
- Mimir: Improving Video Diffusion Models for Precise Text Understanding (CVPR 2025 · similarity 0.87)
- T2V-Turbo-v2: Enhancing Video Model Post-Training through Data, Reward, and Conditional Guidance Design (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.