Research map / Video / Video generation
VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills physics understanding from video self-supervised foundation models (VideoMAEv2) into text-to-video diffusion transformers (CogVideoX) via a token relation distillation loss over spatial and temporal token similarities.
Problem
Text-to-video diffusion models generate physically implausible content, and their internal representations understand physics much worse than video SSL models; naive REPA degrades pretrained T2V models when finetuning.
Approach
Instead of hard feature alignment as in REPA, align pairwise token similarity relations (spatial within frames and temporal across frames) between VDM hidden states and a video foundation model, giving soft guidance suitable for finetuning pretrained T2V models.
Key results
- VideoREPA-5B reaches PC 40.1 overall on VideoPhy versus 31.4 for CogVideoX-5B (about 24.1% relative improvement) and state-of-the-art among compared methods.
- On VideoPhy2, VideoREPA raises PC from 67.97 to 72.54 over CogVideoX with SA unchanged at 21.02.
- Naive REPA finetuning degrades CogVideoX semantic quality, while TRD loss improves VideoPhy PC for VideoREPA-2B from 23.2 to 29.7 with SA 63.6 to 64.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune CogVideoX-2B with TRD | CogVideoX-2B · VideoPhy Overall | PC: 25.6 -> 29.7 | Helped |
| Fine-tune CogVideoX-5B with TRD | CogVideoX-5B · VideoPhy Overall | PC: 32.3 -> 40.1 | Helped |
| Fine-tune CogVideoX with TRD | CogVideoX · VideoPhy2 | PC: 67.97 -> 72.54 | Helped |
| Fine-tune CogVideoX with TRD | CogVideoX · VideoPhy2 | SA: 21.02 -> 21.02 | No clear effect |
| Add TRD to diffusion-loss-only training (ablation) | VideoREPA-2B · VideoPhy | PC: 23.2 -> 29.7 | Helped |
| Add TRD to diffusion-loss-only training (ablation) | VideoREPA-2B · VideoPhy | SA: 63.6 -> 64.2 | Helped |
| Add temporal TRD (ablation) | VideoREPA-2B with spatial TRD · VideoPhy | PC: 27.3 -> 29.7 | Helped |
| Add spatial TRD (ablation) | VideoREPA-2B with temporal TRD · VideoPhy | PC: 27.9 -> 29.7 | Helped |
| Use only spatial or only temporal TRD instead of both (ablation) | VideoREPA-2B · VideoPhy | SA: 64.2 -> 61.0 | Hurt |
| Replace VideoMAEv2 with VideoMAE as alignment target (ablation) | VideoREPA-2B · VideoPhy | PC: 29.7 -> 26.2 | Hurt |
| Replace VideoMAEv2 with V-JEPA as alignment target (ablation) | VideoREPA-2B · VideoPhy | PC: 29.7 -> 24.7 | Hurt |
| Replace VideoMAEv2 with OmniMAE as alignment target (ablation) | VideoREPA-2B · VideoPhy | PC: 29.7 -> 24.7 | Hurt |
Surprising findings
- Naive REPA finetuning of CogVideoX significantly degrades semantic quality, whereas the softer relational loss does not.
- Aligning only spatial or only temporal relations lowers semantic adherence despite preserving some physics gains.
Limitations
- Validated only for finetuning; potential for pretraining VDMs remains unexplored due to compute.
- Evaluation relies on automated VLM-based raters (VideoCon-Physics, VideoPhy2-AutoEval) and a logistic probe on Physion.
- Requires interpolating VDM features and downscaling inputs for the VFM, adding design trade-offs.
Experimental setup
Datasets: OpenVid VideoPhy VideoPhy2 Physion
Benchmarks: VideoPhy VideoPhy2 Physion
Metrics: Semantic Adherence (SA) Physical Commonsense (PC) OCP prediction accuracy
Baselines: CogVideoX REPA WISA PhyT2V HunyuanVideo Cosmos-Diffusion-7B VideoCrafter2 LaVie DreamMachine
Builds on: REPA CogVideoX VideoMAEv2 LoRA
Closest papers by meaning
- PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generation (CVPR 2025 · similarity 0.89)
- WISA: World simulator assistant for physics-aware text-to-video generation (NeurIPS 2025 · similarity 0.89)
- Physics-Informed Video Generation via Mixture-of-Experts Latent Alignment (NeurIPS 2026 · similarity 0.88)
- Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generation (NeurIPS 2025 · similarity 0.87)
- PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation (NeurIPS 2025 · similarity 0.87)
- Latent Intuitive Physics: Learning to Transfer Hidden Physics from A 3D Video (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.