Research map / Video / Video generation
FreePCA: Integrating Consistency Information across Long-short Frames in Training-free Long Video Generation via Principal Component Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FreePCA, a training-free long video generation method that uses PCA in the temporal dimension to split global and local attention features into consistent appearance and motion-intensity components, then progressively fuses them on top of short-video diffusion models.
Problem
Generating long videos with diffusion models trained only on short clips suffers from distribution shift, and existing training-free methods trade off appearance consistency against visual and motion quality.
Approach
Apply PCA along the temporal dimension of attention features from global (long-frame) and local (sliding-window, short-frame) passes, use cosine similarity between their principal components to split consistent appearance from motion intensity, and progressively inject the consistent global components into local features as the window slides, with mean-statistics reuse of initial noise.
Key results
- On VBench with VideoCrafter2, FreePCA reaches 95.54 subject consistency and 59.72 dynamic degree, versus 93.77 and 45.83 for FreeLong and 91.98 and 52.77 for FreeNoise.
- FreePCA attains the best overall consistency (25.69) and imaging quality (63.70) among compared methods, at 4.7 min inference versus 3.6 min for direct sampling on an RTX 4090.
- Ablation shows that setting Kmax=3 gives the best results; removing PCA or the mean-statistics reuse lowers subject consistency to around 89-94.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FreePCA instead of direct sampling | VideoCrafter2 · VBench (326 prompts) | Subject Consistency: 93.38 -> 95.54 | Helped |
| Use FreePCA instead of direct sampling | VideoCrafter2 · VBench (326 prompts) | Dynamic Degree: 44.45 -> 59.72 | Helped |
| Use FreePCA instead of FreeNoise | VideoCrafter2 · VBench (326 prompts) | Overall Consistency: 25.62 -> 25.69 | Helped |
| Use FreePCA instead of FreeLong | VideoCrafter2 · VBench (326 prompts) | Motion Smoothness: 94.49 -> 96.41 | Helped |
| Use FreePCA instead of FreeLong | VideoCrafter2 · VBench (326 prompts) | Inference Time (min): 4.1 -> 4.7 | Hurt |
| Set Kmax = 1 instead of Kmax = 3 (ablation) | FreePCA · VBench (326 prompts) | Subject Consistency: 95.54 -> 89.13 | Hurt |
| Set Kmax = 5 instead of Kmax = 3 (ablation) | FreePCA · VBench (326 prompts) | Subject Consistency: 95.54 -> 89.40 | Hurt |
| Remove the PCA process (ablation) | FreePCA · VBench (326 prompts) | Subject Consistency: 95.54 -> 94.19 | Hurt |
| Replace cosine-similarity selection with random selection (ablation) | FreePCA · VBench (326 prompts) | Subject Consistency: 95.54 -> 94.11 | Hurt |
| Set k = 3 as a fixed value instead of progressively varying it (ablation) | FreePCA · VBench (326 prompts) | Subject Consistency: 95.54 -> 88.98 | Hurt |
| Replace reuse mean statistics with direct reuse (ablation) | FreePCA · VBench (326 prompts) | Subject Consistency: 95.54 -> 94.21 | Hurt |
| Remove reuse mean statistics (ablation) | FreePCA · VBench (326 prompts) | Subject Consistency: 95.54 -> 89.54 | Hurt |
Surprising findings
- Direct sampling of 64 frames has the lowest overall consistency despite relatively acceptable subject consistency, attributed to semantic inaccuracy.
Limitations
- Evaluated only on 16-frame UNet-based models (VideoCrafter2, LaVie) to 64 frames in the main text; DiT results are only in the supplementary material.
- Relies on a hand-set threshold (Kmax=3) and top-k component selection chosen empirically.
Experimental setup
Datasets: VBench prompts (326)
Benchmarks: VBench
Metrics: Subject Consistency Background Consistency Overall Consistency Motion Smoothness Dynamic Degree Imaging Quality inference time
Baselines: Direct sampling FreeNoise FreeLong
Builds on: FreeNoise FreeLong DDIM
Closest papers by meaning
- FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention (NeurIPS 2024 · similarity 0.89)
- LongDiff: Training-Free Long Video Generation in One Go (CVPR 2025 · similarity 0.86)
- FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling (ICLR 2024 · similarity 0.86)
- Video Diffusion Models are Training-free Motion Interpreter and Controller (NeurIPS 2024 · similarity 0.86)
- Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation (NeurIPS 2024 · similarity 0.85)
- MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.