Research map / Video / Video generation
LongDiff: Training-Free Long Video Generation in One Go
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LongDiff, a training-free method that adapts short text-to-video diffusion models (LaVie, VideoCrafter-512) to generate 128-frame videos in one go via position mapping (GROUP/SHIFT) and informative frame selection in temporal attention.
Problem
Short-trained video diffusion models produce temporally inconsistent and detail-poor videos when directly extended to long video generation, due to temporal position ambiguity in relative positional encodings and information dilution in temporal attention.
Approach
Groups relative frame positions into a smaller set of position indices and recovers within-group distinctness via SHIFT operations whose attention is averaged; restricts temporal attention to neighbor frames plus automatically detected key frames (via image entropy and frame differencing) so information is not diluted.
Key results
- On LaVie, LongDiff reaches 98.10 subject consistency and 25.24 overall consistency at 128 frames, versus 92.30 / 24.42 for FreeNoise and 95.16 / 24.56 for FreeLong.
- On VideoCrafter-512 at 128 frames, LongDiff achieves 93.69 subject consistency and 70.03 imaging quality, outperforming FreeNoise (91.43, 68.39) and FreeLong (90.84, 66.62).
- Ablations show removing PM drops LaVie subject consistency from 98.10 to 91.85, and removing IFS drops it to 94.43.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add position mapping (PM) (ablation) | LongDiff on LaVie · VBench prompts, 128 frames | OC: 22.58 -> 25.24 | Helped |
| add informative frame selection (IFS) (ablation) | LongDiff on LaVie · VBench prompts, 128 frames | OC: 23.46 -> 25.24 | Helped |
| replace PM with clipping of out-of-range positions (ablation) | LongDiff on LaVie · VBench prompts, 128 frames | OC: 22.43 vs 25.24 | Hurt |
| replace PM with linear position interpolation (ablation) | LongDiff on LaVie · VBench prompts, 128 frames | OC: 23.01 vs 25.24 | Hurt |
| use GROUP without SHIFT instead of full PM (ablation) | LongDiff on LaVie · VBench prompts, 128 frames | OC: 24.62 vs 25.24 | Hurt |
| use neighbor frames only, excluding detected key frames (ablation) | LongDiff on LaVie · VBench prompts, 128 frames | OC: 24.15 vs 25.24 | Hurt |
| use neighbor frames plus additional neighbor frames instead of IFS (ablation) | LongDiff on LaVie · VBench prompts, 128 frames | OC: 24.21 vs 25.24 | Hurt |
| use detected key frames only, excluding neighbor frames (ablation) | LongDiff on LaVie · VBench prompts, 128 frames | OC: 22.12 vs 25.24 | Hurt |
| use detected key frames plus additional key frames instead of IFS (ablation) | LongDiff on LaVie · VBench prompts, 128 frames | OC: 22.33 vs 25.24 | Hurt |
| select uniform frames plus neighbor frames instead of IFS (ablation) | LongDiff on LaVie · VBench prompts, 128 frames | OC: 24.85 vs 25.24 | Hurt |
| select random frames plus neighbor frames instead of IFS (ablation) | LongDiff on LaVie · VBench prompts, 128 frames | OC: 24.69 vs 25.24 | Hurt |
Surprising findings
- Less than 40% of query and key features satisfy the theoretical distinguishability condition at 128 frames, suggesting limited position discrimination in short models for long sequences.
Limitations
- Relies on pretrained short-video models with relative positional encodings; applicability to other architectures is not shown in main text.
- Key-frame detection uses hand-designed heuristics (image entropy and frame differencing) rather than learned selection.
Experimental setup
Datasets: VBench prompts (200 test prompts)
Benchmarks: VBench
Metrics: Subject Consistency Background Consistency Motion Smoothness Temporal Flickering Imaging Quality Overall Consistency
Baselines: Direct Sliding FreeNoise FreeLong
Builds on: relative positional encoding RoPE FreeNoise FreeLong video key-frame detection
Closest papers by meaning
- FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention (NeurIPS 2024 · similarity 0.93)
- FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling (ICLR 2024 · similarity 0.90)
- StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation (NeurIPS 2024 · similarity 0.89)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.89)
- Ouroboros-Diffusion: Exploring Consistent Content Generation in Tuning-free Long Video Diffusion (AAAI 2025 · similarity 0.89)
- AdaDiff: Adaptive Step Selection for Fast Diffusion Models (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.