Research map / Video / Video generation
Long Video Diffusion Generation with Segmented Cross-Attention and Content-Rich Video Data Curation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presto extends the Allegro DiT video model to 15-second videos by using Segmented Cross-Attention to tie each temporal segment to a progressive sub-caption, trained on a curated 261k-clip LongTake-HD dataset.
Problem
Long video generation (about 15 seconds) that keeps content diversity and long-range scenario coherence, while single-prompt text-to-video models produce short clips and naive multi-prompt or extension methods suffer abrupt transitions or error propagation.
Approach
Split the DiT hidden states along time into segments that each cross-attend to their own progressive sub-caption, with overlapping boundary frames (OSCA) so adjacent segments blend smoothly; no new parameters are added, and a curated dataset of progressive sub-captions supports training.
Key results
- Presto reaches 78.5% VBench Semantic Score and 100% Dynamic Degree, above Allegro (+5.5 points) and Gen-3 (+3.3 points).
- In a 62-prompt user study with 2,232 ratings, Presto wins overall against Allegro (54.9% win vs 27.0% lose) and Gen-3 (45.0% vs 38.8%).
- Ablation: OSCA reaches 74.7 overall vs 73.7 for SSCA and 73.1 for ISCA; removing meticulous data filtering drops overall score by 2.7 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Presto with progressive sub-captions and OSCA instead of Allegro | Presto · VBench | Semantic Score: 73.0 -> 78.5; +5.5% | Helped |
| use Presto with progressive sub-captions and OSCA instead of Gen-3 | Presto · VBench | Semantic Score: 75.2 -> 78.5; +3.3% | Helped |
| replace OSCA with SSCA (ablation) | Presto · VBench ablation | Overall Score: 74.7 -> 73.7 | Hurt |
| replace OSCA with ISCA (ablation) | Presto · VBench ablation | Overall Score: 74.7 -> 73.1 | Hurt |
| remove meticulous dataset filtering (ablation) | Presto · VBench ablation | Overall Score: 74.7 -> 72.0 | Hurt |
| concatenate sub-captions into a single long condition (ablation) | Presto · VBench ablation | Overall Score: 74.7 -> 71.8 | Hurt |
Surprising findings
- Isolated per-segment cross-attention (ISCA) performs worst, indicating that boundary interaction between segments matters.
- Sequential segmented attention gives stronger long-range coherence but reduces content richness.
Limitations
- Quality scores degrade when dynamics are complex and varied, per the authors' hypothesis.
- Uses a fixed number of five segments and does not explore auto-regressive generation.
Experimental setup
Datasets: HD-VILA-100M Panda 70M WebVid-10M LongTake-HD
Benchmarks: VBench
Metrics: VBench Semantic Score Dynamic Degree Overall Score Overall Consistency win rate
Baselines: Allegro Gen-3 TALC Merge Videos
Builds on: Allegro window attention TALC
Closest papers by meaning
- FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention (NeurIPS 2024 · similarity 0.89)
- Vivid-ZOO: Multi-View Video Generation with Diffusion Model (NeurIPS 2024 · similarity 0.89)
- FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling (ICLR 2024 · similarity 0.89)
- OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation (ICLR 2025 · similarity 0.89)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.88)
- SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.