Research map / Video / Video generation
SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SEINE, a short-to-long video diffusion model with a random-mask conditioning layer that generates transition videos between scenes, image-to-video animation, and autoregressive long video prediction.
Problem
Existing AI video generation produces only shot-level clips depicting a single scene; generating smooth, creative transitions between scenes and coherent story-level long videos is under-explored.
Approach
A random-mask video diffusion model that concatenates noisy latents, a binary frame mask, and masked clean latents as input, so the same model handles transition (given first and last frames), prediction (given preceding frames), and image animation (given first frame) by sampling the mask with Bernoulli(p=0.15); initialized from LaVie-base.
Key results
- On MSR-VTT transition evaluation, SEINE reaches CLIPSIM-text 0.2726, CLIPSIM-Scenes 0.7740, CLIPSIM-frames 0.9675, above Morphing (0.2535/0.7707/0.9569) and SD-based transition (0.2665/0.6080/0.8809).
- Human raters (100 videos, 10 raters) preferred SEINE over Morphing 83%, over VQGAN-based 85%, and over SD-based 92% of the time.
- On UCF-101 auto-regressive prediction of 1024 frames, SEINE degrades more slowly than TATS in FVD and avoids the grid-like artifacts and visual collapse seen in TATS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SEINE instead of SD-based Transition | Generative transition · MSR-VTT test | CLIPSIM-text: 0.2665 -> 0.2726 | Helped |
| Use SEINE instead of Morphing | Generative transition · MSR-VTT test | CLIPSIM-Scenes: 0.7707 -> 0.7740 | Helped |
| Use SEINE instead of Morphing | Generative transition · MSR-VTT test | CLIPSIM-frames: 0.9569 -> 0.9675 | Helped |
| Use SEINE instead of Morphing | Transition generation · Human evaluation | Preference: 83% | Helped |
| Use SEINE instead of VQGAN-based Transition | Transition generation · Human evaluation | Preference: 85% | Helped |
| Use SEINE instead of SD-based Transition | Transition generation · Human evaluation | Preference: 92% | Helped |
| Fine-tune SEINE on UCF-101 | SEINE autoregressive video prediction · UCF-101, 1024-frame videos | FVD | Helped |
| Use zero-shot SEINE instead of TATS | Autoregressive video prediction · UCF-101, 1024-frame videos | FVD | Mixed |
Surprising findings
- Its CLIPSIM-scenes and CLIPSIM-frames scores are not much higher than baselines, because blending ghosting artifacts still leave frame coherence scores high.
Limitations
- Transitions look abrupt or inconsistent when source and target scenes are too dissimilar or share few common elements.
- Watermarks from WebVid10M training data can appear in generated videos and persist through autoregressive prediction.
- Output quality depends on alignment between the text prompt and the conditioning image.
Experimental setup
Datasets: WebVid10M internal watermark-free datasets MSR-VTT UCF-101
Benchmarks: MSR-VTT UCF-101
Metrics: CLIPSIM-text CLIPSIM-scenes CLIPSIM-frames human preference FVD
Baselines: Morphing VQGAN-based transition SD-based transition TATS Gen-2
Builds on: LaVie Stable Diffusion DDIM TATS
Closest papers by meaning
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.90)
- StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation (NeurIPS 2024 · similarity 0.90)
- MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequences (ICLR 2025 · similarity 0.89)
- FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling (ICLR 2024 · similarity 0.89)
- EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation (CVPR 2025 · similarity 0.89)
- TokenFlow: Consistent Diffusion Features for Consistent Video Editing (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.