Research map / Video / Video generation
Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Snap Video rewrites the EDM diffusion framework for high-resolution video and replaces the U-Net with a scalable FIT-style transformer that jointly models space and time, scaled to 3.9B parameters for text-to-video generation.
Problem
Image-based diffusion architectures (U-Nets) and training frameworks scale poorly and model motion poorly when repurposed for text-to-video generation, due to redundancy across frames and separable spatial/temporal computation.
Approach
Adjust EDM with an input scaling factor (sigma_in = s*sqrt(T)) to restore SNR for redundant video frames, treat images as infinite-frame-rate videos for joint training, and use a FIT-based transformer that performs joint spatiotemporal computation over a learnable compressed latent token set.
Key results
- On zero-shot UCF-101, Snap Video (512x288) reaches FVD 200.2 vs. 355.2 for PYoCo and 367.2 for Make-A-Video.
- The 500M FIT trains 3.31x faster and infers 4.49x faster than the 284M U-Net while achieving better FID (3.07 vs 4.90) and CLIPSIM.
- In user studies vs. Gen-2, Pika and Floor33, Snap Video was preferred for text alignment in 81%, 80%, 81% of cases and for motion quality in 79%, 71%, 79% of cases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the 284M U-Net with a 500M FIT (ablation) | Snap Video video generator · internal dataset (64 × 36 px) | training throughput (ms/video/GPU): 230.3 -> 69.5 | Helped |
| replace the 284M U-Net with a 500M FIT (ablation) | Snap Video video generator · internal dataset (64 × 36 px) | inference throughput (ms/video/GPU): 105.1 -> 23.4 | Helped |
| scale FIT from 500M to 3.9B parameters (ablation) | Snap Video FIT · internal dataset (64 × 36 px) | FID: 3.07 -> 2.51 | Helped |
| use the scaled diffusion framework instead of original EDM (ablation) | Snap Video FIT 500M · internal dataset (64 × 36 px) | FID: 6.58 -> 3.07 | Helped |
| set σdata to 1 (ablation) | Snap Video scaled diffusion framework · internal dataset (64 × 36 px) | FID: 4.03 -> 3.07 | Helped |
| reduce σin to 1/4.0 (ablation) | Snap Video scaled diffusion framework · internal dataset (64 × 36 px) | FID: 3.07 -> 3.50 | Hurt |
| treat images as infinite-framerate videos during training (ablation) | Snap Video scaled diffusion framework · internal dataset (64 × 36 px) | FID: 4.45 -> 3.07 | Helped |
| use Snap Video instead of PYoCo | text-to-video generation · UCF-101 | FVD: 355.2 -> 200.2 | Helped |
| use Snap Video instead of PYoCo | text-to-video generation · UCF-101 | Inception Score: 47.46 -> 38.89 | Hurt |
| use Snap Video instead of PYoCo | text-to-video generation · MSR-VTT | CLIP-FID: 9.73 -> 8.48 | Helped |
| use Snap Video instead of VideoFactory | text-to-video generation · MSR-VTT | CLIPSIM: 0.3005 -> 0.2793 | Hurt |
| use Snap Video instead of Gen-2 | text-to-video generation · 65 dynamic-scene prompts | video-text alignment preference (%): 81.0% | Helped |
Surprising findings
- Naively introducing the EDM input scaling factor alters the training target and makes the objective explode at small noise levels, requiring a reformulation.
- On MSR-VTT, Snap Video produces lower CLIPSIM than Make-A-Video despite better FVD, which the authors attribute to using T5 rather than CLIP text embeddings.
Limitations
- Relies on an internal proprietary dataset with synthetic captions, so results are not reproducible from public data.
- Baseline comparisons to closed-source tools (Gen-2, Pika, Floor33) rely on a small user study of 65 prompts.
Experimental setup
Datasets: internal video-text dataset (1.265M images, 238k hours of video) UCF-101 MSR-VTT
Benchmarks: UCF-101 MSR-VTT
Metrics: FID FVD CLIPSIM Inception Score CLIP-FID user preference (photorealism, alignment, motion quantity, m
Baselines: CogVideo MagicVideo LVDM Video LDM VideoFactory Make-A-Video PYoCo Gen-2 Pika Floor33
Builds on: EDM FIT U-Net Imagen Video Make-A-Video
Closest papers by meaning
- LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation with Linear Computational Complexity (CVPR 2025 · similarity 0.90)
- EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation (CVPR 2025 · similarity 0.90)
- Towards Precise Scaling Laws for Video Diffusion Transformers (CVPR 2025 · similarity 0.89)
- VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control (ICLR 2025 · similarity 0.89)
- Training-Free Efficient Video Generation via Dynamic Token Carving (NeurIPS 2025 · similarity 0.89)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.