Research map / Video / Video generation
Grid Diffusion Models for Text-to-Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates text-to-video by representing a video as a 2x2 grid image, using a fine-tuned Stable Diffusion for key grid generation and autoregressive grid interpolation trained on 0.1M WebVid samples.
Problem
Text-to-video generation is costly in GPU memory and data, since existing methods use 3D U-Nets or autoregressive generation with large text-video datasets.
Approach
Reduce video to image dimensions by arranging four key frames into a grid image generated by a fine-tuned 2D text-to-image diffusion model, then interpolate frames autoregressively with an InstructPix2Pix-style masked-grid model, so GPU memory stays fixed regardless of frame count.
Key results
- On MSR-VTT zero-shot, achieves CLIPSIM 0.3096 and FVD 375 using 0.1M training samples, beating VideoFusion (0.2930, 550) and Show-1 (0.3072, 538).
- On UCF-101 zero-shot, achieves IS 62.88 and FVD 340, outperforming PYoCo (47.76, 355) and Show-1 (35.42, 394).
- Memory use stays near single-image Stable Diffusion across frame counts, with a 74.08% reduction versus FreeNoise at 128 frames.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train grid diffusion on 0.1 million Webvid-10M samples | Grid Diffusion · MSR-VTT test | CLIPSIM: 0.3072 -> 0.3096 | Helped |
| train grid diffusion on 0.1 million Webvid-10M samples | Grid Diffusion · MSR-VTT test | FVD: 456 -> 375 | Helped |
| train grid diffusion on 0.1 million Webvid-10M samples | Grid Diffusion · UCF-101 | IS: 47.76 -> 62.88 | Helped |
| train grid diffusion on 0.1 million Webvid-10M samples | Grid Diffusion · UCF-101 | FVD: 355 -> 340 | Helped |
| train grid diffusion on 0.1 million Webvid-10M samples | Grid Diffusion · CGcaption | CLIPSIM: 0.3025 -> 0.3282 | Helped |
| generate 64-frame videos with grid diffusion | Grid Diffusion · MSR-VTT, 64 frames | CLIPSIM: 0.2996 -> 0.3044 | Helped |
| generate 128-frame videos with grid diffusion | Grid Diffusion · MSR-VTT, 128 frames | CLIPSIM: 0.3034 -> 0.3000 | Hurt |
| generate 128-frame videos with grid diffusion | Grid Diffusion · MSR-VTT, 128 frames | B-FVD-16: 726 -> 364 | Helped |
| add autoregressive conditioning on previous frames (ablation) | Grid Diffusion · MSR-VTT | CLIPSIM: 0.2982 -> 0.3096 | Helped |
| add autoregressive conditioning on previous frames (ablation) | Grid Diffusion · MSR-VTT | FVD: 504 -> 375 | Helped |
| use a 2×2 key grid instead of a 4×4 key grid (ablation) | Grid Diffusion · MSR-VTT | CLIPSIM: 0.2902 -> 0.3096 | Helped |
| use a 2×2 key grid instead of a 4×4 key grid (ablation) | Grid Diffusion · MSR-VTT | FVD: 343 -> 375 | Hurt |
Surprising findings
- A 4x4 grid variant gives a slightly better FVD than the 2x2 grid despite lower per-frame resolution, while scoring lower on CLIPSIM and IS.
- Freezing convolution layers during fine-tuning hurts results much more than freezing attention layers (FVD 724 vs 512 on MSR-VTT).
Limitations
- Depends on a pre-trained text-to-image model, which limits the scope of generated content.
- Frame count is tied to the grid layout and autoregressive interpolation steps, and video resolution is constrained by grid cell size (254x254 per frame in the main setting).
- Per-frame inference time is not lower than baselines even though memory is lower.
Experimental setup
Datasets: WebVid-10M LAION-5B
Benchmarks: MSR-VTT UCF-101 CGCaption
Metrics: CLIPSIM FVD Inception Score Block-FVD human preference win rate
Baselines: VideoFusion FreeNoise VideoCrafter CogVideo Make-A-Video Video LDM Show-1 PYoCo
Builds on: Stable Diffusion InstructPix2Pix VideoFusion FreeNoise
Closest papers by meaning
- EIDT-V: Exploiting Intersections in Diffusion Trajectories for Model-Agnostic, Zero-Shot, Training-Free Text-to-Video Generation (CVPR 2025 · similarity 0.92)
- Vivid-ZOO: Multi-View Video Generation with Diffusion Model (NeurIPS 2024 · similarity 0.91)
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CVPR 2025 · similarity 0.90)
- StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text (CVPR 2025 · similarity 0.90)
- VideoTetris: Towards Compositional Text-to-Video Generation (NeurIPS 2024 · similarity 0.90)
- LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation with Linear Computational Complexity (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.