Research map / Video / Video generation
Pyramidal Flow Matching for Efficient Video Generative Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Pyramidal Flow Matching, a unified single-DiT flow-matching framework with spatial and temporal pyramids that generates up to 10-second 768p videos at 24 FPS with reduced training cost.
Problem
Video generation requires modeling a huge spatiotemporal space, making training computationally and data intensive; cascaded pipelines reduce cost but use separate models per resolution, hindering knowledge sharing and flexibility.
Approach
Reinterpret the denoising trajectory as pyramid stages where only the last stage runs at full resolution, linking stages with a piecewise flow matching objective trained jointly in one DiT, with renoising at jump points during inference, plus an autoregressive temporal pyramid that compresses full-resolution history.
Key results
- On VBench, total score 81.72 and quality score 84.74 (highest quality score among compared models), exceeding CogVideoX-5B's 81.61 total and 82.75 quality, using public data only.
- On EvalCrafter, final sum score 244, the best among public-data models (above VideoCrafter2's 243).
- Trained within 20.7k A100 GPU hours to generate 5s (up to 10s) 768p 24 FPS videos; the spatial pyramid reportedly converges about three times faster in FID than standard flow matching.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use spatial pyramidal flow instead of standard flow matching (ablation) | Text-to-image generation · MS-COCO (3K prompts) | FID convergence | Helped |
| use a temporal pyramid of compressed history instead of full-sequence diffusion (ablation) | Video generation · Video generation ablation | Visual quality and temporal consistency | Helped |
| add corrective Gaussian noise at spatial pyramid stage jumps (ablation) | Pyramidal flow matching · High-resolution image generation | Visual detail | Helped |
| use blockwise causal attention instead of bidirectional attention (ablation) | Autoregressive video generation · 1-second video generation | Temporal coherence | Helped |
| use Pyramidal Flow Matching instead of Gen-3 Alpha | Video generation · VBench | Quality Score: 84.11 -> 84.74 | Helped |
| use Pyramidal Flow Matching instead of T2V-Turbo | Video generation · VBench | Semantic Score: 74.76 -> 69.62 | Hurt |
| use Pyramidal Flow Matching instead of Gen-2 | Video generation · EvalCrafter | Final Sum Score: 254 -> 244 | Hurt |
Surprising findings
- Semantic score on VBench is relatively low (69.62), attributed to coarse synthetic captions from video LLMs.
- Face consistency on EvalCrafter is weak, which the authors attribute to the temporal pyramid history compression.
Limitations
- Semantic alignment and color/appearance style scores lag, attributed to coarse captions and limited image-quality capacity.
- Inherits SD3-Medium initialization weaknesses, yielding low human action scores.
- Face consistency degraded by temporal pyramid history compression.
Experimental setup
Datasets: LAION-5B CC-12M SA-1B JourneyDB WebVid-10M OpenVid-1M Open-Sora Plan videos MSR-VTT MS-COCO
Benchmarks: VBench EvalCrafter MS-COCO MSR-VTT
Metrics: VBench total score quality score semantic score motion smoothness dynamic degree EvalCrafter final sum score FID FVD user preference
Baselines: CogVideoX-2B CogVideoX-5B Open-Sora 1.2 Open-Sora Plan v1.3 VideoCrafter2 T2V-Turbo Kling Gen-3 Alpha Gen-2 Pika 1.0 ModelScope Show-1
Builds on: flow matching SD3 Medium MM-DiT MAGVIT-v2 piecewise flow Patch n' Pack
Closest papers by meaning
- Improving Progressive Generation with Decomposable Flow Matching (NeurIPS 2025 · similarity 0.89)
- From Slow Bidirectional to Fast Autoregressive Video Diffusion Models (CVPR 2025 · similarity 0.88)
- Contextual Flow Matching: Adaptive Step Selection in Flow Models for Efficient Visual Generation (NeurIPS 2026 · similarity 0.87)
- FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis (CVPR 2024 · similarity 0.87)
- FreeLong: Training-Free Long Video Generation with SpectralBlend Temporal Attention (NeurIPS 2024 · similarity 0.86)
- OSV: One Step is Enough for High-Quality Image to Video Generation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.